DEV Community

Super Lewis
Super Lewis

Posted on Fully Autonomous

A/B test two LLMs with one Python script: Qwen 4 Max vs Claude Opus 5.5 on score, latency, tokens and cost

Comparing two language models fairly takes four numbers per request: time to first token, total time, output tokens and cost. This post gives a small script that collects them through one OpenAI-compatible client, plus what it showed for Qwen 4 Max and Claude Opus 5.5. Short answer: they tied on correctness for everyday tasks, Qwen 4 Max cost about a third as much, and public data favours Opus 5.5 once the work gets large.

Disclosure: I ran this on apimodels.app, which is the API I work on. apimodels.app is a multi-model API gateway: one key and one OpenAI-compatible endpoint for image, video, audio and language models, which is why one client can reach both models. I wrote this from my own test data, with AI assistance for drafting and editing.

The script

import os, time
from openai import OpenAI

client = OpenAI(api_key=os.environ["LLM_API_KEY"],
                base_url=os.environ["LLM_BASE_URL"], timeout=900)

# List prices, USD per million tokens: (input, output)
PRICE = {"qwen4-max": (1.50, 4.50), "claude-opus-5-5": (4.00, 20.00)}

def run(model, prompt):
    t0, first, text, usage = time.time(), None, [], None
    stream = client.chat.completions.create(
        model=model, messages=[{"role": "user", "content": prompt}],
        max_tokens=20000, stream=True,
        stream_options={"include_usage": True})
    for chunk in stream:
        if chunk.usage:
            usage = chunk.usage
        if chunk.choices and chunk.choices[0].delta.content:
            first = first or time.time() - t0
            text.append(chunk.choices[0].delta.content)
    p_in, p_out = PRICE[model]
    cost = (usage.prompt_tokens * p_in + usage.completion_tokens * p_out) / 1e6
    return {"model": model, "answer": "".join(text),
            "first_token_s": round(first or 0, 1),
            "total_s": round(time.time() - t0, 1),
            "output_tokens": usage.completion_tokens,
            "cost_usd": round(cost, 4)}

prompts = open("prompts.txt", encoding="utf-8").read().split("\n---\n")
for prompt in prompts:
    for model in PRICE:
        r = run(model, prompt)
        print(r["model"], r["first_token_s"], r["total_s"],
              r["output_tokens"], r["cost_usd"])
Enter fullscreen mode Exit fullscreen mode

The script only collects. Grade correctness separately, with code: hidden test cases, assertions, or a comparison against a known answer. Reading an answer and deciding it looks right is how weak models pass evaluations.

What I ran

Seven tasks with 93 programmatic checks between them: a timeline-merging function against 50 hidden cases, a cache class with five planted bugs, a refund agent with tools and a policy, five exact questions over 1,500 log lines, a Chinese group chat converted to JSON, three counting puzzles that can be verified by enumeration, and Chinese product copy under five hard constraints. Both models on default thinking, each task once, on November 17, 2026.

Results

Both models passed all 93 checks.

  • Total time: Qwen 4 Max 415 s, Opus 5.5 341 s
  • Time to first token, puzzles excluded: Qwen 4 Max 15 to 62 s, Opus 5.5 8 to 26 s
  • Output tokens including thinking: Qwen 4 Max about 22,100, Opus 5.5 about 37,700
  • Cost at the platform's posted rates: about $0.21 against about $0.64

One thing to watch: on the puzzles Opus 5.5 used the whole 20,000-token output cap. Leave headroom in max_tokens for thinking models, or the answer can be cut off before it starts.

API differences worth knowing

  • Thinking: Qwen 4 Max lets you turn it off per request. Opus 5.5 does not, but you can lower its effort level from the default of medium.
  • Input: Qwen 4 Max takes text, images and video. Opus 5.5 takes text and images.
  • Context: one million tokens on both, with no long-prompt surcharge.
  • Cached input, list price per million: $0.075 for Qwen 4 Max, $0.20 for Opus 5.5. In agent sessions that resend a long context, caching narrows the real gap.

What seven easy tasks cannot show

  • In Alibaba's sixteen-benchmark launch comparison, Qwen 4 Max leads six and Opus 5.5 leads ten. ProgramBench is 31.5 against 65.0 and Terminal-Bench 4.0 is 48.5 against 59.6. Qwen 4 Max leads the legal agent benchmark HLAB, 21.3 against 9.2.
  • Kingy AI ran eight real GitHub issues twice each: 11 solved for Opus 5.5 and 9 for Qwen 4 Max, with median active time of 86 seconds against 431.

Picking one

  • Single functions, bug fixes, structured extraction, rule-bound tool calls: either, and Qwen 4 Max costs about a third.
  • Multi-file refactors, terminal agents, long coding sessions: Claude Opus 5.5.
  • Legal or contract work, or video input: try Qwen 4 Max first.

When this result does not apply

  • Hard, long tasks. Swap your own tasks into the script and measure.
  • Escalation routing. Sending everything to the lower-cost model first and retrying failures on Opus 5.5 cost 2.8 times as much as Opus 5.5 alone in Kingy AI's test.
  • Chinese-language quality. Both models got full marks on my two Chinese tasks, which is too few to draw on.

Benchmark tables and sources are on the Qwen 4 Max vs Claude Opus 5.5 comparison page at apimodels.app.

Top comments (0)