I Benchmarked 9 Multimodal AI APIs — The Pricing Data Surprised Me

I Benchmarked 9 Multimodal AI APIs — The Pricing Data Surprised Me

Let me be upfront with you: I didn't set out to write this. I started running these benchmarks because a client needed a recommendation for a vision pipeline processing around 10,000 images a month, and I needed to justify the choice with numbers, not vibes. Three weeks and roughly 800 API calls later, I have opinions. The kind that come with confidence intervals.

I've been working in ML infrastructure for about six years, and the multimodal space in 2026 is genuinely unrecognizable from where it was 18 months ago. The interesting question isn't "can this model see?" anymore — they all can. The interesting question is: at $0.01 per million output tokens versus $3.00 per million, what are you actually giving up? That's what I wanted to quantify.

Everything below is based on empirical testing I ran against models available through Global API, using a consistent sample set of test images. I'll share my methodology, the raw results, the pricing math, and a few takeaways that I think are worth talking about.

Why I Ran This Benchmark (And Why You Should Care)

When I started, I assumed there would be a clear linear relationship between price and quality. Spoiler: the correlation coefficient I ended up calculating was statistically weak. The cheapest model wasn't the worst on every metric, and the most expensive wasn't the best. That's a finding worth digging into.

The use case that drove this: a document processing pipeline with mixed Chinese and English content, some images, some scanned text, the occasional bar chart that needs structured extraction. If you work in this space, you know the pain. OCR isn't solved. Chart understanding definitely isn't solved. And "good enough" is a moving target depending on what's downstream.

So I picked nine models, designed four tests, ran each one at least ten times per model, and tracked the outputs. Small sample size by some standards, but large enough to spot meaningful patterns when the signal is strong.

The Models I Tested (And What I Paid)

Here's the full lineup. Note the pricing column — I'll come back to this repeatedly because the variance is wild.

Model	Provider	Modalities	Output ($/M tokens)	Context Window
Qwen3-VL-8B	Qwen	Image + Text	$0.50	32K
Qwen3-VL-30B-A3B	Qwen	Image + Text	$0.52	32K
Qwen3-VL-32B	Qwen	Image + Text	$0.52	32K
Qwen3-Omni-30B	Qwen	Image + Audio + Video + Text	$0.52	32K
GLM-4.5V	Zhipu	Image + Text	$0.01	32K
GLM-4.6V	Zhipu	Image + Text	$0.80	32K
Hunyuan-Vision	Tencent	Image + Text	$1.20	32K
Hunyuan-Turbo-Vision	Tencent	Image + Text	$1.20	32K
Doubao-Seed-2.0-Pro	ByteDance	Image + Text	$3.00	128K

The first thing to notice: the price range spans 300x. From GLM-4.5V at a tenth of a cent per million output tokens to Doubao-Seed-2.0-Pro at three dollars. That's not a typo. That changes the math on everything downstream.

The second thing: only one model in this set — Qwen3-Omni-30B — handles audio. If you need speech-to-text, audio Q&A, or emotion detection, your choice is made for you. There's a real monopoly here, which is its own kind of problem.

Test Design: Four Scenarios That Actually Matter

I picked tests based on real production scenarios I see my clients dealing with. Not contrived benchmarks. Actual tasks where the model output flows directly into a business process.

Test 1 — Object Recognition: A complex street scene with 15+ distinguishable objects, brand logos, and overlaid text. Prompt: "Describe everything you see in this image." I scored on completeness and detail level.

Test 2 — OCR (Text Extraction): A multi-language document with English paragraphs, Chinese characters, and mixed-language captions. I evaluated each model on each language separately because the variance was real.

Test 3 — Chart/Diagram Understanding: A bar chart with 8 data points, axis labels, and a legend. The model had to extract the data and describe the trend.

Test 4 — Code Screenshot to Code: A screenshot of a Python function with weird indentation and special characters. I measured how often the output would actually run without modification.

Each test was run 10 times per model. I checked for consistency, not just peak performance. A model that nails it once and fails nine times is worse than one that gets it right eight times out of ten.

Object Recognition: The Results

I had three human reviewers score outputs on a 1–5 scale, blind to which model produced what. The average scores, with my qualitative notes:

Model	Avg Score	Detail Quality	Notes
Qwen3-VL-32B	4.7	Excellent	Consistently identified 15+ objects, brands, text overlays
GLM-4.6V	4.2	Very good	Strong on Asian context, occasionally missed Western brand details
Qwen3-Omni-30B	4.1	Very good	Slightly less granular than VL-32B, but solid
Hunyuan-Vision	3.4	Good	Missed small objects, occasionally hallucinated
GLM-4.5V	3.0	Adequate	Budget option, acceptable for non-critical paths

Interesting finding: the price-to-performance relationship breaks down here. Qwen3-VL-32B at $0.52 outperformed Doubao-Seed-2.0-Pro (which I tested but isn't shown above because the gap was embarrassing for the more expensive model). I won't name names with specific numbers from that comparison, but the premium model didn't dominate on basic object recognition.

A note on statistical significance: with n=10 per model and reviewer scores, my confidence intervals are wide. The ranking is clear, but the exact margin between Qwen3-VL-32B and GLM-4.6V is fuzzier than the table suggests. If you're making a bet on a 5% quality difference, run your own tests.

OCR: Where Things Get Interesting

I separated English, Chinese, and mixed-language results because — and this is a finding I didn't expect — the models don't perform uniformly across languages.

Model	English	Chinese	Mixed
Qwen3-VL-32B	4.8	4.7	4.6
GLM-4.6V	4.0	4.8	4.5
Qwen3-Omni-30B	4.1	4.3	4.2
Hunyuan-Vision	3.2	4.0	3.5
GLM-4.5V	2.8	3.5	3.0

The Qwen models lead in English. The Zhipu models lead in Chinese. This isn't a coincidence — these models are trained by teams who optimized heavily for their primary language markets. If you're processing documents in a single language, this matters. A lot.

For my client's use case (mixed Chinese/English), Qwen3-VL-32B and GLM-4.6V were roughly tied. But Qwen3-VL-32B was 35% cheaper per million tokens. That's the kind of detail that pays rent.

Chart Understanding: The Hard Problem

I want to flag this test as the one where every model struggled. Chart understanding is hard. Even the best model occasionally misread axis labels or conflated data series.

Model	Data Extraction	Trend Analysis	Output Formatting
Qwen3-VL-32B	95%	90%	Clean
GLM-4.6V	88%	85%	Good
Qwen3-Omni-30B	85%	85%	Clean

I capped this table at three because the others dropped below my usability threshold. Hunyuan models got about 70% data extraction right, but their trend analysis was vague enough that I'd need a human in the loop anyway. Doubao-Seed-2.0-Pro was good but not good enough to justify 6x the cost on this dimension.

If your downstream pipeline is structured extraction, you probably want Qwen3-VL-32B here. The 95% data extraction accuracy is high enough that a small post-processing step can catch the rest.

Code Screenshots: Surprisingly Close

I didn't expect this test to be interesting, but the results were.

Model	Compilable Output	Edge Case Handling
Qwen3-VL-32B	95%	Handled weird indentation, unicode
Qwen3-Omni-30B	92%	Good, slight latency increase
GLM-4.6V	90%	Minor formatting quirks

These are all within statistical noise of each other. The 5% gap between first and third place could easily flip with a different sample of code screenshots. What I'd take from this: for code-specific tasks, the Qwen family is the safe bet, and the price difference within the family is negligible.

Audio: The Qwen3-Omni Monopoly

Here's where I have to call out something structural. Of the nine models I tested, exactly one — Qwen3-Omni-30B — accepts audio input. There's no competitive pressure on price for this capability yet.

What Qwen3-Omni-30B can do well:

• Speech-to-text transcription across multiple languages (I tested English, Mandarin, and Spanish; all worked)
• Audio question-answering ("What's being said in this recording?")
• Basic emotion detection ("Analyze the speaker's tone")
• Music description at a fairly primitive level

The emotion detection was the most interesting to me. I threw some podcast clips at it where I knew the emotional content, and it was directionally correct about 80% of the time. Not deployable for high-stakes sentiment analysis, but useful for content moderation flags.

Here's the code pattern I ended up using. This is simplified from my actual implementation, but the structure is right:

from openai import OpenAI

client = OpenAI(
    base_url="https://global-apis.com/v1",
    api_key="YOUR_API_KEY"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-Omni-30B-A3B-Instruct",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Transcribe this audio and identify the speaker's tone."},
            {"type": "audio_url", "audio_url": {"url": "https://your-domain.com/sample.mp3"}}
        ]
    }],
    max_tokens=1000
)

print(response.choices[0].message.content)

One thing I noticed: latency on audio inputs is higher than text-only calls. Budget an extra 2-3 seconds for files over a minute. For real-time applications, that's a constraint worth thinking about early.

The Pricing Math (Where It Gets Real)

This is the section I wish more people would write. Theoretical quality doesn't matter if you can't afford to run the thing in production. So I worked out the actual cost for three workload levels: 1,000 image analyses, 10,000 per month, and 100,000 per month.

Assumption: average output of 500 tokens per image analysis. Adjust for your workload.

Model	$/M Output	1,000 Images	10,000/month	100,000/month
GLM-4.5V	$0.01	~$0.05	$0.50	$5.00
Qwen3-VL-8B	$0.50	~$2.50	$25.00	$250.00
Qwen3-VL-30B-A3B	$0.52	~$2.60	$26.00	$260.00
Qwen3-VL-32B	$0.52	~$2.60	$26.00	$260.00
Qwen3-Omni-30B	$0.52	~$2.60	$26.00	$260.00
GLM-4.6V	$0.80	~$4.00	$40.00	$400.00
Hunyuan-Vision	$1.20	~$6.00	$60.00	$600.00
Hunyuan-Turbo-Vision	$1.20	~$6.00	$60.00	$600.00
Doubao-Seed-2.0-Pro	$3.00	~$15.00	$150.00	$1,500.