I Ran 10 AI Coding Models Through 5 Tasks: A Data Scientist's Take
I'll be honest — I went into this expecting a clear winner. I came out with a scatter plot, three regressions, and a deeper appreciation for why "best" is the most dangerous word in machine learning.
Over the past three weeks I've been grinding through prompts with ten different LLMs, all routed through the same endpoint, scoring every output on a 1–10 rubric that I tried very hard not to bias. The pricing data is pulled directly from the provider pages. The scores are mine. If you disagree with a score, you're probably right — n=1 per task per model is a laughably small sample size, and I say that as someone who publishes papers with bigger samples. But trends still emerged. Let me walk you through what I found.
The Lineup
Before I touch a single benchmark, here's the cast. I've grouped them by family so you can see the obvious concentration in the open-source Chinese ecosystem, which personally I find fascinating — three of the top five are DeepSeek or Qwen variants.
| # | Model | Provider | Output $/M | Category |
|---|---|---|---|---|
| 1 | DeepSeek V4 Flash | DeepSeek | $0.25 | General (strong code) |
| 2 | DeepSeek Coder | DeepSeek | $0.25 | Code-specialized |
| 3 | Qwen3-Coder-30B | Qwen | $0.35 | Code-specialized |
| 4 | DeepSeek V4 Pro | DeepSeek | $0.78 | Premium general |
| 5 | DeepSeek-R1 | DeepSeek | $2.50 | Reasoning (code thinking) |
| 6 | Kimi K2.5 | Moonshot | $3.00 | Premium general |
| 7 | GLM-5 | Zhipu | $1.92 | Premium general |
| 8 | Qwen3-32B | Qwen | $0.28 | General purpose |
| 9 | Hunyuan-Turbo | Tencent | $0.57 | General purpose |
| 10 | Ga-Standard | GA Routing | $0.20 | Smart routing |
One quick note on Ga-Standard — it's a routing layer that picks a backend model per request. So the score fluctuates. I averaged across runs.
How I Tested
Five prompts. Each one designed to probe a different cognitive layer:
- Function implementation — flatten a nested list recursively in Python
- Bug fix — chase down an async/await race condition in JavaScript
- Algorithm — Dijkstra's shortest path in TypeScript with proper types
- Code review — security and performance audit of a Go snippet
- Full feature — Express.js endpoint with pagination and filtering
Scoring rubric: correctness (40%), code quality (25%), documentation (15%), edge-case coverage (20%). I capped myself at 10 minutes per evaluation to avoid scope creep, which is itself a form of bias — slower reviewers tend to dock points for verbosity.
Every model was hit with identical prompts, identical temperature (0.2), identical system message. I rotated the order to control for fatigue.
Aggregate Rankings
| Rank | Model | Score | Price | Value (Score/$) |
|---|---|---|---|---|
| 1 | Qwen3-Coder-30B | 8.8 | $0.35 | 25.1 |
| 2 | DeepSeek V4 Flash | 8.7 | $0.25 | 34.8 |
| 3 | DeepSeek Coder | 8.6 | $0.25 | 34.4 |
| 4 | DeepSeek V4 Pro | 9.1 | $0.78 | 11.7 |
| 5 | DeepSeek-R1 | 9.4 | $2.50 | 3.8 |
| 6 | Kimi K2.5 | 9.0 | $3.00 | 3.0 |
| 7 | Qwen3-32B | 8.3 | $0.28 | 29.6 |
| 8 | GLM-5 | 8.0 | $1.92 | 4.2 |
| 9 | Hunyuan-Turbo | 7.5 | $0.57 | 13.2 |
| 10 | Ga-Standard | 8.5* | $0.20 | 42.5* |
The Ga-Standard asterisk is doing a lot of work here. It's the highest value-per-dollar on the list, but it's not a model — it's a router. Treat it like a moving target.
The Correlation I Didn't Expect
I plotted quality against price and ran a quick Pearson correlation. The result: r = 0.31, p ≈ 0.38. Not statistically significant. In English: spending more does not reliably buy you better code generation in 2026. The cheap tier is competitive with the premium tier more often than the pricing would suggest.
That single chart changed how I think about LLM procurement. We pay 10x for Kimi K2.5 versus DeepSeek V4 Flash and get, on average, 0.3 points of quality. That's not a deal — that's a rounding error.
Task-by-Task Breakdown
Task 1: Flatten a Nested List (Python)
Easy opener. Or so I thought.
| Model | Score | Behavior |
|---|---|---|
| DeepSeek V4 Flash | 9.0 | Recursive, type hints, done |
| Qwen3-Coder-30B | 9.0 | Added iterative variant + edge cases |
| DeepSeek Coder | 8.5 | Correct, slightly verbose |
| Kimi K2.5 | 9.0 | Most readable, real docstring |
| DeepSeek-R1 | 9.5 | Big-O breakdown, three approaches |
DeepSeek-R1 won this round by being the only model that volunteered a complexity analysis without being asked. That's a reasoning premium, and it's the reason R1 sits at $2.50/M output. You're paying for the inner monologue.
Task 2: Async Race Condition (JavaScript)
The bug:
let data = null;
fetch('/api/data').then(r => r.json()).then(d => data = d);
console.log(data); // Always logs null — race condition!
Every model correctly identified the issue. Not a single one missed it. That's either a sign these benchmarks are too easy, or that async debugging is well-represented in training data. Probably both.
| Model | Score | Notes |
|---|---|---|
| DeepSeek V4 Flash | 9.0 | Clear explanation, three fixes |
| Qwen3-Coder-30B | 9.0 | Added error handling |
| DeepSeek Coder | 8.5 | Correct, minimal commentary |
| Qwen3-32B | 8.5 | Good fix, wordy |
Tie: DeepSeek V4 Flash and Qwen3-Coder-30B. I'd give the slight edge to Qwen3-Coder-30B on production-readiness because it wrapped the fix in a try/catch without being prompted.
Task 3: Dijkstra in TypeScript
This is where the tiers separated.
| Model | Score | Notes |
|---|---|---|
| DeepSeek-R1 | 9.5 | Type-safe, priority queue, comments |
| DeepSeek V4 Pro | 9.0 | Clean, slightly less defensive |
| Qwen3-Coder-30B | 8.8 | Solid implementation |
| DeepSeek V4 Flash | 8.5 | Worked first try, no generics |
R1 doesn't just write Dijkstra — it writes the Dijkstra you'd write if you were being graded. That's the reasoning tax. At $2.50/M output, it's also where your cost projection starts to matter.
Task 4: Code Review (Go)
I fed each model a deliberately vulnerable Go snippet: SQL injection, unchecked error, goroutine leak. Pure security triage.
| Model | Score | Caught |
|---|---|---|
| DeepSeek-R1 | 9.5 | All three, plus a race condition I missed |
| Kimi K2.5 | 9.0 | All three |
| DeepSeek V4 Pro | 8.5 | SQL + error, missed goroutine leak |
| DeepSeek V4 Flash | 8.0 | SQL + error, missed goroutine |
The goroutine leak was a deliberately tricky ask. R1 caught it. Premium tier matters here. If you're shipping Go services in prod, the $2.50/M for R1 starts to look cheap relative to a CVE.
Task 5: Full REST Endpoint (Express.js)
The big one. Filter, paginate, error handling, the works.
| Model | Score | Notes |
|---|---|---|
| Kimi K2.5 | 9.2 | Production-ready, validation included |
| DeepSeek V4 Pro | 9.0 | Clean, good comments |
| Qwen3-Coder-30B | 8.7 | Worked, no input validation |
| DeepSeek-R1 | 9.4 | Over-engineered but bulletproof |
R1's output was 180 lines for what should have been 60. But every edge case was covered. Whether that's a feature or a bug depends on your team. In a startup shipping fast, you want Qwen3-Coder-30B. In a regulated environment, you want R1 and a longer PR review cycle.
The Cost Math Nobody Wants to Do
Let's say you're a team of 10 engineers, each making ~200 LLM calls per day averaging 500 output tokens. That's 1M tokens/day.
| Model | Daily Cost | Monthly Cost | Annual Cost |
|---|---|---|---|
| DeepSeek V4 Flash | $0.25 | $7.50 | $90 |
| Qwen3-Coder-30B | $0.35 | $10.50 | $126 |
| DeepSeek Coder | $0.25 | $7.50 | $90 |
| Qwen3-32B | $0.28 | $8.40 | $100.80 |
| Hunyuan-Turbo | $0.57 | $17.10 | $205.20 |
| DeepSeek V4 Pro | $0.78 | $23.40 | $280.80 |
| GLM-5 | $1.92 | $57.60 | $691.20 |
| DeepSeek-R1 | $2.50 | $75.00 | $900 |
| Kimi K2.5 | $3.00 | $90.00 | $1,080 |
| Ga-Standard | $0.20 | $6.00 | $72 |
So your premium-tier stack — R1 + Kimi — costs $1,980/year versus $90/year for DeepSeek V4 Flash. That's a 22x delta. Quality delta? About 0.7 points on my rubric, or 8%. Statistically, depending on your tolerance, that's not nothing. But it's also not a Ferrari-vs-Toyota situation.
If I were CFO of an engineering org I'd route 80% of calls through DeepSeek V4 Flash and reserve R1 for genuinely hard problems. That's a $720 annual saving on a 10-person team while keeping quality within 0.4 points of the best.
How I Actually Wire This Up
Here's the production snippet I've been running my tests against. Global API gives you a single endpoint that hits every model in this benchmark, which means I didn't have to manage ten different SDKs. Genuinely a quality-of-life improvement.
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["GLOBAL_API_KEY"],
base_url="https://global-apis.com/v1"
)
def review_code(model: str, code: str, language: str) -> str:
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": f"You are a senior {language} engineer."},
{"role": "user", "content": f"Review this code for security and performance:\n\n{code}"}
],
temperature=0.2,
max_tokens=1500
)
return response.choices[0].message.content
def smart_review(code: str, language: str, difficulty: str) -> str:
if difficulty == "hard":
return review_code("deepseek-r1", code, language)
return review_code("deepseek-v4-flash", code, language)
That last function is exactly the routing logic I'd ship. Cheap model for the 90% case, expensive model for the 10% that matters.
One More Table Because I Can't Help Myself
I bucketized by tier and computed mean score. Standard deviation included for the skeptics.
| Tier | Mean Score | Std Dev | Models |
|---|---|---|---|
| Budget ($0.20–$0.35) | 8.6 | 0.21 | 5 |
| Mid ($0.50–$1.00) | 8.3 | 0.85 | 2 |
| Premium ($1.90–$3.00) | 8.8 | 0.59 | 3 |
The mid tier is the worst of both worlds — wider variance, no price advantage. The premium tier has higher variance too, but the upside is real (Kimi K2.5 hit 9.2 on the Express task). The budget tier is the most consistent. Mean of 8.6 with a 0.21 std dev is remarkable for $0.25/M.
My Actual Recommendation
If you forced me to pick one model for a coding startup:
- Default: DeepSeek V4 Flash. Best ratio of score to dollar. Reliable. Cheap.
- Code review and security: DeepSeek-R1. The reasoning premium is real here.
- Bulk refactoring or boilerplate: Qwen3-Coder-30B. Code-specialized training shows.
- Don't bother: Hunyuan-Turbo at $0.57/M with a 7.5 score is
Top comments (0)