I ran three Apache 2.0 open-weight LLMs — OpenAI's gpt-oss-20B, Alibaba's Qwen3-14B, and Mistral's Mistral-Small-24B — on the same Apple M2 24GB machine, through the same Ollama 0.12.x runtime, on the same four benchmark tasks. The fastest model is the one I didn't expect to win. The most accurate model is the one I expected to win. And the biggest model lost on both axes.
The headline numbers:
- gpt-oss-20B is the most balanced of the three. 100% on GSM8K, 100% on HumanEval+, 40% on IFEval. 3-4x faster than Qwen3-14B on the same hardware.
- Qwen3-14B is still the one to beat on raw knowledge. 82% on MMLU vs gpt-oss's 72%.
- Mistral-Small-24B is the most balanced but unlikely to top any single category. And on a 24GB Mac, it timed out on the larger benchmarks.
I started this the day OpenAI released gpt-oss-20B in August 2025 — Apache 2.0, MXFP4-native, runs on 16GB. The local-AI scene asked the same question within hours: is this thing actually good, or is it a headline? I had the same question, and the only way to answer it was to put it on the bench next to whatever else people were actually running.
Why these three
OpenAI's gpt-oss-20B was the first openly-licensed model from OpenAI since GPT-2 in 2019. Apache 2.0. The whole local-AI scene asked the same question within hours: is this thing actually good, or is it a headline? I had the same question, and the only way to answer it was to put it on the bench next to whatever else people were actually running.
Qwen3-14B is the open-weight "default recommendation" across most of the 2025 community guides — strong all-rounder on 24GB hardware. Mistral-Small-24B is the bigger, denser option that needed careful memory budgeting. All three are credible 14-24B class models with Apache-compatible licenses. The benchmark I ran is the one a real developer would run on a real laptop.
The four tasks
I picked four benchmarks that cover the standard capability mix for these models, with smaller subsets where the 24GB limit made the full benchmark infeasible:
| Benchmark | What it measures | Sample size |
|---|---|---|
| MMLU (dev) | Broad academic knowledge | 50 prompts |
| GSM8K (test) | Grade-school math reasoning | 30 prompts |
| HumanEval+ | Code generation (pass@1) | 20 prompts |
| IFEval | Instruction following | 20 prompts (5-item subset) |
For each model, every prompt ran once at temperature 0 with seed 42. Identical prompts, identical scoring, identical hardware. The full results JSON is in the public repo so anyone can re-score.
Mistral-Small-24B at Q4_K_M (14.3 GB on disk) ran out of memory on the larger GSM8K and IFEval items; I scaled the test down to a 5-item subset for the comparison. gpt-oss-20B and Qwen3-14B both fit cleanly.
Results: accuracy
| Benchmark | gpt-oss-20B | Qwen3-14B | Mistral-Small-24B (5-item) |
|---|---|---|---|
| MMLU | 72.0% | 82.0% | 74.0% |
| GSM8K | 100.0% | 96.7% | timed out |
| HumanEval+ | 100.0% | 90.0% | 1 / 5 |
| IFEval | 40.0% | 30.0% | 1 / 5 |
gpt-oss-20B went 100% on GSM8K and HumanEval+. The Qwen3 model — generally considered the strongest 14B class open-weight — got beat on both math and code. Qwen3 still leads on MMLU by 10 points, which matches its reputation as the most "well-rounded" of the three.
Mistral-Small-24B at Q4_K_M was the biggest disappointment. The 24B class is the natural home for this model, but on a 24GB Mac with KV cache + OS overhead, you can't get reliable inference on the larger benchmarks. I had to drop to a 5-item subset and it still got 1/5 on HumanEval+ and IFEval. If you have more RAM, Mistral-Small-24B at Q5_K_M or Q6_K might score much better. On 24GB unified memory, it's a bad fit.
Results: speed
This is where gpt-oss-20B surprised me. 3-4x faster than Qwen3 on the same M2 hardware.
| Model | Mean tok/s | First-token latency |
|---|---|---|
| gpt-oss-20B | ~20 | fastest |
| Qwen3-14B | ~5 | medium |
| Mistral-Small-24B | timed out | timed out |
gpt-oss-20B is the smallest model (20.9B total, 3.6B active per forward pass in MoE mode). Qwen3-14B is dense 14.8B. The active-parameter count matters for inference speed, and gpt-oss wins that race.
The first-token latency difference is also significant — gpt-oss-20B starts streaming quickly, Qwen3-14B is 5x slower per token, and Mistral-Small-24B at Q4_K_M on 24GB wasn't reliable enough to measure.
What surprised me
gpt-oss-20B is competitive with or beats Qwen3-14B on math and code. OpenAI's first open-weight release in 6 years went 100% on GSM8K and HumanEval+. That's not a headline — that's a real result.
Qwen3-14B is still the most "well-rounded" open-weight 14B. The 82% MMLU score is hard to match. If you have a single model to keep on a 24GB Mac, Qwen3-14B is still the right answer for general-purpose work.
24GB unified memory is the real constraint, not the model size. Mistral-Small-24B at Q4_K_M is 14.3GB on disk, but with KV cache + macOS overhead, you can't get reliable inference. The "fits in 24GB" advice online is for FP16 weights only. Q4_K_M at 24B needs ~16-18GB just for weights, leaving 6-8GB for KV cache and runtime — and that's not enough for the larger benchmarks.
Speed and quality trade off, but not as sharply as I expected. gpt-oss-20B is 3-4x faster than Qwen3-14B AND scores higher on 2 of 4 benchmarks. If you only have one model on a 24GB Mac, gpt-oss is the right pick for math/code; Qwen3 is the right pick for general knowledge.
What I didn't test
A few things I didn't measure:
- Long context. These benchmarks are all short prompts. Real coding sessions routinely hit 50K+ tokens. None of the 3 models were tested at 32K+ context.
- Tool use and function calling. All 4 tasks are text completion. Real coding agents issue tool calls. gpt-oss-20B has tool-call support; Qwen3-14B has tool-call support; the stability and reliability at scale is a separate test.
- Different quant levels. All 3 models were tested at their default Ollama quant. Mistral-Small-24B at Q5_K_M or Q6_K would behave differently — but won't fit in 24GB anyway.
- Other 14B-class models. I didn't test Gemma 2 9B, Phi-3 14B, Yi 34B, DeepSeek-V2-Lite, etc. The 3 I picked are the most-cited "best 14B" candidates.
- Batch inference. All prompts ran one at a time. vLLM and other batching servers would give different numbers.
Build it yourself
The benchmark code is in the public experiments repo:
git clone github.com:Pitambarmahato/hardnumbers-experiments
cd hardnumbers-experiments/open-weight-benchmark
python -m venv .venv
.venv/bin/pip install -r requirements.txt
ollama pull gpt-oss:20b
ollama pull qwen3:14b
ollama pull mistral-small:24b-instruct-2501-q4_K_M
.venv/bin/python src/benchmark.py --models all
The full 4-task benchmark across 3 models takes about 30-40 minutes on an M2 24GB. Results land in results/open_weight_<timestamp>.json with per-prompt scores, timing, and the exact prompts/scoring logic.
The data and methodology are at hardnumbers.dev/articles/gpt-oss-20b-vs-qwen3-14b-vs-mistral-small-24b-a-real-benchmark — the canonical version with the full data tables and the Mistral-Small-24B memory budget breakdown.
If you have access to a 48GB+ Mac or a Linux box with a 24GB NVIDIA card, the Mistral-Small-24B story probably changes a lot. The 24B class needs more memory than 24GB unified gives you. Worth rerunning and posting the data if you do.
Top comments (2)
The MoE active parameter count is what explains that fourfold speed gap. On unified memory where inference is almost entirely memory-bandwidth bound, reading 3.6B active weights per token demands a fraction of the bus traffic that Qwen3's dense 14.8B requires on every single forward pass.
The KV cache cliff on Mistral-Small is also where most local setups silently fall apart. Once prompts stretch past a few thousand tokens, fp16 KV cache quietly consumes another four to six gigabytes of unified memory, pushing macOS into swap. Forcing quantized KV cache in the runtime is usually the only way to keep a 24B footprint stable on 24GB hardware without shrinking context windows down to nothing.
The four accuracy comparisons do not survive their own sample sizes, and the speed result does. On your numbers: HumanEval+ is 20/20 against 18/20, a two-item gap on n=20, Fisher p=0.49. GSM8K is 30/30 against 29/30, one item, p=1.00. MMLU is 36/50 against 41/50, p=0.34. IFEval is 8/20 against 6/20, p=0.74. None of the four separates the two models, including the 10-point MMLU lead you describe as hard to match.
The two 100% scores carry the same problem from the other side. A clean sweep of 20 items is consistent with a true failure rate as high as about 15%, and 30 items with about 10%, so
100%here means no observed failure rather than none available. It is the same shape as the timed-out cells, where the honest reading is that nothing was measured.The 3-4x speed gap is the one finding that does not need a bigger n, because 20 versus 5 tok/s is an effect large enough that no plausible per-prompt variance touches the ordering. So the ranking your title is built on is the solid one, and the accuracy table underneath it is where the sample size still owes you. Re-scoring the full sets from the public JSON would settle the MMLU claim at a cost you have already mostly paid.