Moksh Gupta

Posted on Sep 9 Originally published at devtoollab.com

Five Open Weight Coding Models, and the Benchmark Problem Nobody Mentions

#ai #programming #opensource #machinelearning

Downloadable models caught up faster than the comparison tooling did. You can now pull frontier-class weights off Hugging Face and serve them on your own GPUs, which means your source code never leaves your infrastructure. What you cannot easily do is find out how much capability that costs you versus a closed model, because the labs on either side of the fence do not report the same benchmarks.

Here is the shape of it. Anthropic's Claude Fable 5.1 publishes Terminal-Bench 4.0 at 55.8% and no SWE-bench Verified figure at all. Moonshot's Kimi K3, the biggest open weights available today, publishes Terminal-Bench 2.1 at 88.3 and also skips SWE-bench Verified. Those two numbers are not comparable, and the "Fable scores 95% on SWE-bench Verified" line circulating in search results traces back to third-party leaderboards rather than to Anthropic.

I dug through eight model cards to sort this out. The longer version on DevToolLab has the full per-model breakdown and screenshots of every card as it read in September 2026.

What Each Lab Publishes

Every figure below is lifted from the vendor's own card, except the Fable 5.1 row, which is measured independently. Read down a column, never across one.

Model	Params (active)	License	Context	SWE-bench Verified	Terminal-Bench 2.1
Claude Fable 5.1 (reference)	not disclosed	proprietary	1M	not published	85.02% (Vals AI)
Kimi K3	2.8T (104B)	Kimi K3, revenue-gated	1M	not published	88.3
DeepSeek V4 Pro	1.6T (49B)	MIT	1M	80.6%	not published
GLM-5.3	753B	glm-5.3 bespoke	1M in evals	not published	88.2
MiniMax M2.5	229B (~10B)	Modified-MIT	not stated	80.2%	not published
Qwen3.8-27B	27B dense	Apache 2.0	262K	not published	73.0

Why the Numbers Do Not Line Up

Anthropic's card for Fable 5.1 covers Terminal-Bench 4.0 (55.8%), Terminal-Bench-Science 0.1 (52.6%), CursorBench 3.2.0 (73.4%) and AutomationBench (31.4%). It skips SWE-bench Verified, SWE-bench Pro and LiveCodeBench entirely, and it benchmarks only against other closed models: Claude Opus 5, Claude Fable 5 and GPT-5.6 Sol. Not one open-weight model appears in its tables. The open labs do close to the reverse, leaning on SWE-bench Verified and Pro, DeepSWE, and Terminal-Bench at older versions.

The version detail is the trap, and it is worth internalizing before you read any leaderboard. Terminal-Bench is a family of benchmarks, and difficulty moves sharply between releases. GLM-5.3's own card lists 88.2 on version 2.1 and 28.3 on version 3.0. Fable 5.1 sits at 55.8% on version 4.0. Line those up as a ranking and GLM-5.3 simultaneously wins by 32 points and loses by 27, which tells you the comparison is meaningless rather than telling you anything about the models.

The One Comparison Run on a Single Harness

Z.ai did something the other labs did not: its GLM-5.3 card runs Claude Fable 5, GPT-5.6 Sol, Kimi K3 and its own models through six benchmarks and publishes the chart. Terminal Bench 3.0 comes out Fable 5 at 33.7, GLM-5.3 at 28.3, Kimi K3 at 17.4. DeepSWE lands much closer, with Fable 5 at 69.7, Kimi K3 at 67.5 and GLM-5.3 at 66.9. On Agents' Last Exam (CLI) the ordering flips: GPT-5.6 Sol 28.6, GLM-5.3 28.5, Kimi K3 27.6, and Fable 5 last at 23.8.

Two things stop that being conclusive. Z.ai ran the evaluation and has a stake in how it reads, and the Anthropic model in the chart is Fable 5 rather than 5.1. Anthropic reports 5.1 as a substantial jump on exactly this kind of task, moving Terminal-Bench 4.0 from 42.0% to 55.8% and Terminal-Bench-Science 0.1 from 24.7% to 52.6%.

For a neutral reading, Vals AI measures Fable 5.1 at 85.02% on Terminal-Bench 2.1, second of 63 models behind GPT-5.6 Sol at 85.77%, and 90.52% on LiveCodeBench, first of 143. Put that 85.02% beside the 88.2 and 88.3 self-reported by Z.ai and Moonshot on the same benchmark version and the gap on agentic terminal work looks narrow. Keep hold of the asymmetry, though: one figure was measured by a third party, the other two were reported by the vendors selling the models.

Kimi K3: Frontier Scale, Datacenter Requirements

Kimi K3 is a mixture-of-experts model at 2.8T total parameters, 104B active per token, with native vision and a 1,048,576-token window. Moonshot released the weights in July 2026 under a bespoke license that is open weight without being OSI open source: commercial inference is gated above a revenue threshold, and products past a certain size have to show "Kimi K3" somewhere in their interface.

Its coding numbers are DeepSWE 67.5, FrontierSWE 81.2, ProgramBench 77.8, Terminal-Bench 2.1 88.3 and Kimi Code Bench 2.0 72.9. One near-overlap is worth noting: its AutomationBench result of 30.8 lands within a point of the 31.4% Anthropic reports, although neither lab pins down a matching harness version. Scale is the catch. Nothing about 2.8T parameters is workstation-friendly, so rent this from a provider unless you operate a multi-node cluster.

DeepSeek V4 Pro: The License Everyone Else Should Copy

Both DeepSeek V4 variants carry a plain MIT license, which makes this the least encumbered option here by a wide margin: no revenue trigger, no attribution clause, no security review to clear. V4-Pro runs 1.6T parameters with 49B active, V4-Flash 284B with 13B active, and both take a million tokens of context.

It is also the only model in this group that publishes the benchmark developers actually search for. The card lists SWE-bench Verified at 80.6%, SWE-bench Pro at 55.4%, SWE-bench Multilingual at 76.2%, Terminal Bench 2.0 at 67.9%, LiveCodeBench Pass@1 at 93.5 and a Codeforces rating of 3206. Two caveats: the card describes the series as a preview, and that LiveCodeBench 93.5 is self-reported while Fable 5.1's 90.52% came from Vals, so the ordering between them is suggestive rather than settled.

GLM-5.3 and MiniMax M2.5: Agentic Focus and Cost Per Task

GLM-5.3 is 753B parameters, and Z.ai is unusually candid that it shares a base model with GLM-5.2 with every gain coming from post-training. The card reports Terminal Bench 2.1 at 88.2, Terminal Bench 3.0 at 28.3, DeepSWE (v1.1) 66.9, FrontierSWE 78.1, SWE-Marathon (v1.1) 42.5 and AutomationBench (v1.0.6) 48.2. Z.ai withheld the weights for roughly two weeks after the API launch to finish a safety review, which no other open-weight lab here bothered to do. The license is bespoke rather than MIT or Apache, and the largest model-as-a-service providers need Z.ai's sign-off before commercial use.

MiniMax M2.5 is the one to look at if throughput cost drives the decision: 229B total parameters but only around 10B active per token, under Modified-MIT, reporting SWE-bench Verified 80.2%, SWE-bench Pro 55.4% and Multi-SWE-Bench 51.3%. The revealing part is the comparison set MiniMax picked for its own chart, placing that 80.2% next to Claude Opus 4.5 at 80.9% and Opus 4.6 at 80.8%. That is the honest summary of where open weights sit on single-issue bug fixing: level with last-generation Opus, not with Fable. MiniMax suggests SGLang for serving, with vLLM as the fallback.

Qwen3.8-27B: The One That Fits on a Desk

Everything above assumes real infrastructure. Qwen3.8-27B does not, and it carries a plain Apache 2.0 license, the only one in this group. It is a dense 27B-class vision-language model with 262,144 tokens of native context, shipped in August 2026, and it loads in Transformers, vLLM, SGLang and TokenSpeed without special handling.

The card reports Terminal-Bench 2.1 at 73.0, up from 63.4 in the prior generation, plus SWE-bench Pro 61.7, DeepSWE 1.1 42.2, OSWorld-Verified 84.3 and LiveCodeBench v6 90.3. That SWE-bench Pro number is the surprise, sitting above the 55.4 reported by both DeepSeek V4 Pro and MiniMax M2.5 from a model orders of magnitude smaller. Harness differences could explain some of it, but it still makes a dense 27B a credible local default. Quantized to 4-bit it needs roughly 14 to 16 GB of VRAM before the KV cache, so a single 24 GB card handles it.

What Each Route Actually Costs

Fable 5.1 runs $10.00 per million input tokens and $50.00 per million output, with cache reads at $0.25 per million. There is no hardware bill, but it does require 30-day data retention and is unavailable under zero data retention without explicit authorization from Anthropic.

Open weights flip that: nothing to pay for the license, nothing leaving your network, and a GPU bill instead. That bill tracks active parameters rather than total, which is why MiniMax M2.5 at roughly 10B active and Qwen3.8-27B at 27B dense are the two realistic self-host picks, while Kimi K3 at 104B active and DeepSeek V4 Pro at 49B active are cluster commitments most teams are better off renting. Before assuming self-hosting wins, price the same workload both ways with the LLM Token Cost Calculator, and check what your repository actually consumes with the AI Token Counter so a 262K context ceiling is a measurement rather than a guess. The original post works through the decision procedure in more detail.

Picking One

Read the license against your revenue rather than your preferences: MIT and Apache 2.0 come with no strings, while the Kimi K3 and glm-5.3 licenses both have revenue triggers and Kimi adds a UI attribution requirement. Match the benchmark version to the work you actually do, comparing Terminal-Bench at one fixed version for agentic tasks and leaning on SWE-bench Verified for single-issue fixes, where DeepSeek V4 Pro's 80.6% leads the downloadable field. Then size the model to the hardware you already own: a 24 GB card means Qwen3.8-27B, one 8-GPU node opens up GLM-5.3 or MiniMax M2.5, and anything beyond that means renting.

The short version for September 2026 is that open weights have pulled level with last-generation Opus on single-issue bug fixing and still trail on long-horizon agentic work, which is precisely what Fable 5.1 was tuned for. Whichever you choose, every number on this page is a public-task harness result. None of them is your codebase, so run your own eval before committing.

DEV Community