KAT-Coder V2.5: The 35B Model That Rewrote the Open-Source Coding Leaderboard
Why the headline score is misleading — and what actually matters for local AI in 2026
On July 23, 2026, Kuaishou — the Chinese short-video giant with 408 million daily users — quietly uploaded a 35-billion-parameter coding model to Hugging Face. Within hours, the community had quantized it to 21GB. Within days, developers were running it on RTX 3060 cards.
The model card claimed 69.4 on SWE-bench Verified. That number put it above every open model in its weight class, including the Qwen base it was built from. But the real story is not the score. It is the scoreboard.
The 21GB vs 1.5TB "Open" Model War
Three days after KAT-Coder V2.5 went live, Moonshot released Kimi K3 — 1.5 terabytes of weights across 96 shards. You need a rack of data-center GPUs to load it. Both were called "open." Both meant completely different things.
KAT-Coder V2.5 fits on a single graphics card. Kimi K3 does not. One of those is actually runnable by the people the license is supposed to serve.
The Architecture Is Not the Story
KAT-Coder V2.5 is not trained from scratch. It starts from Alibaba’s Qwen3.6-35B-A3B — the same base thousands already run at home — then layers on 127,000 supervised examples before reinforcement learning even begins. The architecture is public. Anyone can download it.
The story is what they trained it against.
KuaiPilot’s training pipeline requires three things for every task: a precise description of what needs fixing, a repository that actually builds and runs, and a set of tests that decide whether the fix works. A patch counts only if it passes all three. Assembling 100,000 such environments is an infrastructure problem because most repositories on the internet do not build cleanly on a stranger’s machine.
So they built AutoBuilder — two agents in a loop. One writes install and test scripts. The other runs them in a sandbox and reports what broke. At the start, AutoBuilder could reconstruct 16.5% of repositories. By the end, it was reconstructing 57.2%. That pushed them past 100,000 verifiable environments across 12 languages: Python, JavaScript, TypeScript, Java, Go, C++, Rust, and more.
The Benchmark Lie: SWE-bench vs Terminal-Bench
Here is the part most blog posts skip. Around 16% of the runs, the model was being punished for something the sandbox did wrong, not the model. Wrong reward, wrong lesson — and the model learns to avoid mistakes it never made. KuaiPilot pushed that error rate below 2%.
A broken grader makes more training actively harmful.
They made the same fix in a second place. During RL, the tokens a model produces while exploring and the tokens it later trains on can drift apart. KuaiPilot forced them to be identical, so the lesson matches the attempt it came from.
After 10 RL epochs with explicit penalties for pathologies — 70+ tool calls in one turn, empty tool blocks, self-repetition — mislabelled tool calls dropped from 9.34% to 0.28%. Single-turn repetition went to zero.
Then came the scorecard: 69.4 on SWE-bench Verified, 63 on the multilingual set, 45.96 on SWE-bench Pro.
The Harness Matters More Than the Model
A user named Spacebike opened discussion #12 on the model page and pasted a second column of numbers. Alibaba’s own published figure for the exact Qwen base is not 64.4 — it is 73.4. DeepReinforce’s own figure for Ornith 1.0 35B is not 55.8 — it is 75.6.
Neither side is fabricating anything. They are running different harnesses. Teams tune for their own evaluation setups.
KuaiPilot reran every model themselves under one configuration and published the exact agent, temperature, and context settings in the footnotes. They also admitted what most labs leave out: with that many candidate checkpoints, five repeated runs per benchmark was not affordable. So every number on the chart is a single pass.
Their defense is that seven benchmarks moving together is hard to fake with sampling luck. The proof is on their own model card. Terminal-Bench gives this model 41.02 — an average. Under one agent harness, the same model scores 32.6. Under a different one, 49.4.
Same weights. Same tasks. 17 points apart.
Terminal-Bench 2.1 exists because 28 of the 89 tasks in version two were broken. That is roughly where open-model benchmarking sits in 2026. Every score you read is a number plus a harness — and the harness is usually the part cropped out of the screenshot.
What Actually Happens When You Run It
The chart is a claim, not evidence. The thread on Hugging Face has real answers.
One tester ran it at a 1-million-token working context and reported that the yarn scaling held all the way out, while models he compared it against lost the plot past 32K–64K and their tool calling fell apart.
Another ran a quantized build on an RTX 3060 against a live project of 100+ files and 50,000 lines and put it roughly level with a 27-billion-parameter model on a card that costs less than a phone.
The observation that repeats most often is about thinking tokens. It spends fewer of them than the base it was built from, and the traces read like reasoning instead of rambling. On a model you pay for by the token, that is most of the value right there.
It runs through Transformers, vLLM, SGLang, KTransformers, or llama.cpp. One trap in the docs: the open release is text-only, so vLLM needs a language-model-only flag, or it tries to start a vision tower that is not there.
The Ceiling Is Real
Terminal-Bench 2.1 gives it 41.02. The best systems on that board sit near 89.5. On the Open Weight Leaderboard as of August 2, Ornith 1.0 396B sits at 82.4, DeepSeek V4 Pro at 80.6, MiniMax M3 at 80.5, Kimi K2 at 80.2, Claude Opus 5 at 96.
KAT-Coder V2.5 is 27 points behind the frontier and 13 behind the strongest open weights anyone has published. What it does beat is every open model you can put on hardware you already own. That is a smaller claim than the one on the chart — and a considerably more useful one.
The download numbers suggest people work that out for themselves. The official repository has moved about 13,000 copies in a month. One community 4-bit conversion has moved 96,000 — seven times the traffic to the version that fits on a graphics card.
If you would rather rent than run, the close siblings sit on OpenRouter. Air costs 15 cents in and 60 cents out per million tokens. Pro costs 74 cents in and 296 cents out.
The Part Worth Keeping
The headline score is the least interesting thing about this release.
A team went looking for why their model kept getting punished, found their own graders at fault in one run out of six, and repaired those before they touched the model. That is the difference between scaling compute and scaling correctness.
In a single week, "open" came to mean 21 gigabytes on your desk, or a terabyte and a half in somebody else’s rack. Only one of those is actually yours.
Shakti Tiwari
NIFTY Options Trader | optiontradingwithai.in
Building systematic trading + AI systems that actually work
Tags: #ArtificialIntelligence #OpenSource #LLM #MachineLearning #CodingAgents #LocalAI #Qwen #KAT-Coder #SoftwareEngineering #Benchmarking
Top comments (0)