DEV Community

Cover image for KAT-Coder V2.5 Local Setup Guide: GGUF, vLLM, SGLang
Joske Vermeulen
Joske Vermeulen

Posted on • Originally published at aimadetools.com

KAT-Coder V2.5 Local Setup Guide: GGUF, vLLM, SGLang

#katcoder #localai #coding #gguf

KAT-Coder V2.5-Dev is an open-weight coding model from Kwaipilot (Kuaishou's AI team) that scores 69.40% on SWE-bench Verified while activating only 3 billion parameters per token. The full model has 35 billion total parameters in a sparse Mixture-of-Experts architecture, but the MoE routing means only 3B are active for any given inference step.

This matters for local deployment: the model's computational requirements during inference are closer to a 3B dense model than a 35B one, though you still need enough memory to hold all 35B weights. With GGUF quantization, that becomes manageable on consumer hardware.

The model is Apache 2.0 licensed, text-only (vision weights are not included in the open release), and designed specifically for agentic coding tasks: working inside repositories, using tools, and fixing real bugs across multi-file projects.

What is KAT-Coder V2.5-Dev?

Spec Value
Creator Kwaipilot (KwaiKAT Team, Kuaishou Technology)
Base model Qwen3.6-35B-A3B
Architecture Sparse MoE, 256 routed experts (top-8 + shared expert)
Total parameters 35B
Active parameters 3B per token
Attention 30/40 layers use GatedDeltaNet linear attention
Context window 262,144 tokens (native), extensible via YaRN
License Apache 2.0
Modality Text-only (vision weights not released)
Thinking mode Default on (can be disabled)
Technical report arxiv.org/abs/2607.05471

Kuaishou is a major Chinese technology company (short-video platform, publicly traded). Kwaipilot is their internal AI coding team that has been developing the KAT-Coder series since 2025.

Benchmark context

Reported benchmarks from the official model card, reproduced in-house by Kwaipilot using standardized evaluation pipelines:

Benchmark KAT-Coder V2.5-Dev Qwen3.5-27B Qwen3.6-35B-A3B (base) Ornith-1.0-35B
SWE-bench Verified 69.40% 68.60% 64.40% 55.80%
SWE-bench Multilingual 63.00% 57.67% 57.00% 51.67%
SWE-bench Pro 45.96% 42.13% 40.63% 34.47%
Terminal-Bench 2.1 41.02% 34.83% 42.76% 33.10%

Important caveats:

  • These scores are self-reported by Kwaipilot, not independently verified by a third party.
  • Evaluation used claude_code@2.1.195 as the agent harness with pass@1, temperature=1.0, 256K context.
  • The comparison models were also tested by Kwaipilot (not official numbers from those model providers).
  • Benchmarks measure agentic coding capability (working in repos with tools), not raw code completion. Real-world coding productivity does not map linearly to benchmark percentages.
  • Terminal-Bench 2.1 score (41.02%) is notably lower than frontier closed models (Sol: 88.8%, Opus 5: 86.7%).

The 69.40% SWE-bench Verified is strong for a model in this size class. It significantly outperforms its base model (Qwen3.6-35B-A3B at 64.40%), showing the value of Kwaipilot's post-training (SFT + RL on 127K examples).

Hardware requirements

The 35B total / 3B active architecture creates an unusual hardware profile: you need memory for 35B weights but compute for roughly 3B.

Full precision (BF16)

Setup VRAM GPUs Context
Single GPU ~70GB 1x A100 80GB or H100 Limited context
Tensor parallel (recommended) 8x GPUs 8x A100/H100 Full 262K context

Full precision requires serious hardware. The official documentation shows tensor parallel across 8 GPUs for production deployment.

GGUF quantized (consumer hardware)

Community quantizations from bartowski make consumer deployment realistic:

Quantization File size RAM needed Quality
Q8_0 ~37GB 40GB+ RAM Near-lossless
Q6_K ~28GB 32GB+ RAM Very high quality
Q5_K_M ~25GB 28GB+ RAM High quality
Q4_K_M ~20GB 24GB+ RAM Good quality
Q3_K_M ~16GB 20GB+ RAM Acceptable
IQ4_XS ~18GB 22GB+ RAM Good (imatrix)

Practical consumer hardware:

  • 24GB VRAM GPU (RTX 4090): Q4_K_M fits, with limited context window
  • 32GB unified memory (M2/M3/M4 Mac): Q5_K_M or Q6_K via MLX
  • 64GB RAM (CPU inference via llama.cpp): Q6_K or Q8_0, much slower

MLX on Apple Silicon

A 6-bit MLX quantization is available for Apple Silicon Macs. A Mac with 32GB+ unified memory can run this at reasonable speeds for code generation tasks.

How to run KAT-Coder V2.5-Dev locally

Method 1: llama.cpp / GGUF (consumer hardware)

The most accessible method for consumer GPUs and Macs.

# Download the GGUF (choose quantization based on your RAM)
# From: huggingface.co/bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF

# Using llama.cpp server
./llama-server \
  -m KAT-Coder-V2.5-Dev-Q4_K_M.gguf \
  --port 8080 \
  -ngl 99 \
  -c 32768

# The model is now accessible via OpenAI-compatible API at localhost:8080
Enter fullscreen mode Exit fullscreen mode

For Macs with Metal acceleration, add -ngl 99 to offload all layers to GPU. Adjust -c (context length) based on available memory.

Method 2: vLLM (GPU server)

For multi-GPU servers with full-precision deployment. Requires the --language-model-only flag because vision weights are not included.

pip install vllm

vllm serve Kwaipilot/KAT-Coder-V2.5-Dev \
  --port 8000 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --language-model-only
Enter fullscreen mode Exit fullscreen mode

For tool-use support (agentic coding):

vllm serve Kwaipilot/KAT-Coder-V2.5-Dev \
  --port 8000 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --language-model-only
Enter fullscreen mode Exit fullscreen mode

Important: vLLM 0.19.0+ is required. The --language-model-only flag is mandatory or startup fails.

Method 3: SGLang 

pip install sglang[all]

python -m sglang.launch_server \
  --model-path Kwaipilot/KAT-Coder-V2.5-Dev \
  --port 8000 \
  --tp-size 8 \
  --mem-fraction-static 0.8 \
  --context-length 262144 \
  --reasoning-parser qwen3
Enter fullscreen mode Exit fullscreen mode

SGLang 0.5.10+ is recommended.

Method 4: KTransformers (CPU-GPU heterogeneous)

KTransformers enables running MoE models with CPU-GPU split, keeping active experts on GPU and inactive ones in RAM. This can make the 35B model practical on a single 24GB GPU by offloading dormant experts to system RAM.

See the KTransformers deployment guide for KAT-Coder-specific instructions.

Method 5: MLX on Apple Silicon 

# Using the community MLX quantization
pip install mlx-lm

mlx_lm.server \
  --model leonsarmiento/KAT-Coder-V2.5-Dev-6bit-XL-mlx \
  --port 8080
Enter fullscreen mode Exit fullscreen mode

Requires a Mac with 32GB+ unified memory for the 6-bit quantization.

Thinking mode

KAT-Coder V2.5-Dev thinks by default before responding (chain-of-thought reasoning). This improves coding quality but increases token usage and latency.

To disable thinking for faster, shorter responses:

chat_response = client.chat.completions.create(
    model="Kwaipilot/KAT-Coder-V2.5-Dev",
    messages=messages,
    max_tokens=32768,
    temperature=0.7,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": False},
    },
)
Enter fullscreen mode Exit fullscreen mode

For agentic coding (multi-turn, tool-use), enable preserve_thinking to maintain reasoning context across turns, which can reduce total token consumption by avoiding redundant reasoning:

extra_body={
    "chat_template_kwargs": {"preserve_thinking": True},
}
Enter fullscreen mode Exit fullscreen mode

Compared with other local coding models

A practical comparison based on local deployment characteristics, not an exhaustive benchmark table:

Model Total params Active params Quantized size (Q4) Min hardware Coding focus
KAT-Coder V2.5-Dev 35B 3B ~20GB 24GB GPU or 32GB Mac Agentic repo-level coding
Qwen3-Coder-30B 30B 30B (dense) ~17GB 24GB GPU General coding
DeepSeek-Coder-V2-Lite 14B active (236B MoE) 14B Large Multi-GPU General coding
Devstral 2 (24B) 24B 24B (dense) ~14GB 16GB GPU Agentic coding

KAT-Coder V2.5-Dev's advantage: the 3B active parameter count means faster inference per token than dense models at similar quality, while the 35B total gives it more "knowledge" capacity than a true 3B model. The tradeoff is memory: you store 35B weights to get 3B-speed inference.

Limitations

Text-only: this release does not include vision/multimodal weights. It cannot process images, screenshots, or visual inputs. Only text-based coding.

Memory vs compute mismatch: despite only activating 3B params, you need storage for all 35B. A Q4 quantization is ~20GB, not 3GB. The MoE architecture provides inference speed benefits, not memory savings.

Benchmark context: the 69.40% SWE-bench score uses a specific evaluation harness (claude_code@2.1.195, 256K context). Results with different agent frameworks, shorter contexts, or different prompting strategies may differ.

Not for short completions: this model is trained for agentic, multi-turn coding (working in repositories with tools). For simple autocomplete or single-function generation, smaller dense models may be more practical and faster.

Context window on consumer hardware: the full 262K context requires multi-GPU setup. On a single consumer GPU with quantized weights, practical context is limited to 8K-32K tokens depending on available VRAM.

FAQ

Can I run KAT-Coder V2.5-Dev on a single consumer GPU?

Yes, with quantization. A Q4_K_M GGUF (~20GB) fits on an RTX 4090 (24GB VRAM). Context window will be limited. For full 262K context, you need multi-GPU.

Is this better than DeepSeek V4 Pro for coding?

Different use cases. DeepSeek V4 Pro scores 80.6% SWE-bench Verified but requires serious GPU infrastructure to self-host (or API access at $2.19/$8.76). KAT-Coder V2.5-Dev scores 69.40% but runs on consumer hardware with GGUF quantization. Choose based on whether you need maximum capability (DeepSeek) or local deployment (KAT-Coder).

Does it work with Ollama?

Not officially supported. The model is not in the Ollama library natively. However, you can convert GGUF files for use with Ollama-compatible tools or use llama.cpp directly (which Ollama is built on).

Why does vLLM need the --language-model-only flag?

The model architecture declares multimodal/vision components in its configuration, but the open-weight release ships only language model weights. Without the flag, vLLM tries to initialize vision encoder weights that do not exist and fails.

What is the practical speed on consumer hardware?

With a Q4 GGUF on an RTX 4090: expect roughly 30-50 tokens/second for generation. On Apple Silicon (M3 Max, 64GB) with MLX: expect 15-30 tokens/second. These are approximate and depend on context length and quantization quality.

Is the thinking mode worth the extra tokens?

For complex multi-file coding tasks, yes. Thinking mode lets the model reason through file relationships and potential side effects before writing code. For simple, single-function tasks, disable thinking to save tokens and latency.

Sources

Related Articles

- Edge AI vs Cloud API Cost Calculator

Originally published at https://www.aimadetools.com

Top comments (0)