The two big open releases on everyone's feed right now are Kimi K3 (2.8 trillion parameters, the first open 3T-class model) and Ling 3.0 Flash from Ant Group's inclusionAI. Only one of them can live on hardware a person owns, and it is not the one with the bigger headline. Here is the practical picture for running Ling 3.0 Flash locally, with measured file sizes instead of projections, plus the honest math on why K3 stays in the cloud.
The shape of the model decides everything
Ling 3.0 Flash is a 124B parameter Mixture of Experts that activates only 5.1B parameters per token (8 of 512 routed experts plus one shared). As with every MoE, total parameters set your memory bill and active parameters set your speed. 124B of knowledge at the memory counter, 5.1B of compute at the speed counter: that combination is why this model runs interactively on machines that would crawl under a dense 70B.
Two more architecture notes that matter in practice. It uses hybrid linear attention (35 Kimi Delta Attention layers alternating with 7 gated MLA layers), so long contexts grow memory gently instead of quadratically. And it is a native hybrid reasoner: it thinks before answering by default, and the thinking pass can be switched off per request, so you decide when to pay for reasoning.
The license is plain MIT. Weights are on Hugging Face under inclusionAI/Ling-3.0-flash, including official fp8, fp4 and int4 variants.
The numbers
Community GGUF conversions are up, and these sizes are measured from the repos, not estimated:
| Quant | Size on disk | Realistic minimum |
|---|---|---|
| IQ1_S / IQ1_M | 27 to 30 GB | 36 GB RAM, visible quality cost |
| IQ2_M / Q2_K_XL | 42 to 43 GB | 48 to 64 GB |
| IQ3_XXS | 51 GB | 64 GB |
| Q4_K_M (sweet spot) | 78 GB | 96 GB |
| Q5_K_M | 92 GB | 128 GB |
| Q6_K | 105 GB | 128 GB |
| Q8_0 | 136 GB | 192 GB |
Realistic machine classes:
- A Mac with 96 or 128 GB unified memory runs Q4_K_M comfortably, and 64 GB machines run the 2-bit and small 3-bit quants whole.
- A 24 GB GPU plus 64 to 96 GB of DDR5 works well with MoE offload: attention layers and the shared expert on the GPU, routed experts in system RAM. Because only 5.1B parameters fire per token, generation speed stays in usable double digits.
- 32 GB or less: skip it and run a model that actually fits. The IQ1 files exist, but you will not enjoy them.
With a current llama.cpp:
llama-server -m Ling-3.0-flash-Q4_K_M.gguf \
--ctx-size 32768 \
--n-gpu-layers 99 \
--n-cpu-moe 40
Start with a modest context window; the model was trained out to 256K, but every token of window is memory you could spend on a better quant instead.
The no-terminal route
If you would rather click than type flags, Locally Uncensored (open source, AGPL) wraps the whole thing: install, open the Model Manager, paste a Ling 3.0 Flash GGUF repo, pick the quant that fits your memory, chat. It handles the llama.cpp engine, the offload split, and keeps everything on your machine with no account and no telemetry.
And the Kimi K3 reality check
K3 deserves its headlines: first open 3T-class release, 1M token context, native image input, and it shares the Kimi Delta Attention lineage with Ling. But the smallest usable GGUF conversion, a 1-bit quant, is 466 GB on disk. The 2-bit is 861 GB. Q4 is 1.5 TB. Expert-pruned community builds squeeze toward the 512 GB server class, which is heroic and still not a gaming PC.
So the honest split for the price of one search query: run Ling 3.0 Flash at home, and reach K3 through a hosted API when a task genuinely needs the 1M window or vision. DeepInfra serves both (Ling at $0.03 in / $0.07 out per million tokens, which rounds to free; K3 at $2.85 / $14.25), OpenRouter lists a free Ling tier, and both landed in LU Labs Cloud on every plan this week, with reasoning as a toggle in both cases.
FAQ
Is it actually good? inclusionAI benchmarks it at parity with their previous 1T-class flagship on SWE-Bench Pro, agentic tool suites and long-context tasks. Vendor numbers, as always, but the architecture math is real, and the price of testing that claim yourself is a weekend download.
How fast is it locally? Speed tracks the 5.1B active count, not the 124B total. On Apple Silicon at Q4 and on GPU-plus-RAM hybrids, expect double-digit tokens per second.
Uncensored builds? None yet. The MIT license makes finetunes and abliterations fully legal, and a 5.1B-active model is cheap to tune, so expect community variants. The stock model carries standard alignment.
Top comments (0)