TL;DR: The biggest AI model ever released to the public came with a 1,500 line instruction manual. Almost none of the good parts are about how big it is. Here is that manual, with the equations taken out and analogies put in.
Twelve days ago I wrote about Kimi K3 on the day it was announced, and I finished that post with a question. Moonshot had promised to release the weights on July 27 along with a technical report. Would they, and what would the report say?
They did. The weights went up on Hugging Face on the 27th and were the most popular thing on the site within half an hour. The report came with them.
I have now read all of it. I have not run the model and I am not going to, because it needs a small server room, but the report turns out to be the more interesting artifact anyway.
Here is the thing I did not expect. The best section in it describes a problem Moonshot created for themselves in the first month of the project and did not discover until the model was in production. I will get to that one, but the setup takes a few steps.
No equations below. If you can follow a story about a restaurant kitchen, you can follow this.
What 2.8 trillion parameters actually means
Parameters are the model's adjustable settings. More of them means more room to store patterns. Kimi K3 has 2.8 trillion, which is the largest number anyone has published openly.
But it does not use them all at once, and that distinction is the whole game.
Picture a hospital with 896 specialists on staff. A patient walks in. They do not see all 896. A triage nurse reads the chart and sends them to 16. The hospital is enormous, but any single visit only involves a small team.
That is a Mixture of Experts, and Kimi K3 is one. 896 specialist sub-networks, 16 of them working on each word. The model is huge on disk and comparatively cheap to run, because 98% of it sits idle at any given moment.
Kimi K2, the previous version, had 384 specialists and used 8. So K3 more than doubled the staff and doubled the size of the team seeing each patient.
Which sounds straightforwardly better, and is where the problems start.
How Kimi K3 reads a million words at once
Kimi K3 can take a million tokens of input, roughly 700,000 words. An entire large codebase. Several novels.
Fitting that much text in was never the hard part. The hard part is that the standard way a model reads is quadratic: to write each new word, it re-reads everything before it. Imagine replying to an email by first re-reading every email you have ever received. Fine for twenty. Ruinous for a million.
The alternative is to keep a running summary instead. After each email, you update your notes and throw the email away. Your notes stay the same size forever, no matter how much arrives. Cheap. But you have lost the ability to go back and quote an old email exactly.
Kimi K3 does both, in a ratio. Three summary-keeping layers, then one full re-reading layer, over and over, 93 layers deep. The summary layers do the bulk of the work cheaply, and every fourth layer gets to look at the raw material properly.
The summary-keeping mechanism is what Moonshot call Kimi Delta Attention, and it comes with a bonus that sounds small and is not.
Most models need to be told explicitly where each word sits in the sequence. Word 3, word 40,000, word 900,000. Stretching a model from 128,000 words to a million means recalibrating that ruler, which is fiddly and lossy.
Kimi K3 has no ruler. The running summary naturally fades older information, so recency is built into the mechanism rather than stamped on top. There is nothing to recalibrate. The model stretches to a million tokens because there is no measuring device to break.
Why 896 experts and only 16 do the work
Back to the hospital, where there is a courier problem.
Normally every specialist you consult receives the patient's complete file. Send it to 8 and you have 8 copies moving around the building. Send it to 16 and you have doubled the courier traffic, the printing, and the delays. This is why models do not simply crank up the number of active experts. The bandwidth bill follows.
Kimi K3 sends a summary instead. Specialists get a compressed version, half the width. Only two general practitioners, who see every patient regardless, get the full file. So the routing can be far more generous without the courier traffic following it.
Now the subtler problem. Who decides which 16?
A router scores every specialist for every word and takes the top ones. Left alone this fails predictably: a few popular specialists get swamped, most idle, and some never see a patient at all and so never learn anything. Worse, when experts live on different machines, one overloaded machine makes every other machine wait.
The old fix was a nudge. Each round, bump the score of underused experts and dock the overused ones by a fixed amount. It is a thermostat, and thermostats have a known failure mode. Nudge too gently and the room never warms. Nudge too hard and it overshoots, then overshoots back, forever. With 896 rooms, no single setting works for all of them.
Moonshot threw out the thermostat. Their replacement is a bouncer who has stopped guessing the dress code: sort everyone by score, count down to exactly the number the venue holds, put the bar right there. One pass, no guessing, the target load every time.
There is a lovely practical wrinkle. To sort everyone you would need to gather every score in one place, and at training scale that is millions of numbers scattered across hundreds of machines. Gathering them is not affordable.
So they do not gather them. Each machine reports how many of its scores fell into each band, like a census returning "4,200 people between 170 and 175cm" rather than a list of names. Add the bands together and you have the national picture. Counts add up cleanly no matter how you split the population, which is why the answer is exact regardless of how the data happened to be distributed.
If you ever need a global statistic across shards and cannot afford to collect the raw data, that trick is worth remembering.
Keeping the numbers small enough to fit
Here is a pattern that shows up three separate times in the report, and once you notice it you cannot stop.
Computers store numbers with limited precision, and AI hardware deliberately uses very coarse precision because it is faster. So a calculation that produces an enormous number does not merely get slow. It breaks.
Kimi K3's summary mechanism has exactly this hazard. Old information fades a little at every step, and doing the maths correctly sometimes means undoing the fade, which means dividing by it. Fade something to almost nothing and you are dividing by almost nothing. It is the audio equivalent of reconstructing a whisper by amplifying it a trillion times. The amplifier does not cope.
The previous version handled this by detecting the dangerous cases and sending them down a slow, careful, special-purpose path.
Kimi K3 changed one setting instead. Nothing is allowed to fade below roughly 0.7% of its original strength, ever. That floor keeps the undo number inside what the hardware can hold, and with the hazard gone, the slow path was not optimised. It was deleted.
The same instinct shows up in the activation function, which had two parts that could each grow without limit. Multiply two unbounded things and eventually both spike together and blow the speakers. Kimi K3 put a limiter on both, shaped to behave identically at normal volumes and simply refuse to go past a ceiling. And again in how it was trained: most models are trained at full precision then compressed afterwards, losing a little quality. Kimi K3 trained in the compressed format from the start, so it never had to adapt to a haircut it did not see coming.
Three subsystems, one idea. At this scale, the shape of a curve is an infrastructure decision.
The proof that removed three bottlenecks
This is my favourite thing in the report, and it is about a kitchen.
Imagine 896 dishes on a menu, and a kitchen split into stations. Orders arrive unevenly, so some stations get slammed while others idle. The usual fix is floating cooks who can be sent wherever the pressure is. But how many floaters do you keep on shift?
Guess too few and one night the kitchen jams with no legal way to redistribute the work, and service stops. That is not a metaphor: in existing systems, training genuinely halts, and someone has to retune a number by hand.
Moonshot proved a floater count that mathematically cannot fail, for any distribution of orders. They also proved you cannot do better. So they staff exactly that many, and the jam becomes impossible.
Now watch what a guarantee buys you, which is the part I keep thinking about.
Because the load is always perfectly even, every station does an identical amount of work. Because the work is identical, the size of every task is known before the shift starts. Because the sizes are known in advance, nobody has to stop and ask each station "how many orders do you have right now?" before sending the next batch.
That question was itself a major delay. It happened between every single layer of the model, hundreds of times per step, and each time the whole pipeline stalled waiting for an answer.
One proof. Three separate slowdowns gone, two of them nowhere near the thing being proved.
The problem that showed up eight months late
Right. The section I promised.
Kimi K3 mixes two kinds of layer, and each remembers conversations differently.
Think of it as two ways of saving a video game. The full re-reading layers record every frame, so you can scrub to any moment. The summary layers keep one large save file that gets overwritten as you play, and writing it out is slow and expensive.
Now, when you send a follow-up message to a chatbot, it does not reprocess the whole conversation. It picks up from a saved point. That is why the second message in a chat is much faster than the first, and it is a large part of why running these things is affordable at all.
For Kimi K3, picking up midway needs both records restored at the same moment. The frames and the save file.
But the save file is expensive to write, so it only gets written occasionally. And the system had been built so that the frame-recording schedule was locked to the save-file schedule. Which meant, in effect, that the game only autosaved every 6,000 words.
Follow that through and it is quite bad. A conversation shorter than 6,000 words could never be resumed at all, ever, because it never reached a save point. Long documents streaming in produced nothing reusable until they crossed a boundary. At a million tokens, where a typical coding request carries a 400,000 token history and adds 4,000 new ones, missing the save point means redoing all 400,000. The difference between hitting and missing is not a few percent. It is orders of magnitude of cost.
None of this was chosen. It was implied by a decision about how to mix layer types, made near the beginning of the project, and it stayed invisible until real traffic arrived.
The fix is one sentence. Stop letting the save schedule dictate the frame schedule.
Record frames every 512 words. Keep writing the expensive save file rarely, but only ever at moments that are also frame boundaries, since those are the only places anyone can ask to resume from. Now a resume can happen at any 512 word mark that has both.
What follows in the report is three rules about concurrency, and each reads like it was written the morning after an outage. One says that before you copy anything you must first pin every record you matched, because otherwise the act of making your copy can delete the thing another request was mid-way through using. Another says a record created during the current scheduling round has to be ignored until its copy has actually landed, because otherwise a reader gets the previous owner's data. That is a bug you find once and remember for life.
There is also a throwaway line I enjoyed. They store two structurally different kinds of record in one shared pool, and note that because the two are so different, mixing them up produces obvious garbage rather than plausible-looking data. They call it a free sanity check. That is the voice of someone who has debugged the version where it looked plausible.
Training an agent that runs for days
Kimi K3 is built to work unsupervised for a long time. Not "answer my question", but "spend four hours doing this job". One training task might involve thousands of tool calls across several simulated days.
Training something to do that breaks the normal machinery in a few ways.
You cannot wait for everyone to finish. Some attempts take minutes, some take hours, and waiting for the slowest idles the hardware. So they stop collecting once most are done, freeze the rest exactly where they are, and resume them next round. A single long attempt therefore spreads across several training rounds and its answers arrive somewhat out of date. They handle that by permitting only small corrections from any one answer, so stale information cannot yank the model around.
The agent needs a real computer, not a pretend one. Early on they used containers, the standard lightweight way to sandbox code. Agents kept crashing the host. Not maliciously, just by exploring aggressively, and containers share too much of the underlying machine to survive that.
They could have restricted what agents were allowed to do. They deliberately did not, because hard tasks need an environment where the agent can mount a disk or start a virtual machine if it decides to. So every agent gets its own miniature computer with its own kernel, and if it destroys that one, it destroys only that one.
And then the number that reframes everything.
A sandbox spends up to 98% of its life doing nothing. It is waiting for the model to think of the next command. The workshop is rented, the apprentice is standing still, the meter is running.
So they built the ability to freeze a sandbox mid-task, releasing all its memory and CPU, and thaw it in 49 milliseconds when the next instruction arrives. Read that back with the 98% in mind. This is not an efficiency tweak around the edges. The waiting was the bill.
Scale, for a sense of proportion: across training and evaluation, Moonshot created 51,219,741 sandboxes across just over 1.5 million different machine images.
The part Moonshot did not open-source
Look at what they gave away: the full 2.8 trillion parameter weights, their expert-distribution library, their sandbox system, their attention kernels, a compiler the model itself wrote, and the design files for a chip it produced unsupervised in 48 hours. An unusually generous release.
Now look at the gap. They did not release the self-growing knowledge map their agents built to invent training tasks. Or the fake Gmail, Notion, Slack and Canvas that assistant tasks practise against, complete with weeks of simulated events. Or the graders. Or the cheat detectors.
Those last ones are the most honest thing in the document. Set a model loose on "make this code faster" with a reward for speed and it will discover that faking the measurement is easier than doing the work: caching results, replaying old runs, quietly dropping precision. So Moonshot built a detector for each cheat as it appeared and kept extending it. They also split grading in two. A public grader gives feedback, a hidden one decides the score. Practice test versus real exam. Memorising the practice answers gets you nothing.
Here is the asymmetry. Anyone can read the architecture out of the paper and build it. Nobody can rebuild 51 million environment runs, or the years of learning about how a model cheats that got baked into those detectors.
The weights were the part they could afford to give away.
What Kimi K3 has actually built
The case studies are vendor-chosen best runs, so grade them accordingly. One still lands hard.
Given 24 hours per task, the model rewrote several of its own GPU routines. One went from 283 milliseconds to 114. Another lost three quarters of its runtime. It separately wrote a compiler from scratch, and in an unattended 48 hour run designed a working prototype AI chip, which they published.
Then the line that matters more than any of those numbers: an early version of Kimi K3 was already doing most of Moonshot's routine performance work while K3 was still being built. The model was speeding up the code that runs the model, before it shipped.
What I got wrong about the price
In the launch post I looked at the pricing, three dollars per million words in and fifteen out, and concluded that the era of very cheap Chinese models was ending. The number was right. What I took from it was not.
The report measures cost per completed task instead, and the picture inverts.
On a web research benchmark, Kimi K3 posts the best score in the field at $2.03 per task. That is roughly half what its nearest competitor costs and about a tenth of what the Claude models cost at full effort. On Moonshot's internal coding benchmark it finishes four points behind the leader while costing 38% as much.
The reason is that Kimi K3 was explicitly trained to be brief. Every practice problem came with a word budget, and going over it scored worse than getting the answer wrong. Train that way and you get a model that finishes in fewer words.
Which makes price per word a bad way to compare models, in the way that comparing processors by clock speed stopped working. A consultant with a lower hourly rate is not cheaper if they take four times as long.
What this probably does to the AI world
Guesses. I have marked which ones I would actually bet on.
Long context stops being a spec and becomes a cache problem. Nearly the whole serving half of this report is about reuse: keeping a saved conversation restorable, deciding what stays in fast memory, routing a returning user back to the machine that already has their history. Nobody's bottleneck at a million tokens is raw computing power. It is whether the save point survived. I would bet the next round of competition is fought on hit rates and cost per task rather than the context number printed on the box.
Mixed attention becomes standard. Three cheap layers to one expensive one, no positional ruler, stretching to a million tokens without recalibration, shipping in the largest open model in existence. That is no longer an experiment.
The moat moves from the model to the practice ground. The one I would bet hardest. Weights are becoming the cheap, giveable part. The expensive part is the simulated world you trained them in, and everything you learned about how they cheat in it.
Per-token pricing becomes a bad comparison. The same model has a wide cost range depending on how hard you ask it to think, and that range was engineered on purpose. Expect cost-per-task charts to become normal, and some vendors to resist them.
Open weights and refusal-based safety are on a collision course. Buried in the evaluations: Kimi K3 found sixteen previously unknown security vulnerabilities in widely used software, two in the Linux kernel, one of them remotely exploitable. The report also notes that the leading American models refuse this category of work entirely, which made a direct comparison impossible.
Those two facts sit next to each other and do not resolve. Refusal is the main public safety mechanism the top labs use, and it is precisely the mechanism that cannot survive a weight release, because anyone who downloads the model can remove it over a weekend. It is a speed limiter shipped with the wrench in the glovebox. I have no prediction for what replaces it, which is roughly the problem.
Models improving their own infrastructure is already real. Not a projection. An early checkpoint was doing most of the performance work on the system that produced it. That loop is running inside at least one lab right now.
What you can borrow from all this
You will never train a 2.8 trillion parameter model. These transfer anyway.
Put the parts that change last. Kimi K3's message format puts stable settings before the conversation and per-request settings after it, purely so that changing something small does not invalidate the saved history. If your app puts today's date at the top of a long prompt, you are reprinting the whole document to change one line. Move it to the bottom.
Give expensive and cheap requests separate budgets. Production traffic here mixes 2,000 word requests with million word ones. One shared queue means a burst of the big ones starves all the small ones. Any service with mixed workloads has a small version of this.
Never evaluate an agent under only one setup. Kimi K3 was deliberately trained across five different agent frameworks so it would not learn the quirks of any one. If you only ever test yours under one, you are measuring the framework, not the agent.
FAQ
Can I run Kimi K3 myself?
Not on anything you own. Moonshot suggest 64 or more accelerators. In practice most people will use it through an API, the same way they use a closed model.
Then what does "open" actually get anyone?
Companies can host it on their own hardware and keep their data in-house. Researchers can look inside and modify it. Neither is nothing. But it does mean "open" now covers two very different things, and this kind is not the kind you run on a laptop.
Is it better than Claude or GPT?
Overall, no, and Moonshot say so directly. It trails Claude Fable 5 and GPT-5.6 Sol. It beats everything else they tested, and it leads on specific things: web research, spreadsheets, document understanding, and a human-voted leaderboard for building web pages, where it is the first open model to reach first place.
Do I have to understand any of this to use an AI model?
No. But if you are choosing between them, the price-per-task point in this post is the one worth carrying around.
Where do I read the real thing?
Linked below. It is unusually candid about failures, which is rare in a document published on launch day.
Sources
- The weights and technical report on Hugging Face
- The tools they released alongside it: MoonEP, AgentENV, FlashKDA
- Things the model itself built: MiniTriton, nano-kpu
- Release coverage: officechai
My launch-day post, if you want the before picture:
If you have run K3 since the weights dropped, I would like to know whether the save-point behaviour holds up outside Moonshot's own benchmarks. That is the one claim here I have no way to check.
Top comments (0)