DEV Community: Pranav Raj

Kimi K3 explained in plain terms, and why "open" is getting complicated

Pranav Raj — Fri, 17 Jul 2026 08:12:25 +0000

TL;DR: Moonshot AI released Kimi K3 on July 16, a 2.8 trillion parameter open-weight model with a 1 million token context window. This post explains what those numbers actually mean, goes one level into the architecture (MoE, Kimi Delta Attention), and ends with my take: the era of dirt-cheap Chinese models may be ending, and "open" now means something you can inspect but probably never run.

I have not used K3 myself yet. This is me reading the launch material and independent coverage carefully so you get the picture in ten minutes instead of an afternoon.

What just happened

A Google News notification about "the largest open-source model ever" is what pulled me in. The claim sounded big enough to be worth checking properly.

Moonshot AI, the Beijing startup behind the Kimi models, released Kimi K3. The headline claims from their launch post:

2.8 trillion total parameters, which they call the first open model in the 3T class
A 1 million token context window
Native multimodality: it processes images and video in the same model, not through a bolted-on vision encoder
Benchmarks that, by Moonshot's own admission, still trail Claude Fable 5 and GPT 5.6 Sol, but beat everything else they tested, including Claude Opus 4.8 and GLM-5.2

The weights are not out yet. They are promised by July 27. Until then, nobody outside Moonshot can independently verify or run the model. Keep that in mind for everything below: these are launch-day numbers from the vendor, partially confirmed by early independent testing from Artificial Analysis.

The numbers, translated

2.8 trillion parameters does not mean 2.8 trillion at once. K3 is a Mixture of Experts (MoE) model. Think of it as 896 specialist sub-networks ("experts") sitting behind a router. For each token, the router picks 16 of those 896 to actually do the work. So the model has a huge library of knowledge on disk, but only a small slice of it fires per token. That is how you make a 2.8T model affordable to serve at all. This is the same broad trick behind most large open models today; K3 just pushes the sparsity further than usual.

A 1 million token context window means roughly 700,000+ words of input. That is an entire large codebase, or several books, in one prompt. The catch with long context has never been "can you fit it" but "can the model actually use token 900,000 when answering," and that is exactly what K3's attention changes are aimed at (next section).

"Open" here means open weights, later. You will be able to download the model and run it yourself. But at this scale, "yourself" means a serious GPU cluster. Moonshot recommends deploying on nodes with 64 or more accelerators. So it is open in the sense that researchers and companies can inspect, fine-tune, and self-host it. It is not open in the sense that you or I will run it on anything we own.

One level deeper: what is actually new in the architecture

Two named components carry the release, and both are about how information flows through the model.

Kimi Delta Attention (KDA). Standard attention gets expensive as context grows, because every token can look at every other token. KDA is Moonshot's answer for scaling attention efficiently to very long sequences. They also mention it breaks conventional prefix caching, and that they contributed a fix to vLLM, which tells you they expect people to actually serve this thing, not just benchmark it.

Attention Residuals (AttnRes). In a deep network, information from early layers has to survive dozens or hundreds of layers to influence the output. AttnRes, per Moonshot, lets the model selectively retrieve representations from earlier depths instead of accumulating everything uniformly. Intuition: rather than a message being whispered down a long chain of people, later layers can go back and re-read earlier notes directly.

On top of that, a stack of training tricks with names like Quantile Balancing (keeping the 896 experts evenly loaded without a fragile hyperparameter) and quantization-aware training in MXFP4/MXFP8 (so the released weights run efficiently on a wide range of hardware). Moonshot claims all of this together gives roughly 2.5x better scaling efficiency than their previous K2 generation, meaning more capability per unit of compute.

The full technical report is not out yet, so the honest status of every claim above is "vendor-described, plausible, unverified." The July 27 weights release is when the community gets to check.

The parts of the launch post worth raising an eyebrow at

The demos are genuinely impressive on paper: a Triton-like GPU compiler built from scratch, a chip designed in a 48-hour autonomous run, a research pipeline reproducing astrophysics results in two hours. But launch-day case studies are always the best runs, selected by the vendor. The Decoder's coverage notes that independent testing broadly confirmed the benchmark story, but also that K3's hallucination rate went up compared to its predecessor. Moonshot's own limitations section admits the model can be "excessively proactive," making decisions on your behalf when instructions are ambiguous. For an agent meant to run unsupervised for hours, that cuts both ways.

My take: two quiet shifts hiding in this release

First, cheap Chinese models may be over. K3 costs $3 per million input tokens and $15 per million output tokens. That is not DeepSeek-style pricing; that is Western mid-tier pricing. The old story was that Chinese labs compete on being 10x cheaper. K3's story is "we are nearly frontier, pay accordingly." If that holds, the pricing gap that made a lot of hobby projects viable is closing.

Second, "open source AI" is splitting into two different things. There is open-as-in-runnable, models like the smaller Llama and Qwen variants that a student with a decent GPU can actually use. And there is open-as-in-inspectable, 2.8T monsters that are technically downloadable but practically only serve as public research artifacts and enterprise self-hosting options. Both are valuable. But when headlines say "the largest open model ever," it is worth being clear that its openness will be experienced by almost everyone through an API, same as a closed model.

Where does that leave me? Honestly, nowhere near K3. It is far too big to run or experiment with locally, and I do not currently have a project that needs an AI API. Which is kind of the point: a model this "open" changes nothing about my day to day, and I suspect that is true for most individual developers reading this.

What I would watch next

July 27: do the weights actually land, and what does the technical report reveal about KDA and AttnRes?
Whether independent benchmarks hold up once anyone can run it.
DeepSeek's response, which reporting suggests is close.

If you have already tried K3 through the API or Kimi Code, I would genuinely like to hear how it compares to whatever you use now. That is the data point launch posts can't give us.

Sources: Moonshot's K3 launch post, VentureBeat, The Decoder, TechCrunch.

I built vector search by hand out of curiosity, and my toy caught a mistake I did not know I had made

Pranav Raj — Tue, 14 Jul 2026 14:20:24 +0000

TL;DR: I shipped a "second brain" app at a hackathon that searched notes by meaning, and I did not understand the search part. So I rebuilt it by hand in numpy, 20 fake products in 8 dimensions. The mechanism turned out to be about ten lines. The interesting part was that the toy quietly told me two of my four categories overlapped, which I had done to myself and never noticed. That is the thing real embeddings will never let you see.

The question that started it

At a hackathon I built second-brain, which stores notes as memories and searches them by meaning instead of exact words. It runs sentence-transformers (all-MiniLM-L6-v2) to turn text into 384-dimensional vectors, and Qdrant to store and search them. It worked. You could type a rough idea and get back the note you meant, with none of the same words in it.

But I had wired that up without knowing what it did. It was an embed() call and a Qdrant query, and it felt like magic. Magic in my own project bothers me, so I wrote down the actual question:

What happens between embed(query) and the results coming back?

Not "what is an embedding". I could recite that. I wanted the mechanism, the part that takes a pile of vectors and picks the close ones. So I built one. No model, no vector database, no framework. Just numpy, 20 made-up products across four categories (electronics, clothing, cooking, stationary), each clustered around a hand-picked center in 8-dimensional space with noise added so they were not identical.

import numpy as np

# each category is a cluster center in 8-dimensional space
electronics_center = np.array([0.9, 0.1, 0.2, 0.08, 0.1, 0.3, 0.79, 0.2])
clothing_center    = np.array([0.1, 0.8, 0.09, 0.1, 0.09, 0.2, 0.1, 0.89])
cooking_center     = np.array([0.2, 0.3, 0.95, 0.2, 0.1, 0.9, 0.3, 0.1])
stationary_center  = np.array([0.8, 0.2, 0.86, 0.9, 0.08, 0.1, 0.2, 0.1])

# real embeddings vary, so simulate that with noise around each center
noise = 0.1
electronics = electronics_center + noise * np.random.randn(5, 8)
# ...same for the other three categories

Using fake vectors was the whole point, not a shortcut. I did not want to learn how a transformer produces a vector. I wanted to isolate the half I did not understand, which is what you do with the vectors once you have them. Remember the four centers above. They come back later and they are the best part of this post.

Question 1: how do you actually compare two vectors?

The answer is short enough to be annoying. Normalize every vector to unit length, then rank by dot product. For unit vectors, the dot product is cosine similarity, because the denominator in the cosine formula is just the two magnitudes, and you already made both of them 1.

def normalize(vectors: np.ndarray) -> np.ndarray:
    norms = np.linalg.norm(vectors, axis=1, keepdims=True)
    norms = np.where(norms == 0, 1e-10, norms)  # avoid divide by zero
    return vectors / norms

class VectorIndex:
    def __init__(self):
        self.vectors = None
        self.labels = None

    def add(self, vectors: np.ndarray, labels: list):
        self.vectors = normalize(vectors)   # normalize once, at index time
        self.labels = labels

    def search(self, query_vector: np.ndarray, top_k: int = 3):
        query = normalize(query_vector.reshape(1, -1))
        scores = np.dot(self.vectors, query.T).flatten()  # cosine similarity
        top = np.argsort(scores)[::-1][:top_k]
        return [(self.labels[i], float(scores[i])) for i in top]

That is the entire engine. It ran, and every query came back with the right category on top:

Query: 'Cooking'
  1. [0.9948] Ceramic pan
  2. [0.9816] Iron cast pan
  3. [0.9749] wodden spoon

Query: 'Clothing'
  1. [0.9935] Silk saree
  2. [0.9808] Oversized black t-shirt
  3. [0.9775] Women's crop top

Two things clicked here that no amount of reading had done. Normalizing is not a ritual, it is what buys you the cheap dot product later. And it belongs in add(), not search(), because you index once and you query forever.

The other thing I finally understood is why it is cosine and not plain Euclidean distance. Cosine only looks at direction, so it ignores magnitude. For text that matters, because a long document and a short one about the same topic can differ a lot in magnitude while pointing the same way. Distance would call them far apart. Cosine calls them the same thing.

Qdrant adds a great deal on top of this, and I will get to what, but the core of what my second-brain runs on is those ten lines.

Question 2: can I see the space?

8 dimensions, so no. Not directly. I projected down to 2D with PCA and plotted the products as dots and the queries as stars. The four clusters separated cleanly and each query star landed inside its cluster. Very satisfying.

Then I printed how much of the variance those two components actually captured:

PC1: 45.92%
PC2: 26.45%
Total (PC1 + PC2): 72.36%

So the picture I was happily trusting is showing me about 72 percent of the structure. More than a quarter of what distinguishes these vectors is not on that plot at all, and 8 dimensions is a toy. Real embeddings are 384 or 1536. Whatever a t-SNE or UMAP plot of those looks like, the honest reading is that it is a heavily lossy sketch, and two points sitting on top of each other in the picture may not be near each other in the space.

That was the first thing I got out of the build that I did not expect: a healthy distrust of every embedding visualization I had ever nodded along to, including my own.

Question 3: is a high score proof the result is right?

Top-3 always looked perfect, which is exactly the problem, because top-3 is all a search UI ever shows you. So I stopped looking at the top and scored the query against all twenty products, then sorted.

That is when the run got interesting. Here is the tail of each query, the highest-scoring wrong-category item:

Query	Worst in-category score	Best wrong-category score	Margin
Clothing	0.9572	0.4457 (wodden spoon)	0.51
Electronics	0.9441	0.6542 (Parker pen)	0.29
Cooking	0.9649	0.6689 (Glue stick)	0.30
Stationary	0.9704	0.7110 (Nakiri knives)	0.26

A cutoff around 0.8 separates right from wrong for every query here, so the toy did not fail. But look at the margins. Clothing has twice the breathing room that stationary does. Same code, same noise, same everything. Why would one category be twice as easy as another?

The thing I did not go looking for

The queries were generated near the cluster centers. So if some categories are getting confused with each other, maybe the centers themselves are the problem. I had never thought to check them, because I picked those numbers by hand, in about thirty seconds, to "look different". So I measured the cosine similarity between the four centers:

electronics  vs clothing     0.3318
electronics  vs cooking      0.5287
electronics  vs stationary   0.6202
clothing     vs cooking      0.3872
clothing     vs stationary   0.2911
cooking      vs stationary   0.6475

There it is. Clothing sits between 0.29 and 0.39 from everything else, off in its own corner. Stationary is 0.62 from electronics and 0.65 from cooking, wedged right between them. Go back and look at the centers I hand-wrote: stationary is high on dimension 0 (0.8), and so is electronics (0.9). Stationary is also high on dimension 2 (0.86), and so is cooking (0.95). I built a category that shares a strong axis with two others without noticing, and then I built the two other categories it overlaps with.

And the search told me. The two most-confused pairs in the search results, stationary and cooking bleeding into each other at 0.71 and 0.67, are exactly the two highest-similarity pairs in that table. The retrieval quality was a direct readout of a geometry mistake I had made and could not see.

That is the payoff of using fake vectors, and it is the opposite of what I expected. I thought the toy would teach me the algorithm and the algorithm was the boring part. What the toy actually gave me is the only setting where I get to see the ground truth of the space and the search behavior at the same time, and compare them. With all-MiniLM-L6-v2 I get 384 numbers I cannot interpret, and if two of my note categories overlap in that space, nothing tells me. The search just quietly gets worse and I blame the threshold.

What this changed about my actual app

The retrieval in second-brain is not pure similarity, and building the naive version showed me why it cannot be:

payload = {
    "vector": embed(query),         # semantic similarity
    "limit": limit,
    "score_threshold": 0.5,         # drop "near but irrelevant" hits
    "with_payload": True,
    "filter": {                     # hard constraints, not similarity
        "must": [
            {"key": "memory_type", "match": {"value": "knowledge"}},
            {"key": "importance", "range": {"gte": 0.3}},
        ]
    },
}
requests.post(f"{QDRANT_HOST}/collections/knowledge_memory/points/search", json=payload)

Cosine similarity does exactly one thing: find the nearest vector. Nearest is not "correct", and it is definitely not "allowed". So the similarity finds candidates and everything else enforces the rules. "Notes from last week tagged work" is a database query wearing a search box, not a similarity question, and no amount of embedding quality will turn it into one.

The same logic explains the failure I have hit most in practice: exact matches get worse, not better. Search a file name, an ID, or a tag, and semantic search will confidently hand you something close in meaning and bury the exact hit. Plain string matching wins there and it is not close.

I also now think that score_threshold: 0.5 is a number I made up. It works, in the sense that the app stopped returning nonsense. I have no principled reason for it, and the margin table above is why that bothers me. The gap between right and wrong moved by 2x across four categories in a dataset I built myself. In a real embedding space I have no idea what it is.

The questions I left with

The mechanism took an afternoon. These did not:

How do you pick a score threshold honestly? Mine is a vibe. Doing it properly seems to need labelled query and result pairs and a look at where the score distributions actually cross, which means I need ground truth for my own notes, which I do not have. This is the one I want to solve next.

How does the geometry change at 384 dimensions? Everything above is 8 dimensions, where I can reason about a single axis being shared. I have read that similarity scores bunch up as dimensions increase, so everything starts looking moderately similar to everything. I have not tested it, and my whole intuition here was built in a space small enough to be unrepresentative.

How does a real index avoid comparing everything? My search() is a dot product against every vector, O(n) per query, which is fine for 20 products and absurd for a million. Qdrant uses HNSW to approximate. So it can be wrong, on purpose, for speed. I would like to build that next and find out what "approximate" actually costs, because right now I take that trade on faith.

Filtering and approximate search must fight each other. If HNSW is walking a graph to find near neighbors, and I also demand importance >= 0.3, does it filter before it walks or after? Filter after and you can get back fewer results than you asked for. Filter before and it is not clear how the graph still works. I know Qdrant solves this. I do not know how.

How do you route a query to the right search? The exact-match problem has a known answer, hybrid search: run keyword and vector search, merge the rankings. What I do not have is a clean way to decide which path a query deserves, or whether you just always run both and fuse. This is where I am stuck.

If you have built hybrid retrieval in production, I would like to know how you make that routing decision, and whether you calibrated your score threshold or picked it the way I did. That is the honest state of it.

Second-brain is here:

CoffeeAurCode / second-brain-ai_MVP

second-brain-ai (qdrant-based)

this project is a minimal Second brain AI system using qdrant as the primary vector memory store

How to Run

Start Qdrant locally docker run -p 6333:6333 qdrant/qdrant
Install dependencies: pip install -r requirements.txt
Run the any of the demo file: python Demo_1.py/Demo_2.py/Demo_3.py

The demo will store knowledge, retrieve relevant memories, apply memory evolution, and retrieve again.

Dependencies

This project uses Qdrant as the primary vector search engine. Qdrant is run locally using Docker, which is the official and recommended deployment method.

Docker is required only to start the Qdrant service. All application logic runs in Python.

Memory Architecture

The system uses Qdrant as the primary memory store.

Three typed memory collections are defined:

knowledge_memory: long-term semantic knowledge(hard disk)
context_memory: short-lived working context(RAM)
interaction_memory: conversation traces and user intent(EQ)

All future retrieval and memory evolution operates exclusively on these collections.

Embeddings

Text is converted into semantic vector representations…

View on GitHub

The paper reported jumps like 21% to 97%. My replication got +2%.

Pranav Raj — Tue, 07 Jul 2026 14:43:50 +0000

TL;DR: I ran a small replication of the paper "Prompt Repetition Improves Non-Reasoning LLMs": 100 MMLU questions, one non-reasoning model, prompt sent once versus twice. Baseline 59%, repetition 61%. Probably not statistically significant, and that turned out to be the interesting part. The gap between the paper's headline numbers and my +2% taught me more about transformer attention than the trick itself.

Why I tried this

The claim sounds like a joke: paste your prompt twice and a non-reasoning LLM gets better. No chain-of-thought, no fine-tuning, no extra instructions. The paper, Prompt Repetition Improves Non-Reasoning LLMs from Google Research, tested Gemini, GPT, Claude, and Deepseek models across seven benchmarks and found repetition won 47 of 70 model-benchmark combinations with zero losses, including improvements as large as 21% to 97% on long-context tasks like NameIndex, where the model has to pull one item out of a long list.

A trick that cheap deserves a replication, and I wanted to know two things. Does it show up on ordinary short questions? And if it does, why does duplicating text change anything at all inside a transformer?

The setup

I kept it deliberately small and fixed everything I could:

Dataset: 100 multiple-choice questions sampled from MMLU science and math subjects (high school and college math, physics, chemistry, biology, abstract algebra, astronomy). Four options, one correct answer. The dataset was built once with a fixed seed and frozen to questions.json so both conditions saw identical questions.
Model: llama-3.1-8b-instant via the Groq API. A small, fast, genuinely non-reasoning model, which is the population the paper is about.
Conditions: the baseline sends the question once with "Answer with one letter." The repetition condition sends the exact same block twice (n = 2), nothing else changed.

# baseline                      # repetition (n = 2)
<Question>                      <Question>
A. ...                          A. ...
B. ...                          B. ...
C. ...                          C. ...
D. ...                          D. ...
Answer with one letter          Answer with one letter
                                <Question>
                                A. ...
                                ...
                                Answer with one letter

Results

Method	Correct	Accuracy
Baseline (n = 1)	59 / 100	59%
Repeated prompt (n = 2)	61 / 100	61%

Two more questions correct. Before calling that a win, the honest check: for paired correct/incorrect outcomes on the same questions, the right test is McNemar's, which looks only at the questions where the two conditions disagree. With 100 questions and a swing this small, the improvement is likely not statistically significant. I am not claiming the effect; I am reporting what 100 questions showed.

So the replication "worked" in the least satisfying way possible: a positive direction, too small to trust on its own. The useful part was figuring out why the effect should be small here when the paper's numbers are so large.

Why my effect was small and the paper's was big

The paper's dramatic gains come from long-context retrieval tasks. Give a model a list of 50 names and ask for the 25th, and attention has to hold position across hundreds of tokens. Attention weight spreads thin across a long sequence, the signal for any single token gets diluted, and the model misses. Repeating the prompt puts a second copy of every important token in the context, doubling the chances that the answer-relevant tokens get attended to. On tasks that stress attention, that rescue is worth a lot.

My questions were short. A four-option MCQ barely stresses attention at all; the model can already see everything clearly. Repetition amplifies a signal that was never weak, so there is not much headroom to gain. The effect size is not a property of the trick, it is a property of how starved the task is for attention. That is the actual takeaway, and it is also a testable prediction: rerun this on long-context questions and the gap should open up.

Why repetition changes anything at all

This is the part that got me to actually read about attention instead of nodding along to diagrams.

When the model predicts its answer token, attention is distributed across every prompt token. A key concept might get a weight of, say, 0.07. With the prompt repeated, each copy might get 0.05 and 0.04, but the combined signal is stronger than before. Nothing about the model changed; the input just gave the mechanism two chances to find the same information.

The KV cache view says the same thing: during inference every prompt token's key and value vectors sit in the cache, and repetition puts two copies of the relevant vectors in there. Retrieval by attention becomes more reliable when the target exists twice. Prompt repetition is signal amplification you perform from outside the model, which is also a decent one-line explanation of why prompt engineering works at all.

The paper itself adds a framing I had not considered: causal masking. In a causal LLM, tokens can only attend backwards, so a token early in your question never gets to see the tokens that come after it. In the second copy of the prompt, every token effectively has the full query behind it. Repetition is the cheapest possible workaround for one-directional attention.

Limitations, plainly

100 questions is small, and the headline result is within noise.
Short MCQs only, which the previous section argues is the worst case for this trick.
One model, one repetition factor (n = 2).

What I would run next

The obvious follow-ups, roughly in order of what I am most curious about: long-context tasks like the paper's NameIndex setup where the prediction says the effect should be large, repetition factors n = 3 to 5 to find where returns diminish, and a comparison against chain-of-thought on the same frozen dataset, since both tricks spend extra tokens to buy accuracy.

Repo with the dataset builder, notebook, and raw per-question CSVs:

CoffeeAurCode / Prompt-Repetition-Improves-Non-Reasoning-LLMs

I tried replicating the paper “Prompt Repetition Improves Non-Reasoning LLMs.”

Prompt Repetition Improves Non-Reasoning LLMs

A Small Replication Study

This repository contains a small experiment replicating the core idea from the paper:

Prompt Repetition Improves Non-Reasoning LLMs

The goal of this experiment was to understand whether repeating a prompt improves accuracy on a custom multiple-choice dataset and to explore why such a simple trick can affect transformer behavior.

Rather than focusing on achieving large gains, the purpose of this project was to investigate the mechanics of prompt repetition and understand what it reveals about transformer attention.

Motivation

Recent work has shown that repeating a prompt multiple times can significantly improve the performance of non-reasoning LLMs on certain tasks.

Example reported improvements from the paper: 21% -> 97% accuracy

on tasks requiring long context reasoning such as NameIndex.

This raises interesting questions:

Why does prompt repetition help?
Does it work on simple MCQ tasks?
What does this reveal about…

View on GitHub

If you have run prompt repetition on anything with real context length, I would like to know whether the effect showed up for you, and how large it was. And if you see a flaw in the setup that could explain the +2% besides the trick itself, that is exactly the kind of comment I am hoping for.