DEV Community: Cheng

Making Chroma Work in Chinese

Cheng — Mon, 21 Sep 2026 02:40:53 +0000

Note:

This tutorial picks up where Chroma's official quick start guide leaves off.

This article was revised after feedback from Ahmet Özel (see the Baseline section for the change he proposed).

So you've finished Chroma's quick start — congratulations. Everything works, right up until you feed Chroma some Chinese text.

Then it doesn't.

Chroma's default embedding model, all-MiniLM-L6-v2, is English-only. Given Chinese or Japanese input, it embeds it as though the language were noise. You need a model that was trained on the language you care about.

You have two ways to get one:

	Local model	Online model
Runs where	on your machine	vendor's server
Network	not needed	required
Setup	install packages, download weights	an API key
Cost	disk space (hundreds of MB)	money per query
Offline	works	does not
Privacy	good	bad

This tutorial walks through both, and then a third option that fixes a problem neither of them solves: ranking.

To be concrete, we'll assume Chinese is the language you need to add. Swap the model name and the same code supports any language.

Baseline of the Default Model

Let's first establish a baseline: how well does Chroma's default model actually do on Chinese?

The first step is to create a client and a collection.

import chromadb
client0 = chromadb.Client()
collection0 = client0.create_collection(name='demo0')

We did not pass a custom embedding function to create_collection(), so it uses the default all-MiniLM-L6-v2.

Next come the documents that will serve as our knowledge base:

docs = [
    'Python 是一种流行的编程语言，广泛用于数据处理、机器学习以及广义上的人工智能。',
    '数据处理包括将原始数据清洗、转换和聚合成有用的格式。',
    '机器学习通过训练模型来发现数据中的模式，并对新样本进行预测。',
    '深度学习使用具有多层的神经网络来学习分层表示。',
    '神经网络是受大脑启发的计算系统，由相互连接的节点层组成，这些节点层学习从数据中识别模式并进行预测。',
    'Transformer 使用自注意力来建模序列中词元之间的关系。',
    '注意力机制让模型在生成每个输出表示时，对输入的不同部分进行加权。',
    'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
    '大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。',
    "预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。",
    "微调使用有标注数据或指令数据，使预训练模型适应特定任务或领域。",
    "后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。",
    "向量嵌入将文本、图像或其他数据映射到高维空间中的点，以进行相似度搜索。",
    "检索增强生成（RAG）将来自外部知识源的检索与大语言模型（LLM）的生成相结合。",
    "一个基础的 RAG 流程会切分文档、对文本块进行嵌入、将其存入向量索引、检索前 k 个文本块，并将它们传给 LLM。",
]

A few things to note about this corpus:

Each item of docs represents a document, albeit a short one with a single sentence.
The sentences are mostly Chinese, except a few technical terms that are deliberately English.
Each sentence is about a single idea, e.g., the first one is the definition of 'Python', the sixth about 'Transformer', etc.
The central idea of the sentence usually appears at the beginning. All sentences share the structure of '{CENTRAL IDEA} is/does ...'.
So, for what-is questions, you get one or two most relevant documents serving as their ground truth.
- For example, if you ask 'What is Transformer?', it should give you the sixth sentence, which contains 'Transformer' as its central idea and serves as the query's ground truth.
- Similarly, if you ask about RAG, the last two sentences are most relevant, hence the ground truth.

We will use this set of documents throughout this article to compare the performance of different embedding models.

Now add these documents to the collection:

collection0.add(ids=[f'id{i+1}' for i in range(len(docs))], documents=docs)

The part [f'id{i+1}' for i in range(len(docs))] looks a bit busy, but it simply expands to a list of automatically-numbered IDs: 'id1', 'id2', ....

Now let's prepare two typical questions:

queries = [
    '什么是Transformer？',  # 'What is Transformer?'  Most relevant: 'id6'
    '什么是预训练？',        # 'What is pretraining?'  Most relevant: 'id10' 
]

These are what-is questions deliberately designed to match two central ideas of docs.
We will fix the questions in all of our experiments.
One caveat: the thing we ask about is English ('Transformer') in the first case, but Chinese ('预训练') in the second.

Now actually ask the model the two questions:

collection0.query(query_texts=queries, n_results=3)

The output is

{'ids': [['id6', 'id8', 'id12'], ['id12', 'id11', 'id7']],
 'embeddings': None,
 'documents': [['Transformer 使用自注意力来建模序列中词元之间的关系。',
   'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
   '后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。'],
  ['后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。',
   '微调使用有标注数据或指令数据，使预训练模型适应特定任务或领域。',
   '注意力机制让模型在生成每个输出表示时，对输入的不同部分进行加权。']],
 'uris': None,
 'included': ['metadatas', 'documents', 'distances'],
 'data': None,
 'metadatas': [[None, None, None], [None, None, None]],
 'distances': [[0.32853031158447266, 0.7125810384750366, 1.0673757791519165],
  [0.7402779459953308, 0.7472522854804993, 0.8964452743530273]]}

First, we notice that the responses are not answers to the questions, but instead collections of relevant documents.

For the first question about 'Transformer', the model selects 'id6' (ground truth), 'id8', 'id12'. So we may think that Chroma's default embedding is fine for Chinese.

But not so fast. The keyword 'Transformer' in this case is English, which lies in the comfort zone of the default embedding.

Now for the second question, in which we ask about an idea in Chinese ('预训练', or 'pretraining'). The ground truth is 'id10'. But the default model failed to identify it in the top-3 relevant items.

This shows the default model cannot handle Chinese properly. And this is why we need to use a custom embedding model.

What is more concerning (thanks to Ahmet Özel for raising this) is that the model did not throw a warning or notice in the second case. This kind of garbage output can readily sneak into production.

Custom Embedding Model

Whatever embedding model you choose, the hook into Chroma is the same: the optional embedding_function argument of client.create_collection().

# Replace with your own class
ef = YourEmbeddingFunction(...)

collection = client.create_collection(
    name="my_collection",
    embedding_function=ef)

That's the whole interface. The rest of this tutorial is about what to put in place of YourEmbeddingFunction.

Option 1: Local Models

The first option is about deploying the embedding model locally: you download the model to your disk, and load its weights from your disk. No network connection needed.

Choosing a model

Assuming you want Chinese support from a local model, these are the usual candidates:

Model	Language	Dim.	Notes
`paraphrase-multilingual-MiniLM-L12-v2`	Multilingual	384	lightweight
`BAAI/bge-small-zh-v1.5`	Chinese	512	good for Chinese
`BAAI/bge-base-zh-v1.5`	Chinese	768	same family, stronger
`BAAI/bge-m3`	Multilingual	1024	multilingual
`text2vec-base-chinese`	Chinese	768
`m3e-base`	Chinese	768
`all-MiniLM-L6-v2`	English	384	Chroma's default (for reference)

Roughly:

More dimensions means better quality and a bigger download.

The smallest multilingual option is only 384-dimensional and stays under 500 MB.
bge-m3 is 1024-dimensional and will cost you well over 2 GB.
For Chinese specifically, bge-small-zh-v1.5 is the cheapest thing that works properly, so that's what we'll use below.

Installing the prerequisite package

To use a local model you need one extra Python package, sentence-transformers. Chroma does not ship it by default, for a good reason:

The model is loaded through embedding_functions.SentenceTransformerEmbeddingFunction.
That class needs the sentence-transformers package.
Which in turn pulls in a string of other packages:
- torch (PyTorch)
- transformers
- huggingface-hub
- tokenizers
- numpy, scipy and friends.

That's pretty much a full deep-learning stack — hundreds of MB of dependencies. If your environment already has the stack, fine. If your project is, say, a small web service that just happens to need a vector store, that deep-learning stack is a bit of overkill. So Chroma makes you opt in:

pip install sentence-transformers

Downloading the model

Now we can actually build the embedding function.

The first time you construct SentenceTransformerEmbeddingFunction('BAAI/bge-small-zh-v1.5'), it downloads the weights from huggingface.co, where the model lives by default.

If you live in China, or anywhere the connection to huggingface.co is poor, point it at a mirror first. On Linux:

export HF_ENDPOINT="https://hf-mirror.com"

On Windows PowerShell:

$env:HF_ENDPOINT="https://hf-mirror.com"

On Windows CMD:

set HF_ENDPOINT=https://hf-mirror.com

Or, if you prefer setting it from inside Python — often the most convenient — use os.environ['VAR'] = ...:

from chromadb.utils import embedding_functions

import os
os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
        model_name='BAAI/bge-small-zh-v1.5')

The output is something like this:

Loading weights: 100%|███████████████████████████████████████████████████████████████| 71/71 [00:00<00:00, 9559.74it/s]

What to notice in that output:

The last call takes a while, especially the first time, to download the model and load its weights.
The progress bar above is about loading the weights, not about downloading the model.
Run it a second time and the download is skipped, so startup gets noticeably faster.
Loading the weights still takes a moment every session — don't expect a split-second import.

By default the model lands in ~/.cache/huggingface/hub on Linux, or C:\Users\<you>\.cache\huggingface\hub on Windows.

Caching the model inside your project

A user-level cache is convenient, but it makes the project non-portable: a teammate clones the repo, and still has to download ~92 MB before anything runs. Downloading into the project directory instead fixes that — one hf command does it (the same tool was previously called huggingface-cli):

$env:HF_ENDPOINT="https://hf-mirror.com"
pip install "huggingface_hub[cli]"
hf download BAAI/bge-small-zh-v1.5 --local-dir ./models/bge-small-zh-v1.5 --exclude "*.bin"

By --exclude "*.bin", you skip the legacy PyTorch weight files and avoid storing the same weights twice—cutting disk usage roughly in half.

After this command, ./models/bge-small-zh-v1.5 contains roughly 92 MB of files.

Then point model_name at the local path. The leading ./ is what tells SentenceTransformerEmbeddingFunction this is a directory and not a HuggingFace model ID:

from chromadb.utils import embedding_functions
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
        model_name="./models/bge-small-zh-v1.5")

The output:

Loading weights: 100%|██████████████████████████████████████████████████████████████| 71/71 [00:00<00:00, 14191.55it/s]

You can tell it worked: the loading is faster (no network round-trip to resolve the model).

Peeking at the source (Optional)

Curious about what exactly a sentence transformer is? Take a look at the source code:

import inspect
print(inspect.getsource(embedding_functions.SentenceTransformerEmbeddingFunction))

The output is long, so only the beginning matters here:

class SentenceTransformerEmbeddingFunction(EmbeddingFunction[Documents]):
    # Since we do dynamic imports we have to type this as Any
    models: Dict[str, Any] = {}

    # If you have a beefier machine, try "gtr-t5-large".
    # for a full list of options: https://huggingface.co/sentence-transformers, https://www.sbert.net/docs/pretrained_models.html
    def __init__(
        self,
        model_name: str = "all-MiniLM-L6-v2",
        device: str = "cpu",
        normalize_embeddings: bool = False,
        **kwargs: Any,
    ):
        ...

Two things worth noticing.

The default model_name is Chroma's English default — which is the whole reason this tutorial exists.
And there's a device argument: pass "cuda" to move embedding onto a GPU if you have one. If you're familiar with PyTorch, this is the same thing as torch.device("cuda").

Using the local model

Time to put it to work. If you've run this before, the collection already exists and Chroma will refuse to create it again — hence the commented-out cleanup line, which you'll want in a notebook where cells get re-run:

# if you've created the collection before, uncomment and run the next line
#client1.delete_collection(name='demo1')

import chromadb
client1 = chromadb.Client()
collection1 = client1.create_collection(
        name="demo1",
        embedding_function=ef)

Now add the documents to the collection:

collection1.add(ids=[f'id{i+1}' for i in range(len(docs))], documents=docs)

And ask the questions:

collection1.query(query_texts=queries, n_results=3)

The return value looks like this:

{'ids': [['id8', 'id6', 'id9'], ['id12', 'id10', 'id8']],
 'embeddings': None,
 'documents': [['GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
   'Transformer 使用自注意力来建模序列中词元之间的关系。',
   '大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。'],
  ['后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。',
   '预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。',
   'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。']],
 'uris': None,
 'included': ['metadatas', 'documents', 'distances'],
 'data': None,
 'metadatas': [[None, None, None], [None, None, None]],
 'distances': [[0.3059161305427551, 0.36248522996902466, 0.4628119468688965],
  [0.34848546981811523, 0.39471912384033203, 0.4147259593009949]]}

Assessing the results

Question 1: 'What is Transformer?'

The top two hits, id6 and id8, are both genuinely about Transformers. Good.
The third hit, id9, is about LLMs, which was less relevant.
The first two hits got relatively low distances, 0.31 and 0.36, much less than the third's 0.46. Good.

Question 2: 'What is pretraining?'

There is a huge improvement over the default embedding model.

The ground truth 'id10' is now included in the top-3 list.

Yet the ground truth is not ranked first. So there is still room for improvement.

What pushes id12 above id10 has something to do with the nature of vector embedding — this is just my guess. An embedding model compresses a whole sentence into a single vector, and the vector cares more about what words appear than what roles they play in the sentence. Both items mention "预训练" (pretraining); but id12, the item mainly about post-training, edges out because it has the token "训练" (training) appearing twice, while the ground truth id10 uses it only once. Same story for id8. Nothing in the embedding stage distinguishes "mentions pretraining" from "explains pretraining" — that distinction requires reading the query and the candidate together, which is a different architecture. We'll come back to it in the last section.

Option 2: Online Models

If you'd rather not download models, a hosted embedding API is the alternative. The trade-offs are the ones from the table at the top: you need an API key, and you pay per call, but there's nothing to install and the model is somebody else's problem to keep up to date.

Writing the embedding function

Chroma doesn't know about your vendor, so you need to write a small adapter. The reason DashScope does not provide an official adapter for Chroma is, I would guess, that there are simply too many vector search solutions out there to cover. All the adapter needs is a class with a __call__ method that maps a list of strings to a list of vectors.

Here we use Alibaba's DashScope, whose API is OpenAI-compatible — which means the official openai client works as-is, with only the base_url changed:

import os
from openai import OpenAI
from chromadb.api.types import EmbeddingFunction

class DashScopeEmbeddingFunction(EmbeddingFunction):
    def __init__(self, model='qwen3.7-text-embedding-flash'):
        self.model = model
        self.client = OpenAI(
            base_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
            api_key=os.getenv("DASHSCOPE_API_KEY"))

    def __call__(self, input: list[str]):
        response = self.client.embeddings.create(
            model=self.model, input=input)
        return [item.embedding for item in response.data]

A few notes on the adapter:

The API key is read from the DASHSCOPE_API_KEY environment variable when you don't pass one in. Never hard-code it in a notebook you might share.
__call__ receives a batch of texts and returns one vector each — Chroma always batches, so don't loop over the input and make one request per document.
The returned objects use .embedding and .index; if you ever need to be defensive about ordering, sort by index before returning.

Then build a second collection on top of it:

#Run this if the next cell complains that the collection 'demo2' already exists
#client2.delete_collection(name='demo2')

import chromadb
client2 = chromadb.Client()

ds_ef = DashScopeEmbeddingFunction()
collection2 = client2.create_collection(
        name='demo2',
        embedding_function=ds_ef)

The same documents, the same questions — only the model changed:

collection2.add(ids=[f'id{i+1}' for i in range(len(docs))], documents=docs)

collection2.query(query_texts=queries, n_results=3)

The output:

{'ids': [['id8', 'id6', 'id3'], ['id10', 'id12', 'id8']],
 'embeddings': None,
 'documents': [['GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
   'Transformer 使用自注意力来建模序列中词元之间的关系。',
   '机器学习通过训练模型来发现数据中的模式，并对新样本进行预测。'],
  ['预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。',
   '后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。',
   'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。']],
 'uris': None,
 'included': ['metadatas', 'documents', 'distances'],
 'data': None,
 'metadatas': [[None, None, None], [None, None, None]],
 'distances': [[0.817658543586731, 0.844660222530365, 1.2180488109588623],
  [0.653022050857544, 0.763019859790802, 0.8332760334014893]]}

Better results for the second question.

id10 — the sentence that actually defines pretraining — now comes first.

One caution on the numbers: distances are not comparable across models. The local model returned ~0.35 and this one returns ~0.65 for its best hit, but that does not mean it is worse. Every model has its own scale and its own notion of distance; compare rankings, not raw scores, when models differ.

Reranker Models

So we've fixed the language problem, and we now have a ranking that looks sensible. But it took trying two models to get there, and nothing guarantees the third query works as well.

The underlying weakness hasn't gone away. The embedding stage still compares the query against each document separately, through a lossy single vector. That's what makes it fast — the whole corpus can be indexed once, offline — and it's also what makes it approximate.

A reranker replaces that approximation with a slow, careful look:

Retrieve generously. Ask the embedding model for more candidates than you need — if you want 3, fetch 10.
Rerank. Hand the query and all candidates to the reranker together, and let it score each one by reading them side by side.
Keep the top K. Pass only the winners to the LLM.

Because the reranker sees query and document jointly, it can tell "mentions pretraining" from "explains pretraining" — exactly the distinction the embedding model blurred. The price is that it can't be precomputed: it must run on every query against every candidate, which is why it's used to reorder a shortlist rather than to search the corpus.

Rerankers are also much bigger. BAAI/bge-reranker-v2-m3 needs about 2.3 GB of disk — a lot to carry for a ranking refinement. So instead, we'll use a hosted one, qwen3.7-text-rerank.

That said, if privacy is a priority, the local option is probably the better choice — possibly the only one.

1. Collecting candidates

The embedding model and the reranker are completely independent — nothing requires them to come from the same vendor, or even to be the same kind of thing. Here, the retrieval comes from the local bge-small-zh-v1.5 and the reranking from the online qwen3.7-text-rerank.

We keep using collection1 from Option 1 (local deployment), and this time ask for 10 candidates instead of 3:

# 1. collect more candidates
res = collection1.query(
    query_texts=queries,
    n_results=10)

Examine the top-3 among the 10 results:

candidates, ids = [None]*2, [None]*2
for i in range(2):
    candidates[i] = res['documents'][i]
    ids[i] = res['ids'][i]   # for later use
    for j, s in enumerate(candidates[i][:3]):
        print(f'{i+1}.{j+1}: {s}')
    print('===')

Same results as before:

1.1: GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。
1.2: Transformer 使用自注意力来建模序列中词元之间的关系。
1.3: 大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。
===
2.1: 后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。
2.2: 预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。
2.3: GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。
===

Note which sentence is on top. ids is carried along in parallel with candidates, because the reranker returns positions into the list you handed it — you'll need those IDs to report results afterward.

2. Calling the reranker

The DashScope API doesn't follow the OpenAI format for reranking, so we need to use its own SDK in this step.

pip install dashscope

You also need to log into the Alibaba Cloud website and apply for a DashScope workspace ID — something like ws-xxx.

import dashscope
dashscope_workspace_id = 'ws-xxx'   # your own workspace ID

dashscope.base_http_api_url = f'https://{dashscope_workspace_id}.cn-beijing.maas.aliyuncs.com/api/v1'

# Don't do this!
#dashscope.base_http_api_url = 'https://dashscope.aliyuncs.com/compatible-mode/v1'

rerank_resp = [dashscope.TextReRank.call(
    model="qwen3.7-text-rerank",
    query=queries[i],
    documents=candidates[i],
    top_n=3) for i in range(2)]

Note the shape of the call: one query, many documents, and a top_n that decides how many survive. queries and candidates are reusing the variables from the previous section.

For those who wonder what happens if you point the SDK at the OpenAI-compatible endpoint instead:

dashscope.base_http_api_url = 'https://dashscope.aliyuncs.com/compatible-mode/v1'

The answer is that you get a 404 error:

ReRankResponse(status_code=404, ...)

Examining the output

for i in range(2):
    for item in rerank_resp[i].output.results:
        print(f"[{i}] [{item.relevance_score:.3f}] {ids[i][item.index]}. {candidates[i][item.index][:60]}...")
    print('===')

[0] [0.851] id6. Transformer 使用自注意力来建模序列中词元之间的关系。...
[0] [0.750] id8. GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transform...
[0] [0.509] id5. 神经网络是受大脑启发的计算系统，由相互连接的节点层组成，这些节点层学习从数据中识别模式并进行预测。...
===
[1] [0.985] id10. 预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。...
[1] [0.809] id8. GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transform...
[1] [0.758] id12. 后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。...
===

There it is. The definition sentence for "预训练" ("pretraining") is now first — a gap of 0.985 vs 0.809 over the runner-up, where the embedding model had them in the wrong order entirely.

Two details in that code are worth keeping in mind:

item.index is a position in candidates[i], so candidates[i][item.index] recovers the text and ids[i][item.index] recovers the Chroma ID. That parallel-list lookup is the only fiddly part of using a reranker.
The scores are not cosine similarities and are not comparable with the distances from either embedding model. They're the reranker's own relevance scale — only useful for ordering, and only within a single call.

Putting it together

The recipes above are the pieces; in practice they compose into one retrieval rule.

def retrieve(query, k=3):
    # 1. cheap, approximate recall over the whole corpus
    res = collection1.query(query_texts=query, n_results=max(k * 4, 10))
    candidates, ids = res['documents'][0], res['ids'][0]

    # 2. expensive, accurate reordering of the shortlist
    resp = dashscope.TextReRank.call(
        model="qwen3.7-text-rerank",
        query=query,
        documents=candidates, top_n=k)

    # 3. hand back documents and IDs, best first
    return [(ids[r.index], candidates[r.index], r.relevance_score)
            for r in resp.output.results]

Two heuristics are hidden in those numbers. The k * 4 (with a floor of 10) is the retrieval depth: fetch several times what you need, so that the reranker has something worth reordering, but not so much that you're paying for the whole corpus twice. And when there are fewer candidates than k, the reranker simply returns what it got — no special case needed.

Avoiding the DashScope SDK

If you want to get rid of the dashscope dependency, you can handroll your own POST:

def retrieve(query, k=3):
    # 1. cheap, approximate recall over the whole corpus
    res = collection1.query(query_texts=query, n_results=max(k * 4, 10))
    candidates, ids = res['documents'][0], res['ids'][0]

    # 2. expensive, accurate reordering of the shortlist
    import requests
    resp = requests.post(
        f'https://{DASHSCOPE_WORKSPACE_ID}.cn-beijing.maas.aliyuncs.com/api/v1/services/rerank/text-rerank/text-rerank',
        headers={
            'Content-Type': 'application/json',
            'Authorization': f'Bearer {DASHSCOPE_API_KEY}',},
        json={
            'model': 'qwen3.7-text-rerank',
            'input': {
                'query': query,
                'documents': candidates,
            },
            'parameters': { 'top_n': k, },
        },
        timeout=30)

    # 3. hand back documents and IDs, best first
    return [(ids[r['index']], candidates[r['index']], r['relevance_score'])
            for r in resp.json()['output']['results']]

The request body comes from the official API reference.

A few reasons to skip the DashScope SDK:

The install is large, and most of it has nothing to do with reranking. dashscope depends on many packages, aiohttp, cryptography, httpx, etc., that cost you about 31 MB of disk space, while requests costs nothing: Chroma already includes it via its dependencies.
The knobs stay in your hands. timeout=30 is in the snippet because you can set it.
You see what is on the wire. When the response shape shifts you get a missing dict key. Debugging an HTTP call you wrote is a shorter path than debugging a wrapper you did not.

The trade is typed access: resp.output.results vs. resp.json()['output']['results']. The code also gets messier. And the URL may change over time. If that is worth 31 MB, keep the SDK. I personally prefer the HTTP call.

Recap

The three options, and when each makes sense:

	Best for	Cost	Watch out for
Local model	privacy, offline use, tight budgets	~92 MB of disk	needs a deep-learning stack installed
Online model	fastest to a working demo, best quality per line of code	per-query fees, API key	vendor lock-in, network dependency
Reranker	any retrieval where ranking matters	a second model in the loop	latency — it runs on every query

The thread running through all of it: embeddings get you a shortlist, they don't get you an answer.

An embedding model has to compress a sentence into one vector, and it must do so before it ever sees your query — which is precisely why it's fast, and precisely why it confuses "mentions the topic" with "answers the question." A reranker fixes that by reading query and candidate together, at the cost of doing the work fresh each time. Retrieve broad, rerank narrow, then generate.

Swap BAAI/bge-small-zh-v1.5 for any other row in the model table and the code above is unchanged — including the all-MiniLM-L6-v2 you started with, back in its natural habitat of English text.

Micro-RAG

Cheng — Sat, 19 Sep 2026 12:17:53 +0000

Large language models (LLMs) have a problem: hallucination — from time to time, they state falsehoods with confidence.The reason is simple. Models know only what they read during training, hence nothing about today's weather, or your company's financial report this year.

RAG, short for Retrieval-Augmented Generation, is a technology designed to address exactly this problem.

The idea of RAG is simple: feed the model facts before letting it answer a question.

A typical pipeline has five steps:

Split the documents into chunks
Turn each chunk into a vector
Organize the chunks and vectors as an index
Before answering a question, retrieve the top-K most relevant chunks
Hand them to the LLM to generate the answer

That may sound like a lot. So we'll just walk you through the whole process with a short piece of Python code.

Note that our code cuts plenty of corners and is very much a toy model. This way, you can see the overall flow without getting bogged down in details.

1. Preparing the Corpus

First, we need a knowledge base. Every question that follows is answered against it.

For simplicity, we'll organize it as a list of strings — one string per document.

docs = [
    "Python is a popular programming language for data processing, machine learning, and AI in general.",
    "Data processing involves cleaning, transforming, and aggregating raw data into a useful format.",
    "Machine learning trains models to find patterns in data and make predictions on new examples.",
    "Deep learning uses neural networks with many layers to learn hierarchical representations.",
    "Neural networks are computing systems inspired by the brain, made of interconnected layers of nodes that learn to recognize patterns and make predictions from data.",
    "Transformers use self-attention to model relationships between tokens in a sequence.",
    "Attention lets a model weigh different parts of the input when producing each output representation.",
    "GPT stands for Generative Pre-trained Transformer, a decoder-only Transformer model trained to predict the next token.",
    "Large language models (LLM) are neural networks trained on massive text corpora to predict the next token.",
    "Pretraining learns general representations from large unlabeled data before task-specific fine-tuning.",
    "Fine-tuning adapts a pretrained model to a specific task or domain using labeled or instruction data.",
    "Post-training adapts a pretrained model with instruction tuning, preference optimization, or safety alignment.",
    "Retrieval-Augmented Generation (RAG) combines retrieval from an external knowledge source with generation from an LLM.",
    "A basic RAG pipeline chunks documents, embeds the chunks, stores them in a vector index, retrieves top-k chunks, and passes them to the LLM.",
    # rest omitted
]

Don't worry about what these sentences say. They were generated by an LLM anyway.

So docs is about the most bare-bones "knowledge base" you can imagine. In real life it might be a several-hundred-page product manual — but the idea is the same.

One more thing: our documents are already short, so we skip the first step of an ordinary RAG pipeline — chunking.

2. Tokenization — Splitting Sentences into Words

Computers don't know sentences or articles. They only know numbers. So the first step is to split a sentence into tokens. Later we'll treat these tokens as the dimensions of a high-dimensional space, and a sentence or an article becomes a vector in that space.

For simplicity, let's treat each token as an English word and write the simplest tokenizer we can:

import re
def tokenize(doc):
    return re.findall(r"[a-z]+", doc.lower())

Code notes

doc.lower() turns the text into lowercase first.
The regular expression [a-z]+ means "a run of consecutive lowercase letters" — whatever matches is treated as one token.
To keep things simple, digits and punctuation are both dropped.
re.findall() finds every piece that matches the pattern and returns a list.
And that's the splitting, in effect.

With a different corpus, though, dropping punctuation could change the meaning. Tokenization strategy is itself a trade-off.

A small example

Run it on one sample, say tokenize(docs[1]):

"Data processing involves cleaning, transforming, ..."

and you get

['data', 'processing', 'involves', 'cleaning', 'transforming', ...]

3. The Vocabulary — Giving Every Word an ID

def build_vocab(docs):
    blacklist = ['is', 'are', 'was', 'were', 'a', 'an', 'the',
        'and', 'or', 'of', 'to', 'in', 'on', 'for', 'from', 'with', ]
    vocab = set([w for doc in docs for w in tokenize(doc) if w not in blacklist])
    return sorted(vocab)

vocab = build_vocab(docs)
word_to_id = {w: i for i, w in enumerate(vocab)}

Code notes

This code does two things:

Build the vocabulary: walk every document's words, collect them, and toss out the low-information ones (the blacklist); then sort what's left into a word list.
Build the ID table: word_to_id is a dict that maps each word to its position in the vocabulary (0, 1, 2, …).

The blacklist holds articles, the verb "to be", and a few other words of that kind. They turn up constantly yet carry almost no meaning. Keep them, and two completely unrelated sentences will be judged "similar" just for sharing a couple of throwaway words like of and to.
Once they're filtered out, every remaining word in this corpus has an ID of its own.

4. Word Vectors — Representing a Passage as a Set of Numbers

Now the main event: how do we turn a piece of text into a vector?

The scheme here is the classic one, and the simplest — bag of words.

Bag of words treats each word as one dimension of a high-dimensional space.
The vocabulary length is the number of dimensions.
So each word is, in effect, a unit vector along one axis of that space.

How do we use that to represent a passage?

Whichever words appear in the passage, we add up their unit vectors, and the sum represents the whole passage.
Under this scheme, if a word appears N times in the passage, the sum has a magnitude of N in that dimension — not 1.

from collections import Counter

def embed(doc):
    ''' compute the word vector '''
    emb_vec = [0] * len(vocab)          # start with a long all-zero vector
    for w, count in Counter(tokenize(doc)).items():
        if w in word_to_id:
            emb_vec[word_to_id[w]] = count   # write the count into that word's slot
    norm = sum(x**2 for x in emb_vec)**0.5 or 1.0
    return [x / norm for x in emb_vec]       # normalize

Code notes

First, Counter. It counts things: Counter([3, 1, 4, 1, 5]) hands back something dict-like — {3: 1, 1: 2, 4: 1, 5: 1} — telling you how many times each element appeared.
So the logic of embed() is plain: the value in dimension i is how many times the i-th word appears in this passage. The more often a word shows up, the more weight it carries — and the more it contributes to the similarity score.
The or 1.0 at the end of the second-to-last line is a safety net. If none of a text's words are in the vocabulary, the vector is all zeros, its norm is 0, and the division blows up. In Python, 0.0 or 1.0 is 1.0 — one little "or" dodges the divide-by-zero.
Why bother with the safety net? Words from the knowledge base are in the vocabulary, of course — no need there. But we also run this function on the user's question, and the user's words are not guaranteed to be in the vocabulary.
One more thing: every component of this vector is non-negative, and it has been normalized (last line), so the dot product of two vectors is exactly the cosine similarity (see below).
Normalization also keeps the comparison fair between sentences of different lengths: it stops a longer sentence from gaining an edge just by having more words — and hence a larger dot product.

5. Similarity — How Alike Are Two Passages?

With vectors in hand, we can measure how "far apart" two passages are:

def dot(a, b):
    ''' cosine similarity '''
    return sum(x * y for x, y in zip(a, b))

Code notes

The dot product of two vectors (given both are normalized) is the cosine similarity. The intuition is simple: the more words two passages share, the larger the dot product.

Maximum 1: the two texts point in exactly the same direction — in bag-of-words terms, identical word-count distributions.
Minimum 0: the two texts share no vocabulary at all. Word counts can't be negative, so the dot product never drops below 0.

An example

Start with the extremes. In this corpus, the most similar pair is #10 and #11 (counting from zero — the Fine-tuning and Post-training entries). They share the content words tuning, pretrained, model, and instruction, for a similarity of 0.44. The least similar pairs share no words at all and sit squarely at 0.

But look only at the most similar pair, and it's easy to believe the system "understood" something. In fact, the whole approach has a problem.

For instance, #5 is about Transformer*s* (plural). Ask about the singular transformer, and it comes up empty just the same.

More word overlap, higher score; words fail to line up, the score collapses — even when both passages are about the same thing.

So this version of vectors and similarity doesn't "understand" meaning. It's just counting word overlap.

6. Building the Index — Precomputing Every Document

index = [(doc, embed(doc)) for doc in docs]

The index is a table of "each document ↔ its vector". With it, comparing two articles costs a single dot product — no re-tokenizing, no re-counting.

Think of it as a library's card catalog: every book's summary is written up in advance, so at lookup time you just flip through the cards.

7. Retrieval — Finding the k Most Relevant Documents

def retrieve(query, k=3, threshold=0.0):
    q_vec = embed(query)
    doc_scores = [(doc, dot(q_vec, vec)) for doc, vec in index]
    doc_scores.sort(key=lambda x: x[1], reverse=True)
    return [(doc, score) for doc, score in doc_scores[:k] if score > threshold]

Code notes

It goes like this:

Turn the query into a vector.
Score it against every document in the store.
Sort by score, highest first.
Take the top k, dropping anything below the threshold.

Note that retrieve() does not guarantee k documents. The threshold gate filters out irrelevant ones; if no document scores above it, you get an empty list. That's deliberate — a missing answer beats a forced one.

8. Generation — Feeding What We Found to the LLM

def dumb_llm(prompt):
    m = re.search(r'CONTEXT:\s+(.*?)\s+QUESTION:', prompt, re.DOTALL)
    return m.group(1) if m and m.group(1) else "Sorry, I have no idea."

def ask(query, hits):
    context = '\n'.join(f"- {doc}" for doc, score in hits)
    prompt = f"""Answer according to the following context:

CONTEXT:
{context}

QUESTION:{query}
"""
    return dumb_llm(prompt)

Code notes

ask() does the one thing every RAG system does: stuff the retrieved results into the prompt as "context" and hand it to the model.
dumb_llm() is a stand-in toy. It generates nothing at all; it just uses a regular expression to copy the CONTEXT block back out verbatim. In a real system, this is where GPT or DeepSeek sits — it reads the context and puts an answer together in its own words.

An example

Try it:

ask('What is a neural network?', [(docs[4], 0.9), (docs[3], 0.5)])

It will dutifully splice those two entries into the context, and dumb_llm() spits them straight back out.

This toy actually points at something essential: retrieval sets the quality ceiling of RAG. Hand the LLM the wrong material, and even a strong model can only answer wrong — or refuse with that Sorry, I have no idea.

9. The Complete RAG

def rag(query):
    hits = retrieve(query, k=5)
    ans = ask(query, hits)
    return ans, hits

Driver code:

import sys

if __name__ == '__main__':
    if len(sys.argv) > 1:
        query = ' '.join(sys.argv[1:])
        ans, hits = rag(query)
        print(f'Q: {query}\nA: {ans}')
        print('\n=============')
        for doc, score in hits:
            print(f'[{score:.2f}] {doc}')

Four steps fold into one: retrieve → assemble → generate → return. Ask it What is RAG? and it returns the two RAG entries from the corpus, scored 0.26 and 0.20 — not high, since the question is short and shares few words with the documents, but enough to push the right material to the top.

Now ask about something the corpus has nothing to say — say, Why do cats chase rats? There is no cat and no rat in the vocabulary, so the query vector is nearly all zeros; no document scores above the threshold, retrieval returns an empty list, and what you finally see is:

Sorry, I have no idea.

Empty retrieval, an honest refusal — this is exactly where RAG beats letting the model wing it.

10. From Toy to the Real World

The RAG here is a teaching version. A production system upgrades every stage, but the skeleton stays the same:

Stage	Teaching version	Real version
Tokenization	regex word splitting	subword tokenization (BPE / SentencePiece)
Vectorization	bag-of-words counts	neural embeddings (OpenAI Embeddings, BERT, and so on)
Similarity	dot product	cosine / inner product + approximate nearest-neighbor index (FAISS, HNSW)
Retrieval	full scan of the store	vector databases (Pinecone, Milvus, pgvector)
Generation	a toy regex	an actual LLM, plus citations and reranking

The design choices planted along the way — stop-word filtering, normalization, threshold filtering, refusal — all have counterparts in industry. A few dozen lines, and the backbone of RAG is clear.

Closing

The idea behind RAG is simple: don't expect a model to know everything. Teach it to look things up.

Retrieval finds the right information.

Generation puts it into good prose.

Next time you ask an AI assistant about something that happened after its training cutoff, and it gives you an accurate answer, there's a good chance a pipeline like this is running behind it — from a list of strings to a program that knows how to look things up.

An Old Trick in Networking, Rediscovered by AI

Cheng — Fri, 18 Sep 2026 08:11:46 +0000

Open a chat interface and ask it to write you a function, and you'll get an answer in a few seconds — maybe it works, maybe it doesn't. Either way, the moment the model hands you an answer, the transaction ends.

Now try the same request in a coding agent like Claude Code or OpenAI's Codex. Neither just hands you a block of text and walks away. It writes the code, runs it, reads whatever error comes back, makes some changes, and runs it again. Only after five or ten rounds -- when it finally works -- does it show you the result.

Even with the same underlying model, reliability differs dramatically. The difference has a fancy name: the agentic loop.

But it turns out this exact trick — turning something unreliable into something reliable by trying, checking, and trying again — was invented decades before any of this existed, for a technology from a completely different domain -- networking, but for exactly the same reason.

One protocol, then two: how IP and TCP came to be

You may not come from a background of networking, but chances are you've heard the term "TCP/IP" -- the pair of protocols quietly handling just about all of your everyday internet traffic. What's less well known is that they didn't start out as a pair. They were one protocol, and the reason they split apart turns out to be the missing piece of this whole story.

In 1974, Vint Cerf and Bob Kahn (both are recipients of the 2004 Turing Award) published the design for a single protocol called TCP — at the time, short for Transmission Control Program (not Protocol yet). It did everything: addressed every packet, moved it across the network, and guaranteed it arrived complete and in the right order. For the next few years, that was the entire plan.

Then real-time applications showed up — voice, video, radar tracking — and exposed a problem nobody had designed around. These applications did not want TCP's guarantee. A phone call that pauses to retransmit a lost syllable is worse than one that just drops it and moves on; waiting for certainty was actively harmful when speed mattered more. But the original TCP offered no "fast and occasionally lossy" plan — full reliability was baked into every packet, non-optional.

In the spring of 1978, at a meeting at USC's Information Sciences Institute, Cerf, Jon Postel, and Danny Cohen resolved this by cutting the protocol in two. The bottom layer kept only the minimum job and got a new name: IP. Its task was just to get an addressed packet somewhere, best-effort, no promises. The top layer kept the name TCP, now much narrower: acknowledgments, retransmission, ordering — all the guarantee logic, but optional, layered on top instead of forced onto every packet. From that point on, applications could choose: use IP alone and accept some loss for speed, or add TCP on top and accept some delay for certainty.

To carry forward, we just need to remember this:

IP is one-shot best-effort delivery with no guarantee.
TCP adds, on top of that, reliability by retry-until-confirmed.

The two exist as separate layers specifically so you can choose which one you actually need.

The split wasn't just an engineering cleanup — it was an admission that "guaranteed correct" and "fast" are sometimes in tension, and the right answer depends on the task. That's exactly the choice AI tools hand you today, just dressed up in words and code instead of packets.

The same trick, dressed differently

Going to the opening example, it is easy to see the parallel.

A one-shot answer from a chat interface is a single forward pass: you ask the model a question, it gives you an answer, and that's final -- right or wrong, no checking, no second attempt. That's IP: get something out the door, with no guarantee attached.

Claude Code or Codex working in a loop, however, is TCP wearing different clothes. Take an action, observe what happened -- did the tests pass? did the tool call error out? — and if it failed, try again (perhaps with a somewhat different approach), until it succeeds or hits a retry limit. Send, check for acknowledgment, retry on failure, give up after a timeout: same structure, just applied to code instead of packets.

So the reliability gain from looping isn't magic -- it's the exact same trick TCP uses to turn "best-effort" into "reliable."

Where the analogy strains

It's worth being honest about the limits, because that's where the real texture is.

TCP's retry is dumb: it resends the identical packet, and success is judged by a binary signal — a checksum either matches or it doesn't.

An agent's retry is not that. When code fails, the model reads the actual error message, reasons about why it failed, and typically tries something different the second time.

The reliability-via-retries structure matches, but the intelligence level does not.

This idea shows up twice

Here's the part that's easy to miss. The story so far is about how a finished model is used. But the same best-effort-then-corrected pattern also describes how the model was built in the first place.

Pretraining is the stage where large language models cram the entire corpus of Internet text into their weights. The model works by doing only one thing: predict the next token and try to make it match what the Internet says it should be. No double-checking of any sort. The errors we feed it also come out ready-made: deprecated API calls lingering in old GitHub repositories, Stack Overflow answers that looked plausible but were actually wishful thinking, off-by-one errors in textbook examples, mathematical "proofs" with sign errors, etc. A model trained on all of this simply cannot tell a buggy pattern from a correct one, as long as both show up enough often in the text. That's best-effort, IP-style: fast, with no guarantee of correctness.

Post-training is where corrections get introduced. The job is to go back and break the model out of its ruts. For coding and reasoning tasks, it works by asking the model to generate its own attempts, then checking those attempts against ground truth--Does the code pass its tests? Does the math check out?--and adjusting the model weights to narrow the gap between success and failure. Some approaches even reintroduce a dedicated critic model: a second network whose whole job is to judge whether each step in a long attempt checks out. That's the TCP mindset applied to training: take a best-effort attempt, check it, correct it, repeat.

Same principle, two timescales

So the connecting thread isn't a coincidence — it's the same idea, applied at two different points in time, with the bill just arriving on a different schedule.

At runtime, retries happen live: every task pays its own cost, and whatever gets corrected only applies to that one output.

At training time, the cost of retries is paid up front by whoever trained the model, and what gets corrected is a general tendency, baked permanently into the weights.

That's why a model that's been through solid post-training usually doesn't need to loop at all for tasks resembling the ones it trained on — the retrying already happened, exhaustively, in advance.

Put simply: it no longer needs to "retry until it works" because it has practiced so many similar problems that it can just read the answer off the top of its head.

Where the whole analogy runs out

A couple of honest caveats, so this doesn't overclaim.

TCP/IP layers are strictly stacked — every packet passes through all four layers, every time, in the same order. Pretraining and post-training aren't like that; they're sequential, one-time training phases, not a repeating per-request pipeline.

And post-training's correction is narrower than it might sound. It works best where correctness is checkable — code that runs, mathematical proofs that checks out. However, it doesn't generally clean up factual errors or misconceptions the model absorbed from bad sources during pretraining, since there's no automatic verifier for "is this historical claim true." That's a separate, harder problem, and one reason models can still confidently state incorrect facts even after heavy post-training.

A couple of open questions

Two things this raises that don't have clean answers yet.

As post-training keeps improving, will agentic loops eventually become unnecessary even for genuinely novel problems -- or will there always be some type of task too new for any amount of training-time practice to have anticipated, meaning the live retry loop never fully goes away?

Another question. Is there an equivalent of UDP for AI -- a deliberately no-retry, good-enough-and-fast mode, chosen on purpose for situations where speed matters more than certainty, the same way real-time voice chose plain IP over TCP? Live captioning or real-time translation might be exactly this case: a slightly wrong word now beats a perfect one two seconds late.

The takeaway

Reliability in a complex system is never free -- it's always bought with retries somewhere.

Network architecture figured this out decades ago: IP is honest about being unreliable, and TCP is the layer that buys back reliability through repetition and acknowledgment. AI systems are rediscovering the same trade-off at two different timescales -- once permanently, during training, and once transiently, every time an agent loops at runtime. The interesting engineering question, in either domain, was never whether to pay for reliability. It's when, and how much of it you actually need.