Making Chroma Work in Chinese

Note:

1. This tutorial picks up where Chroma's official quick start guide leaves off.
2. This article was revised after feedback from Ahmet Özel (see the Baseline section for the change he proposed).

So you've finished Chroma's quick start — congratulations. Everything works, right up until you feed Chroma some Chinese text.

Then it doesn't.

Chroma's default embedding model, all-MiniLM-L6-v2, is English-only. Given Chinese or Japanese input, it embeds it as though the language were noise. You need a model that was trained on the language you care about.

You have two ways to get one:

	Local model	Online model
Runs where	on your machine	vendor's server
Network	not needed	required
Setup	install packages, download weights	an API key
Cost	disk space (hundreds of MB)	money per query
Offline	works	does not
Privacy	good	bad

This tutorial walks through both, and then a third option that fixes a problem neither of them solves: ranking.

To be concrete, we'll assume Chinese is the language you need to add. Swap the model name and the same code supports any language.

Baseline of the Default Model

Let's first establish a baseline: how well does Chroma's default model actually do on Chinese?

The first step is to create a client and a collection.

import chromadb
client0 = chromadb.Client()
collection0 = client0.create_collection(name='demo0')

We did not pass a custom embedding function to create_collection(), so it uses the default all-MiniLM-L6-v2.

Next come the documents that will serve as our knowledge base:

docs = [
    'Python 是一种流行的编程语言，广泛用于数据处理、机器学习以及广义上的人工智能。',
    '数据处理包括将原始数据清洗、转换和聚合成有用的格式。',
    '机器学习通过训练模型来发现数据中的模式，并对新样本进行预测。',
    '深度学习使用具有多层的神经网络来学习分层表示。',
    '神经网络是受大脑启发的计算系统，由相互连接的节点层组成，这些节点层学习从数据中识别模式并进行预测。',
    'Transformer 使用自注意力来建模序列中词元之间的关系。',
    '注意力机制让模型在生成每个输出表示时，对输入的不同部分进行加权。',
    'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
    '大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。',
    "预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。",
    "微调使用有标注数据或指令数据，使预训练模型适应特定任务或领域。",
    "后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。",
    "向量嵌入将文本、图像或其他数据映射到高维空间中的点，以进行相似度搜索。",
    "检索增强生成（RAG）将来自外部知识源的检索与大语言模型（LLM）的生成相结合。",
    "一个基础的 RAG 流程会切分文档、对文本块进行嵌入、将其存入向量索引、检索前 k 个文本块，并将它们传给 LLM。",
]

A few things to note about this corpus:

• Each item of docs represents a document, albeit a short one with a single sentence.
• The sentences are mostly Chinese, except a few technical terms that are deliberately English.
• Each sentence is about a single idea, e.g., the first one is the definition of 'Python', the sixth about 'Transformer', etc.
• The central idea of the sentence usually appears at the beginning. All sentences share the structure of '{CENTRAL IDEA} is/does ...'.
• So, for what-is questions, you get one or two most relevant documents serving as their ground truth.
  • For example, if you ask 'What is Transformer?', it should give you the sixth sentence, which contains 'Transformer' as its central idea and serves as the query's ground truth.
  • Similarly, if you ask about RAG, the last two sentences are most relevant, hence the ground truth.

We will use this set of documents throughout this article to compare the performance of different embedding models.

Now add these documents to the collection:

collection0.add(ids=[f'id{i+1}' for i in range(len(docs))], documents=docs)

The part [f'id{i+1}' for i in range(len(docs))] looks a bit busy, but it simply expands to a list of automatically-numbered IDs: 'id1', 'id2', ....

Now let's prepare two typical questions:

queries = [
    '什么是Transformer？',  # 'What is Transformer?'  Most relevant: 'id6'
    '什么是预训练？',        # 'What is pretraining?'  Most relevant: 'id10' 
]

• These are what-is questions deliberately designed to match two central ideas of docs.
• We will fix the questions in all of our experiments.
• One caveat: the thing we ask about is English ('Transformer') in the first case, but Chinese ('预训练') in the second.

Now actually ask the model the two questions:

collection0.query(query_texts=queries, n_results=3)

The output is

{'ids': [['id6', 'id8', 'id12'], ['id12', 'id11', 'id7']],
 'embeddings': None,
 'documents': [['Transformer 使用自注意力来建模序列中词元之间的关系。',
   'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
   '后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。'],
  ['后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。',
   '微调使用有标注数据或指令数据，使预训练模型适应特定任务或领域。',
   '注意力机制让模型在生成每个输出表示时，对输入的不同部分进行加权。']],
 'uris': None,
 'included': ['metadatas', 'documents', 'distances'],
 'data': None,
 'metadatas': [[None, None, None], [None, None, None]],
 'distances': [[0.32853031158447266, 0.7125810384750366, 1.0673757791519165],
  [0.7402779459953308, 0.7472522854804993, 0.8964452743530273]]}

First, we notice that the responses are not answers to the questions, but instead collections of relevant documents.

For the first question about 'Transformer', the model selects 'id6' (ground truth), 'id8', 'id12'. So we may think that Chroma's default embedding is fine for Chinese.

But not so fast. The keyword 'Transformer' in this case is English, which lies in the comfort zone of the default embedding.

Now for the second question, in which we ask about an idea in Chinese ('预训练', or 'pretraining'). The ground truth is 'id10'. But the default model failed to identify it in the top-3 relevant items.

This shows the default model cannot handle Chinese properly. And this is why we need to use a custom embedding model.

What is more concerning (thanks to Ahmet Özel for raising this) is that the model did not throw a warning or notice in the second case. This kind of garbage output can readily sneak into production.

Custom Embedding Model

Whatever embedding model you choose, the hook into Chroma is the same: the optional embedding_function argument of client.create_collection().

# Replace with your own class
ef = YourEmbeddingFunction(...)

collection = client.create_collection(
    name="my_collection",
    embedding_function=ef)

That's the whole interface. The rest of this tutorial is about what to put in place of YourEmbeddingFunction.

Option 1: Local Models

The first option is about deploying the embedding model locally: you download the model to your disk, and load its weights from your disk. No network connection needed.

Choosing a model

Assuming you want Chinese support from a local model, these are the usual candidates:

Model	Language	Dim.	Notes
paraphrase-multilingual-MiniLM-L12-v2	Multilingual	384	lightweight
BAAI/bge-small-zh-v1.5	Chinese	512	good for Chinese
BAAI/bge-base-zh-v1.5	Chinese	768	same family, stronger
BAAI/bge-m3	Multilingual	1024	multilingual
text2vec-base-chinese	Chinese	768
m3e-base	Chinese	768
all-MiniLM-L6-v2	English	384	Chroma's default (for reference)

Roughly:

More dimensions means better quality and a bigger download.

• The smallest multilingual option is only 384-dimensional and stays under 500 MB.
• bge-m3 is 1024-dimensional and will cost you well over 2 GB.
• For Chinese specifically, bge-small-zh-v1.5 is the cheapest thing that works properly, so that's what we'll use below.

Installing the prerequisite package

To use a local model you need one extra Python package, sentence-transformers. Chroma does not ship it by default, for a good reason:

• The model is loaded through embedding_functions.SentenceTransformerEmbeddingFunction.
• That class needs the sentence-transformers package.
• Which in turn pulls in a string of other packages:
  • torch (PyTorch)
  • transformers
  • huggingface-hub
  • tokenizers
  • numpy, scipy and friends.

That's pretty much a full deep-learning stack — hundreds of MB of dependencies. If your environment already has the stack, fine. If your project is, say, a small web service that just happens to need a vector store, that deep-learning stack is a bit of overkill. So Chroma makes you opt in:

pip install sentence-transformers

Downloading the model

Now we can actually build the embedding function.

The first time you construct SentenceTransformerEmbeddingFunction('BAAI/bge-small-zh-v1.5'), it downloads the weights from huggingface.co, where the model lives by default.

If you live in China, or anywhere the connection to huggingface.co is poor, point it at a mirror first. On Linux:

export HF_ENDPOINT="https://hf-mirror.com"

On Windows PowerShell:

$env:HF_ENDPOINT="https://hf-mirror.com"

On Windows CMD:

set HF_ENDPOINT=https://hf-mirror.com

Or, if you prefer setting it from inside Python — often the most convenient — use os.environ['VAR'] = ...:

from chromadb.utils import embedding_functions

import os
os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
        model_name='BAAI/bge-small-zh-v1.5')

The output is something like this:

Loading weights: 100%|███████████████████████████████████████████████████████████████| 71/71 [00:00<00:00, 9559.74it/s]

What to notice in that output:

• The last call takes a while, especially the first time, to download the model and load its weights.
• The progress bar above is about loading the weights, not about downloading the model.
• Run it a second time and the download is skipped, so startup gets noticeably faster.
• Loading the weights still takes a moment every session — don't expect a split-second import.

By default the model lands in ~/.cache/huggingface/hub on Linux, or C:\Users\<you>\.cache\huggingface\hub on Windows.

Caching the model inside your project

A user-level cache is convenient, but it makes the project non-portable: a teammate clones the repo, and still has to download ~92 MB before anything runs. Downloading into the project directory instead fixes that — one hf command does it (the same tool was previously called huggingface-cli):

$env:HF_ENDPOINT="https://hf-mirror.com"
pip install "huggingface_hub[cli]"
hf download BAAI/bge-small-zh-v1.5 --local-dir ./models/bge-small-zh-v1.5 --exclude "*.bin"

By --exclude "*.bin", you skip the legacy PyTorch weight files and avoid storing the same weights twice—cutting disk usage roughly in half.

After this command, ./models/bge-small-zh-v1.5 contains roughly 92 MB of files.

Then point model_name at the local path. The leading ./ is what tells SentenceTransformerEmbeddingFunction this is a directory and not a HuggingFace model ID:

from chromadb.utils import embedding_functions
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
        model_name="./models/bge-small-zh-v1.5")

The output:

Loading weights: 100%|██████████████████████████████████████████████████████████████| 71/71 [00:00<00:00, 14191.55it/s]

You can tell it worked: the loading is faster (no network round-trip to resolve the model).

Peeking at the source (Optional)

Curious about what exactly a sentence transformer is? Take a look at the source code:

import inspect
print(inspect.getsource(embedding_functions.SentenceTransformerEmbeddingFunction))

The output is long, so only the beginning matters here:

class SentenceTransformerEmbeddingFunction(EmbeddingFunction[Documents]):
    # Since we do dynamic imports we have to type this as Any
    models: Dict[str, Any] = {}

    # If you have a beefier machine, try "gtr-t5-large".
    # for a full list of options: https://huggingface.co/sentence-transformers, https://www.sbert.net/docs/pretrained_models.html
    def __init__(
        self,
        model_name: str = "all-MiniLM-L6-v2",
        device: str = "cpu",
        normalize_embeddings: bool = False,
        **kwargs: Any,
    ):
        ...

Two things worth noticing.

• The default model_name is Chroma's English default — which is the whole reason this tutorial exists.
• And there's a device argument: pass "cuda" to move embedding onto a GPU if you have one. If you're familiar with PyTorch, this is the same thing as torch.device("cuda").

Using the local model

Time to put it to work. If you've run this before, the collection already exists and Chroma will refuse to create it again — hence the commented-out cleanup line, which you'll want in a notebook where cells get re-run:

# if you've created the collection before, uncomment and run the next line
#client1.delete_collection(name='demo1')

import chromadb
client1 = chromadb.Client()
collection1 = client1.create_collection(
        name="demo1",
        embedding_function=ef)

Now add the documents to the collection:

collection1.add(ids=[f'id{i+1}' for i in range(len(docs))], documents=docs)

And ask the questions:

collection1.query(query_texts=queries, n_results=3)

The return value looks like this:

{'ids': [['id8', 'id6', 'id9'], ['id12', 'id10', 'id8']],
 'embeddings': None,
 'documents': [['GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
   'Transformer 使用自注意力来建模序列中词元之间的关系。',
   '大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。'],
  ['后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。',
   '预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。',
   'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。']],
 'uris': None,
 'included': ['metadatas', 'documents', 'distances'],
 'data': None,
 'metadatas': [[None, None, None], [None, None, None]],
 'distances': [[0.3059161305427551, 0.36248522996902466, 0.4628119468688965],
  [0.34848546981811523, 0.39471912384033203, 0.4147259593009949]]}

Assessing the results

Question 1: 'What is Transformer?'

• The top two hits, id6 and id8, are both genuinely about Transformers. Good.
• The third hit, id9, is about LLMs, which was less relevant.
• The first two hits got relatively low distances, 0.31 and 0.36, much less than the third's 0.46. Good.

Question 2: 'What is pretraining?'

There is a huge improvement over the default embedding model.

The ground truth 'id10' is now included in the top-3 list.

Yet the ground truth is not ranked first. So there is still room for improvement.

What pushes id12 above id10 has something to do with the nature of vector embedding — this is just my guess. An embedding model compresses a whole sentence into a single vector, and the vector cares more about what words appear than what roles they play in the sentence. Both items mention "预训练" (pretraining); but id12, the item mainly about post-training, edges out because it has the token "训练" (training) appearing twice, while the ground truth id10 uses it only once. Same story for id8. Nothing in the embedding stage distinguishes "mentions pretraining" from "explains pretraining" — that distinction requires reading the query and the candidate together, which is a different architecture. We'll come back to it in the last section.

Option 2: Online Models

If you'd rather not download models, a hosted embedding API is the alternative. The trade-offs are the ones from the table at the top: you need an API key, and you pay per call, but there's nothing to install and the model is somebody else's problem to keep up to date.

Writing the embedding function

Chroma doesn't know about your vendor, so you need to write a small adapter. The reason DashScope does not provide an official adapter for Chroma is, I would guess, that there are simply too many vector search solutions out there to cover. All the adapter needs is a class with a __call__ method that maps a list of strings to a list of vectors.

Here we use Alibaba's DashScope, whose API is OpenAI-compatible — which means the official openai client works as-is, with only the base_url changed:

import os
from openai import OpenAI
from chromadb.api.types import EmbeddingFunction

class DashScopeEmbeddingFunction(EmbeddingFunction):
    def __init__(self, model='qwen3.7-text-embedding-flash'):
        self.model = model
        self.client = OpenAI(
            base_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
            api_key=os.getenv("DASHSCOPE_API_KEY"))

    def __call__(self, input: list[str]):
        response = self.client.embeddings.create(
            model=self.model, input=input)
        return [item.embedding for item in response.data]

A few notes on the adapter:

• The API key is read from the DASHSCOPE_API_KEY environment variable when you don't pass one in. Never hard-code it in a notebook you might share.
• __call__ receives a batch of texts and returns one vector each — Chroma always batches, so don't loop over the input and make one request per document.
• The returned objects use .embedding and .index; if you ever need to be defensive about ordering, sort by index before returning.

Then build a second collection on top of it:

#Run this if the next cell complains that the collection 'demo2' already exists
#client2.delete_collection(name='demo2')

import chromadb
client2 = chromadb.Client()

ds_ef = DashScopeEmbeddingFunction()
collection2 = client2.create_collection(
        name='demo2',
        embedding_function=ds_ef)

The same documents, the same questions — only the model changed:

collection2.add(ids=[f'id{i+1}' for i in range(len(docs))], documents=docs)

collection2.query(query_texts=queries, n_results=3)

The output:

{'ids': [['id8', 'id6', 'id3'], ['id10', 'id12', 'id8']],
 'embeddings': None,
 'documents': [['GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
   'Transformer 使用自注意力来建模序列中词元之间的关系。',
   '机器学习通过训练模型来发现数据中的模式，并对新样本进行预测。'],
  ['预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。',
   '后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。',
   'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。']],
 'uris': None,
 'included': ['metadatas', 'documents', 'distances'],
 'data': None,
 'metadatas': [[None, None, None], [None, None, None]],
 'distances': [[0.817658543586731, 0.844660222530365, 1.2180488109588623],
  [0.653022050857544, 0.763019859790802, 0.8332760334014893]]}

Better results for the second question.

id10 — the sentence that actually defines pretraining — now comes first.

One caution on the numbers: distances are not comparable across models. The local model returned ~0.35 and this one returns ~0.65 for its best hit, but that does not mean it is worse. Every model has its own scale and its own notion of distance; compare rankings, not raw scores, when models differ.

Reranker Models

So we've fixed the language problem, and we now have a ranking that looks sensible. But it took trying two models to get there, and nothing guarantees the third query works as well.

The underlying weakness hasn't gone away. The embedding stage still compares the query against each document separately, through a lossy single vector. That's what makes it fast — the whole corpus can be indexed once, offline — and it's also what makes it approximate.

A reranker replaces that approximation with a slow, careful look:

1. Retrieve generously. Ask the embedding model for more candidates than you need — if you want 3, fetch 10.
2. Rerank. Hand the query and all candidates to the reranker together, and let it score each one by reading them side by side.
3. Keep the top K. Pass only the winners to the LLM.

Because the reranker sees query and document jointly, it can tell "mentions pretraining" from "explains pretraining" — exactly the distinction the embedding model blurred. The price is that it can't be precomputed: it must run on every query against every candidate, which is why it's used to reorder a shortlist rather than to search the corpus.

Rerankers are also much bigger. BAAI/bge-reranker-v2-m3 needs about 2.3 GB of disk — a lot to carry for a ranking refinement. So instead, we'll use a hosted one, qwen3.7-text-rerank.

That said, if privacy is a priority, the local option is probably the better choice — possibly the only one.

1. Collecting candidates

The embedding model and the reranker are completely independent — nothing requires them to come from the same vendor, or even to be the same kind of thing. Here, the retrieval comes from the local bge-small-zh-v1.5 and the reranking from the online qwen3.7-text-rerank.

We keep using collection1 from Option 1 (local deployment), and this time ask for 10 candidates instead of 3:

# 1. collect more candidates
res = collection1.query(
    query_texts=queries,
    n_results=10)

Examine the top-3 among the 10 results:

candidates, ids = [None]*2, [None]*2
for i in range(2):
    candidates[i] = res['documents'][i]
    ids[i] = res['ids'][i]   # for later use
    for j, s in enumerate(candidates[i][:3]):
        print(f'{i+1}.{j+1}: {s}')
    print('===')

Same results as before:

1.1: GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。
1.2: Transformer 使用自注意力来建模序列中词元之间的关系。
1.3: 大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。
===
2.1: 后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。
2.2: 预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。
2.3: GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。
===

Note which sentence is on top. ids is carried along in parallel with candidates, because the reranker returns positions into the list you handed it — you'll need those IDs to report results afterward.

2. Calling the reranker

The DashScope API doesn't follow the OpenAI format for reranking, so we need to use its own SDK in this step.

pip install dashscope

You also need to log into the Alibaba Cloud website and apply for a DashScope workspace ID — something like ws-xxx.

import dashscope
dashscope_workspace_id = 'ws-xxx'   # your own workspace ID

dashscope.base_http_api_url = f'https://{dashscope_workspace_id}.cn-beijing.maas.aliyuncs.com/api/v1'

# Don't do this!
#dashscope.base_http_api_url = 'https://dashscope.aliyuncs.com/compatible-mode/v1'

rerank_resp = [dashscope.TextReRank.call(
    model="qwen3.7-text-rerank",
    query=queries[i],
    documents=candidates[i],
    top_n=3) for i in range(2)]

Note the shape of the call: one query, many documents, and a top_n that decides how many survive. queries and candidates are reusing the variables from the previous section.

For those who wonder what happens if you point the SDK at the OpenAI-compatible endpoint instead:

dashscope.base_http_api_url = 'https://dashscope.aliyuncs.com/compatible-mode/v1'

The answer is that you get a 404 error:

ReRankResponse(status_code=404, ...)

Examining the output

for i in range(2):
    for item in rerank_resp[i].output.results:
        print(f"[{i}] [{item.relevance_score:.3f}] {ids[i][item.index]}. {candidates[i][item.index][:60]}...")
    print('===')

[0] [0.851] id6. Transformer 使用自注意力来建模序列中词元之间的关系。...
[0] [0.750] id8. GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transform...
[0] [0.509] id5. 神经网络是受大脑启发的计算系统，由相互连接的节点层组成，这些节点层学习从数据中识别模式并进行预测。...
===
[1] [0.985] id10. 预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。...
[1] [0.809] id8. GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transform...
[1] [0.758] id12. 后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。...
===

There it is. The definition sentence for "预训练" ("pretraining") is now first — a gap of 0.985 vs 0.809 over the runner-up, where the embedding model had them in the wrong order entirely.

Two details in that code are worth keeping in mind:

• item.index is a position in candidates[i], so candidates[i][item.index] recovers the text and ids[i][item.index] recovers the Chroma ID. That parallel-list lookup is the only fiddly part of using a reranker.
• The scores are not cosine similarities and are not comparable with the distances from either embedding model. They're the reranker's own relevance scale — only useful for ordering, and only within a single call.

Putting it together

The recipes above are the pieces; in practice they compose into one retrieval rule.

def retrieve(query, k=3):
    # 1. cheap, approximate recall over the whole corpus
    res = collection1.query(query_texts=query, n_results=max(k * 4, 10))
    candidates, ids = res['documents'][0], res['ids'][0]

    # 2. expensive, accurate reordering of the shortlist
    resp = dashscope.TextReRank.call(
        model="qwen3.7-text-rerank",
        query=query,
        documents=candidates, top_n=k)

    # 3. hand back documents and IDs, best first
    return [(ids[r.index], candidates[r.index], r.relevance_score)
            for r in resp.output.results]

Two heuristics are hidden in those numbers. The k * 4 (with a floor of 10) is the retrieval depth: fetch several times what you need, so that the reranker has something worth reordering, but not so much that you're paying for the whole corpus twice. And when there are fewer candidates than k, the reranker simply returns what it got — no special case needed.

Avoiding the DashScope SDK

If you want to get rid of the dashscope dependency, you can handroll your own POST:

def retrieve(query, k=3):
    # 1. cheap, approximate recall over the whole corpus
    res = collection1.query(query_texts=query, n_results=max(k * 4, 10))
    candidates, ids = res['documents'][0], res['ids'][0]

    # 2. expensive, accurate reordering of the shortlist
    import requests
    resp = requests.post(
        f'https://{DASHSCOPE_WORKSPACE_ID}.cn-beijing.maas.aliyuncs.com/api/v1/services/rerank/text-rerank/text-rerank',
        headers={
            'Content-Type': 'application/json',
            'Authorization': f'Bearer {DASHSCOPE_API_KEY}',},
        json={
            'model': 'qwen3.7-text-rerank',
            'input': {
                'query': query,
                'documents': candidates,
            },
            'parameters': { 'top_n': k, },
        },
        timeout=30)

    # 3. hand back documents and IDs, best first
    return [(ids[r['index']], candidates[r['index']], r['relevance_score'])
            for r in resp.json()['output']['results']]

The request body comes from the official API reference.

A few reasons to skip the DashScope SDK:

• The install is large, and most of it has nothing to do with reranking. dashscope depends on many packages, aiohttp, cryptography, httpx, etc., that cost you about 31 MB of disk space, while requests costs nothing: Chroma already includes it via its dependencies.
• The knobs stay in your hands. timeout=30 is in the snippet because you can set it.
• You see what is on the wire. When the response shape shifts you get a missing dict key. Debugging an HTTP call you wrote is a shorter path than debugging a wrapper you did not.

The trade is typed access: resp.output.results vs. resp.json()['output']['results']. The code also gets messier. And the URL may change over time. If that is worth 31 MB, keep the SDK. I personally prefer the HTTP call.

Recap

The three options, and when each makes sense:

	Best for	Cost	Watch out for
Local model	privacy, offline use, tight budgets	~92 MB of disk	needs a deep-learning stack installed
Online model	fastest to a working demo, best quality per line of code	per-query fees, API key	vendor lock-in, network dependency
Reranker	any retrieval where ranking matters	a second model in the loop	latency — it runs on every query

The thread running through all of it: embeddings get you a shortlist, they don't get you an answer.

An embedding model has to compress a sentence into one vector, and it must do so before it ever sees your query — which is precisely why it's fast, and precisely why it confuses "mentions the topic" with "answers the question." A reranker fixes that by reading query and candidate together, at the cost of doing the work fresh each time. Retrieve broad, rerank narrow, then generate.

Swap BAAI/bge-small-zh-v1.5 for any other row in the model table and the code above is unchanged — including the all-MiniLM-L6-v2 you started with, back in its natural habitat of English text.

Comments

[Ahmet Özel]

The specific failure is worth naming loudly, because it does not look like a failure. An English-only model given Chinese input still returns vectors, still returns neighbours, and still returns a confident top-k. Nothing errors; the recall is just noise. Same class of bug as a parser that silently drops pages.

Adding the reranking section is the right call, since it is the part people skip and it does the most work in a language where token-level similarity is weakest.

One check worth putting in the tutorial: verify the swap against a handful of known query-document pairs before and after. A language switch is exactly the change where you want ground truth rather than the impression that results look better.

[Cheng]

Yeah, you are absolutely right. Should've been more careful about the swap and the choice of the query. Thanks.

[Ahmet Özel]

Query choice can make a model swap look better than it is, especially when the example repeats wording from the target document. I would include a Chinese paraphrase and a mixed Chinese/English technical query for each known target, keeping the target IDs fixed across the swap.

Also record the query and document encoding settings alongside the results. A mismatched prefix or truncation setting can turn a useful model comparison into a pipeline comparison without an obvious error.