Making Chroma Work in Chinese

Note: This tutorial picks up where Chroma's official quick start guide leaves off.

So you've finished Chroma's quick start — congratulations. Everything works, right up until you feed Chroma some Chinese text.

Then it doesn't.

Chroma's default embedding model, all-MiniLM-L6-v2, is English-only. Given Chinese or Japanese input, it embeds it as though the language were noise. You need a model that was trained on the language you care about.

You have two ways to get one:

	Local model	Online model
Runs where	on your machine	vendor's server
Network	not needed	required
Setup	install packages, download weights	an API key
Cost	disk space (hundreds of MB)	per query
Offline	works	does not
Privay	good	bad

This tutorial walks through both, and then a third option that fixes a problem neither of them solves: ranking.

To be concrete, we'll assume Chinese is the language you need to add. Swap the model name and the same code supports any language.

Custom Embedding Model

Whatever you choose, the hook into Chroma is the same: the optional embedding_function argument of client.create_collection().

# Replace with your own class
ef = YourEmbeddingFunction(...)

collection = client.create_collection(
    name="my_collection",
    embedding_function=ef)

That's the whole interface. The rest of this tutorial is about what to put in place of YourEmbeddingFunction.

Option 1: Local Models

Choosing a model

Assuming you want Chinese support from a local model, these are the usual candidates:

Model	Language	Dim.	Notes
paraphrase-multilingual-MiniLM-L12-v2	Multilingual	384	lightweight
BAAI/bge-small-zh-v1.5	Chinese	512	good for Chinese
BAAI/bge-base-zh-v1.5	Chinese	768	same family, stronger
BAAI/bge-m3	Multilingual	1024	multilingual
text2vec-base-chinese	Chinese	768
m3e-base	Chinese	768
all-MiniLM-L6-v2	English	384	Chroma's default (for reference)

Roughly:

More dimensions means better quality and a bigger download.

• The smallest multilingual option is only 384-dimensional and stays under 500 MB.
• bge-m3 is 1024-dimensional and will cost you well over 2 GB.
• For Chinese specifically, bge-small-zh-v1.5 is the cheapest thing that works properly, so that's what we'll use below.

Installing the prerequisite package

To use a local model you need one extra Python package, sentence-transformers. Chroma deliberately does not ship it, for a good reason:

• The model is loaded through embedding_functions.SentenceTransformerEmbeddingFunction.
• That class needs the sentence-transformers package.
• Which in turn pulls in
  • torch (PyTorch)
  • transformers
  • huggingface-hub
  • tokenizers
  • numpy, scipy and friends.

That's a deep-learning stack — hundreds of MB of dependencies. If your environment already has PyTorch, fine. If your project is, say, a small web service that just happens to need a vector store, that stack is a bit of overkill. So Chroma makes you opt in:

pip install sentence-transformers

Downloading the model

Now we can actually build the embedding function.

The first time you construct SentenceTransformerEmbeddingFunction('BAAI/bge-small-zh-v1.5'), it downloads the weights from huggingface.co, where the model lives by default.

If you live in China, or anywhere the connection to huggingface.co is poor, point it at a mirror first. On Linux:

export HF_ENDPOINT="https://hf-mirror.com"

On Windows PowerShell:

$env:HF_ENDPOINT="https://hf-mirror.com"

Or set it from inside Python, which is often the most convenient:

from chromadb.utils import embedding_functions

import os
os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
        model_name="BAAI/bge-small-zh-v1.5")

The output is like this:

D:\app\anaconda3\envs\ai\lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Loading weights: 100%|███████████████████████████████████████████████████████████████| 71/71 [00:00<00:00, 9559.74it/s]

What to notice in that output:

• The download takes a while, especially the first time.
• The progress bar above is the weight loading at the end, not the download itself, so the wait looks longer than it reads.
• Run it a second time and the download is skipped, so startup gets noticeably faster.
• Loading the weights still takes a moment every session — don't expect a split-second import.

By default the model lands in ~/.cache/huggingface/hub on Linux, or C:\Users\<you>\.cache\huggingface\hub on Windows.

Caching the model inside your project

A user-level cache is convenient, but it makes the project non-portable: a teammate clones the repo, and still has to download 100 MB before anything runs. Downloading into the project directory instead fixes that — one hf command does it:

$env:HF_ENDPOINT="https://hf-mirror.com"
pip install "huggingface_hub[cli]"
hf download BAAI/bge-small-zh-v1.5 --local-dir ./models/bge-small-zh-v1.5

Then point model_name at the local path. The leading ./ is what tells SentenceTransformerEmbeddingFunction this is a directory and not a HuggingFace model ID:

ef = embedding_functions.SentenceTransformerEmbeddingFunction(
        model_name="./models/bge-small-zh-v1.5")

The output:

Loading weights: 100%|██████████████████████████████████████████████████████████████| 71/71 [00:00<00:00, 14191.55it/s]

You can tell it worked: the loading is faster (no network round-trip to resolve the model), and ./models/bge-small-zh-v1.5 now contains roughly 184 MB of files.

Peeking at the source

Curious about the internal workings? Let's take a look at the source code:

import inspect
print(inspect.getsource(embedding_functions.SentenceTransformerEmbeddingFunction))

The output is long, so only the beginning matters here:

class SentenceTransformerEmbeddingFunction(EmbeddingFunction[Documents]):
    # Since we do dynamic imports we have to type this as Any
    models: Dict[str, Any] = {}

    # If you have a beefier machine, try "gtr-t5-large".
    # for a full list of options: https://huggingface.co/sentence-transformers, https://www.sbert.net/docs/pretrained_models.html
    def __init__(
        self,
        model_name: str = "all-MiniLM-L6-v2",
        device: str = "cpu",
        normalize_embeddings: bool = False,
        **kwargs: Any,
    ):
        ...

Two things worth noticing.

• The default model_name is Chroma's English default — which is the whole reason this tutorial exists.
• And there's a device argument: pass "cuda" to move embedding onto a GPU if you have one.

Using the local model

Time to put it to work. If you've run this before, the collection already exists and Chroma will refuse to create it again — hence the commented-out cleanup line, which you'll want in a notebook where cells get re-run:

# if you've created the collection before, uncomment and run the next line
#client.delete_collection(name='demo1')

import chromadb
client = chromadb.Client()
collection = client.create_collection(
        name="demo1",
        embedding_function=ef)

Now add some documents and query them. These are short, self-contained sentences about programming languages and machine learning — deliberately, so that "what is Transformer?" has a defensible right answer:

docs = [
    'Python 是一种流行的编程语言，广泛用于数据处理、机器学习以及广义上的人工智能。',
    '数据处理包括将原始数据清洗、转换和聚合成有用的格式。',
    '机器学习通过训练模型来发现数据中的模式，并对新样本进行预测。',
    '深度学习使用具有多层的神经网络来学习分层表示。',
    '神经网络是受大脑启发的计算系统，由相互连接的节点层组成，这些节点层学习从数据中识别模式并进行预测。',
    'Transformer 使用自注意力来建模序列中词元之间的关系。',
    '注意力机制让模型在生成每个输出表示时，对输入的不同部分进行加权。',
    'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
    '大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。',
    "预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。",
    "微调使用有标注数据或指令数据，使预训练模型适应特定任务或领域。",
    "后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。",
    "检索增强生成（RAG）将来自外部知识源的检索与大语言模型（LLM）的生成相结合。",
    "一个基础的 RAG 流程会切分文档、对文本块进行嵌入、将其存入向量索引、检索前 k 个文本块，并将它们传给 LLM。",
    "向量嵌入将文本、图像或其他数据映射到高维空间中的点，以进行相似度搜索。",
]

Here, each item of docs represents a document, albeit a short one.

Add documents to the collection:

collection.add(ids=[f'id{i+1}' for i in range(len(docs))], documents=docs)

And ask a question:

resp = collection.query(query_texts=['什么是Transformer？'], n_results=3)

The return value resp looks like this:

{'ids': [['id8', 'id6', 'id9']],
 'embeddings': None,
 'documents': [['GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
   'Transformer 使用自注意力来建模序列中词元之间的关系。',
   '大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。']],
 'uris': None,
 'included': ['metadatas', 'documents', 'distances'],
 'data': None,
 'metadatas': [[None, None, None]],
 'distances': [[0.3059161305427551, 0.36248522996902466, 0.4628119468688965]]}

Assessing the results

• The question asks "什么是 Transformer?" — meaning "What is a Transformer?"
• The top two hits, id8 and id6, are both genuinely about Transformers. Good.
• But id8 beats id6, yet id8 is mainly about GPT — it happens to mention "Transformer" three times, while id6 is the sentence that actually defines the Transformer.
• The third hit, id9, is about LLMs, which was not asked about at all.

The distances also tell a story: 0.31 for the top hit is a fairly loose match. The model is unsure, and the ranking quietly disagrees with any human reading of the question.

The cause is worth naming, because it recurs. An embedding model compresses a whole sentence into a single vector, and the vector is dominated by what words appear, not by what the sentence is doing with them. id8 is stuffed with the token "Transformer"; id6 uses it once. For a similarity search over a bag of tokens, id8 simply looks more relevant. Nothing in the embedding stage distinguishes "mentions Transformer" from "explains Transformer" — that distinction requires reading the query and the candidate together, which is a different architecture. We'll come back to it in the last section.

Option 2: Online Models

If you'd rather not manage packages and downloads, a hosted embedding API is the alternative. The trade-offs are the ones from the table at the top: you need an API key, and you pay per call, but there's nothing to install and the model is somebody else's problem to keep up to date.

Writing the embedding function

Chroma doesn't know about your vendor, so you write a small adapter. All it needs is a class with a __call__ method that maps a list of strings to a list of vectors.

Here we use Alibaba's DashScope, whose API is OpenAI-compatible — which means the official openai client works as-is, with only the base_url changed:

import os
from openai import OpenAI
from chromadb.api.types import EmbeddingFunction, Embeddings

class DashScopeEmbeddingFunction(EmbeddingFunction):
    def __init__(self, api_key : str = None, model : str = "qwen3.7-text-embedding-flash"):
        self.model = model
        self.client = OpenAI(
            base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
            api_key=api_key or os.getenv("DASHSCOPE_API_KEY"))

    def __call__(self, input: list[str]) -> Embeddings:
        response = self.client.embeddings.create(
            model=self.model,
            input=input,
        )
        return [item.embedding for item in response.data]

A few notes on the adapter:

• The API key is read from the DASHSCOPE_API_KEY environment variable when you don't pass one in. Never hard-code it in a notebook you might share.
• __call__ receives a batch of texts and returns one vector each — Chroma always batches, so don't loop over the input and make one request per document.
• The returned objects use .embedding and .index; if you ever need to be defensive about ordering, sort by index before returning.

Then build a second collection on top of it:

#Run this if it complains about the collection 'demo2' already exists
#client.delete_collection(name='demo2')

import chromadb
client = chromadb.Client()

ds_ef = DashScopeEmbeddingFunction()
collection2 = client.create_collection(
        name="demo2",
        embedding_function=ds_ef)

The same documents, the same query — only the model changed:

collection2.add(ids=[f'id{i+1}' for i in range(len(docs))],
                documents=docs)

collection2.query(query_texts=['什么是Transformer？'], n_results=3)

The output:

{'ids': [['id6', 'id8', 'id3']],
 'embeddings': None,
 'documents': [['Transformer 使用自注意力来建模序列中词元之间的关系。',
   'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
   '机器学习通过训练模型来发现数据中的模式，并对新样本进行预测。']],
 'uris': None,
 'included': ['metadatas', 'documents', 'distances'],
 'data': None,
 'metadatas': [[None, None, None]],
 'distances': [[0.8229950666427612, 0.8484762907028198, 1.2087467908859253]]}

Better. id6 — the sentence that actually defines the Transformer — now comes first, and id8 second. The ordering finally matches what a human would say.

Don't read too much into the third hit, id3 (about machine learning): it's an artefact of asking for three results when only two documents are genuinely on topic. The third slot has to be filled by something, and something marginally related is the best available answer. Nevertheless, its distance ~1.21 is much larger than the first two ~0.82 and ~0.85.

One caution on the numbers: distances are not comparable across models. The local model returned ~0.31 and this one returns ~0.82 for its best hit, but that does not mean it is worse. Every model has its own scale and its own notion of distance; compare rankings, never raw scores, when models differ.

Reranker Models

So we've fixed the language problem, and we now have a ranking that looks sensible. But it took trying two models to get there, and nothing guarantees the third query works as well.

The underlying weakness hasn't gone away. The embedding stage still compares the query against each document separately, through a lossy single vector. That's what makes it fast — the whole corpus can be indexed once, offline — and it's also what makes it approximate.

A reranker replaces that approximation with a slow, careful look:

1. Retrieve generously. Ask the embedding model for more candidates than you need — if you want 3, fetch 10.
2. Rerank. Hand the query and all candidates to the reranker together, and let it score each one by reading them side by side.
3. Keep the top K. Pass only the winners to the LLM.

Because the reranker sees query and document jointly, it can tell "mentions Transformer" from "explains Transformer" — exactly the distinction the embedding model blurred. The price is that it can't be precomputed: it must run on every query against every candidate, which is why it's used to reorder a shortlist rather than to search the corpus.

Rerankers are also much bigger. BAAI/bge-reranker-v2-m3 needs about 2.3 GB of disk — a lot to carry for a ranking refinement. So instead, we'll use a hosted one, qwen3.7-text-rerank.

That said, if privacy is a priority, the local option is probably the better choice — possibly the only one.

1. Collecting candidates

The embedding model and the reranker are completely independent — nothing requires them to come from the same vendor, or even to be the same kind of thing. Here, the retrieval comes from the local bge-small-zh-v1.5 and the reranking from the online qwen3.7-text-rerank.

We keep using collection from Option 1, and this time ask for 10 candidates instead of 3:

# 1. collect more candidates

query = '什么是Transformer？'
res = collection.query(
    query_texts=[query],
    n_results=10)

candidates = res['documents'][0]
ids = res['ids'][0]
for s in candidates[:3]: print(s)

GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。
Transformer 使用自注意力来建模序列中词元之间的关系。
大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。

Note which sentence is on top. ids is carried along in parallel with candidates, because the reranker returns positions into the list you handed it — you'll need those IDs to report results afterward.

2. Calling the reranker

The DashScope API doesn't follow the OpenAI format for reranking, so this step uses its own SDK.

pip install dashscope

You also need to log into the Alibaba Cloud's official website and apply for a DashScope workspace ID -- it looks like ws-xxx.

import dashscope
dashscope_workspace_id = 'ws-xxx'   # your own workspace ID

dashscope.base_http_api_url = f'https://{dashscope_workspace_id}.cn-beijing.maas.aliyuncs.com/api/v1'

rerank_resp = dashscope.TextReRank.call(
    model="qwen3.7-text-rerank",
    query=query,
    documents=candidates,
    top_n=3)

Note the shape of the call: one query, many documents, and a top_n that decides how many survive. query and candidates are reusing the variables from the previous section.

For those who wonder what happens if you point the SDK at the OpenAI-compatible endpoint instead:

dashscope.base_http_api_url = 'https://dashscope.aliyuncs.com/compatible-mode/v1'

The answer is you get a 404 error:

ReRankResponse(status_code=404, request_id='', code='Unknown', message='', output=None, usage=None, headers={'vary': 'Accept-Encoding', 'date': 'Mon, 21 Sep 2026 04:45:50 GMT', 'server': 'istio-envoy', 'connection': 'close', 'content-length': '0'})

Examining the output

for item in rerank_resp.output.results:
    print(f"[{item.relevance_score:.3f}] {ids[item.index]}. {candidates[item.index][:60]}...")

[0.851] id6. Transformer 使用自注意力来建模序列中词元之间的关系。...
[0.780] id8. GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transform...
[0.509] id5. 神经网络是受大脑启发的计算系统，由相互连接的节点层组成，这些节点层学习从数据中识别模式并进行预测。...

There it is. The definition sentence for "Transformer" is now first, the GPT sentence second — a gap of 0.851 vs 0.780, where the embedding model had them within 0.03 of each other and in the wrong order.

Two details in that code are worth keeping in mind:

• item.index is a position in candidates, so candidates[item.index] recovers the text and ids[item.index] recovers the Chroma ID. That parallel-list lookup is the only fiddly part of using a reranker.
• The scores are not cosine similarities and are not comparable with the distances from either embedding model. They're the reranker's own relevance scale — only useful for ordering, and only within a single call.

Putting it together

The recipes above are the pieces; in practice they compose into one retrieval rule.

def retrieve(query, k=3):
    # 1. cheap, approximate recall over the whole corpus
    res = collection.query(query_texts=[query], n_results=max(k * 4, 10))
    candidates, ids = res['documents'][0], res['ids'][0]

    # 2. expensive, accurate reordering of the shortlist
    resp = dashscope.TextReRank.call(
        model="qwen3.7-text-rerank",
        query=query,
        documents=candidates, top_n=k)

    # 3. hand back documents and IDs, best first
    return [(ids[r.index], candidates[r.index], r.relevance_score)
            for r in resp.output.results]

Two heuristics are hidden in those numbers. The k * 4 (with a floor of 10) is the retrieval depth: fetch several times what you need, so that the reranker has something worth reordering, but not so much that you're paying for the whole corpus twice. And when there are fewer candidates than k, the reranker simply returns what it got — no special case needed.

Recap

The three options, and when each makes sense:

	Best for	Cost	Watch out for
Local model	privacy, offline use, tight budgets	~184 MB of disk	needs a deep-learning stack installed
Online model	fastest to a working demo, best quality per line of code	per-query fees, API key	vendor lock-in, network dependency
Reranker	any retrieval where ranking matters	a second model in the loop	latency — it runs on every query

The thread running through all of it: embeddings get you a shortlist, they don't get you an answer.

An embedding model has to compress a sentence into one vector, and it must do so before it ever sees your query — which is precisely why it's fast, and precisely why it confuses "mentions the topic" with "answers the question." A reranker fixes that by reading query and candidate together, at the cost of doing the work fresh each time. Retrieve broad, rerank narrow, then generate.

Swap BAAI/bge-small-zh-v1.5 for any other row in the model table and the code above is unchanged — including the all-MiniLM-L6-v2 you started with, back in its natural habitat of English text.

Comments

[Ahmet Özel]

The specific failure is worth naming loudly, because it does not look like a failure. An English-only model given Chinese input still returns vectors, still returns neighbours, and still returns a confident top-k. Nothing errors; the recall is just noise. Same class of bug as a parser that silently drops pages.

Adding the reranking section is the right call, since it is the part people skip and it does the most work in a language where token-level similarity is weakest.

One check worth putting in the tutorial: verify the swap against a handful of known query-document pairs before and after. A language switch is exactly the change where you want ground truth rather than the impression that results look better.