Building local hybrid search with Qdrant and BGE-M3

I wanted a local semantic search setup for internal documents: no hosted vector database, no embedding API, and enough flexibility to compare dense search, sparse search, hybrid search, and reranking.

The interesting part was not "how to start Qdrant." It was how to store both dense and sparse vectors for the same document chunk, then switch the retrieval strategy at query time.

This is the shape I ended up testing:

• Qdrant running locally in Docker
• BGE-M3 as a local embedding model
• Qdrant named vectors: dense and sparse on the same point
• dense search, sparse search, hybrid search with RRF
• optional cross-encoder reranking with BAAI/bge-reranker-v2-m3

This is a small PoC, not a benchmark. The goal is to make the moving parts explicit.

Why BGE-M3 with Qdrant named vectors?

BGE-M3 can produce dense embeddings and sparse lexical weights from the same model family. Qdrant can store multiple vectors on a single point using named vectors.

That combination is convenient for document search because each chunk can be stored once:

PointStruct(
    id=chunk_id,
    vector={
        "dense": dense_vector,
        "sparse": sparse_vector,
    },
    payload={
        "text": chunk_text,
        "path": path,
    },
)

At query time, I can choose:

• using="dense" for semantic search
• using="sparse" for lexical-style search
• both, then fuse rankings with Reciprocal Rank Fusion
• a second reranking stage for the top candidates

The practical benefit is that I do not need to rebuild the collection just to compare retrieval modes.

Local setup

The environment I used:

• macOS on Apple Silicon
• Python 3.13
• Docker 28.x
• Qdrant in a local Docker container

Install the Python packages:

python -m pip install --upgrade pip setuptools wheel
python -m pip install -U \
  "qdrant-client>=1.15.0" "pydantic>=2.7,<2.12" \
  sentence-transformers FlagEmbedding scipy \
  torch torchvision torchaudio \
  pandas pymupdf

Download BGE-M3 locally:

python - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    "BAAI/bge-m3",
    local_dir="./models/bge-m3",
    local_dir_use_symlinks=False,
)
print("Downloaded to ./models/bge-m3")
PY

Start Qdrant with persistent local storage:

docker run --name qdrant \
  -p 6333:6333 \
  -v $(pwd)/qdrant_storage:/qdrant/storage \
  qdrant/qdrant

After startup, the dashboard is available at:

http://localhost:6333/dashboard

Test corpus

For the first pass, I used a tiny mixed-language corpus. It is too small to prove quality, but good enough to check whether each retrieval path behaves as expected.

I kept the sample texts in Japanese because the real target is Japanese internal documentation. For readers who do not read Japanese, this is the meaning of each row:

id	Text	Meaning
1	今日は天気がいい	The weather is nice today
2	年末年始の休暇規定を確認してください	Check the year-end and New Year vacation policy
3	申請の締切日は12月15日です	The application deadline is December 15
4	API 標準設計ガイドの参照はこちら	See here for the standard API design guide
5	会議は月曜日に延期になりました	The meeting was postponed to Monday
6	正しいPythonの環境構築方法	How to set up a proper Python environment
7	Please check the holiday policy for New Year vacation.	English sentence with similar holiday-policy meaning

TEXTS = [
    "今日は天気がいい",
    "年末年始の休暇規定を確認してください",
    "申請の締切日は12月15日です",
    "API 標準設計ガイドの参照はこちら",
    "会議は月曜日に延期になりました",
    "正しいPythonの環境構築方法",
    "Please check the holiday policy for New Year vacation.",
]

QUERY = "正月の休暇と申請締切"
TOP_K = 5

The query means:

Query	Meaning
正月の休暇と申請締切	New Year vacation and the application deadline

In this corpus, the relevant answers are split across multiple sentences:

• holiday policy: 年末年始の休暇規定を確認してください
• deadline: 申請の締切日は12月15日です
• related English sentence: Please check the holiday policy for New Year vacation.

Normalizing BGE-M3 sparse output

One detail that is easy to miss: BGE-M3 sparse output may come back in different shapes depending on the library path. I normalized both forms into Qdrant's SparseVector.

from typing import Any
from qdrant_client.http.models import SparseVector


def to_qdrant_sparse_list(m3_output: dict[str, Any]) -> list[SparseVector]:
    sv_list: list[SparseVector] = []

    if "sparse_vecs" in m3_output:
        for sp in m3_output["sparse_vecs"]:
            sv_list.append(
                SparseVector(
                    indices=sp.indices.tolist(),
                    values=sp.data.tolist(),
                )
            )
    elif "lexical_weights" in m3_output:
        for lw in m3_output["lexical_weights"]:
            items = sorted((int(k), float(v)) for k, v in lw.items())
            sv_list.append(
                SparseVector(
                    indices=[i for i, _ in items],
                    values=[v for _, v in items],
                )
            )
    else:
        raise ValueError("No sparse output found in BGEM3 result")

    return sv_list

Creating dense, sparse, or hybrid collections

For a dense-only collection:

q.create_collection(
    collection_name=COLL,
    vectors_config={
        "dense": qm.VectorParams(
            size=dim,
            distance=qm.Distance.COSINE,
        )
    },
)

For a sparse-only collection:

q.create_collection(
    collection_name=COLL,
    vectors_config=None,
    sparse_vectors_config={
        "sparse": qm.SparseVectorParams(),
    },
)

For the hybrid version, store both named vectors:

q.create_collection(
    collection_name=COLL,
    vectors_config={
        "dense": qm.VectorParams(
            size=dim,
            distance=qm.Distance.COSINE,
        )
    },
    sparse_vectors_config={
        "sparse": qm.SparseVectorParams(),
    },
)

That last shape is the one I would normally use for document search. It keeps the operational model simple: one point per chunk, both retrieval signals attached.

Upserting both vectors on the same point

Dense embeddings came from SentenceTransformer using the local BGE-M3 directory. Sparse vectors came from BGEM3FlagModel.

from FlagEmbedding import BGEM3FlagModel
from qdrant_client.http.models import PointStruct
from sentence_transformers import SentenceTransformer


dense_model = SentenceTransformer("./models/bge-m3")
m3 = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)

dense_vecs = [
    dense_model.encode(t, normalize_embeddings=True).tolist()
    for t in TEXTS
]

sparse_out = m3.encode(TEXTS, return_sparse=True)
sparse_vecs = to_qdrant_sparse_list(sparse_out)

points = []
for id_, text, dense_vec, sparse_vec in zip(
    range(1, len(TEXTS) + 1),
    TEXTS,
    dense_vecs,
    sparse_vecs,
):
    points.append(
        PointStruct(
            id=id_,
            vector={
                "dense": dense_vec,
                "sparse": sparse_vec,
            },
            payload={"text": text},
        )
    )

q.upsert(collection_name=COLL, points=points)

Dense and sparse search

Dense search uses the dense vector name:

def search_dense(q, dense_model, qtext: str, top_k: int = 5):
    qvec = dense_model.encode(qtext, normalize_embeddings=True).tolist()
    res = q.query_points(
        collection_name=COLL,
        query=qvec,
        using="dense",
        limit=top_k,
        search_params=qm.SearchParams(hnsw_ef=128),
    )
    return res.points

Sparse search uses the sparse vector name:

def search_sparse(q, m3, qtext: str, top_k: int = 5):
    out = m3.encode([qtext], return_sparse=True)
    q_sp = to_qdrant_sparse_list(out)[0]
    res = q.query_points(
        collection_name=COLL,
        query=q_sp,
        using="sparse",
        limit=top_k,
    )
    return res.points

Hybrid search with RRF

For the hybrid mode, I searched dense and sparse separately, then fused the ranked IDs with Reciprocal Rank Fusion.

def rrf_fuse(id_lists: list[list[int]], k: int = 60) -> list[int]:
    score: dict[int, float] = {}
    for id_list in id_lists:
        for rank, pid in enumerate(id_list, start=1):
            score[pid] = score.get(pid, 0.0) + 1.0 / (k + rank)

    return [
        pid
        for pid, _ in sorted(
            score.items(),
            key=lambda x: x[1],
            reverse=True,
        )
    ]

The search function is intentionally boring:

def search_hybrid_rrf(q, dense_model, m3, qtext: str, top_k: int = 5):
    dense_hits = search_dense(q, dense_model, qtext, top_k=top_k)
    sparse_hits = search_sparse(q, m3, qtext, top_k=top_k)

    dense_ids = [h.id for h in dense_hits]
    sparse_ids = [h.id for h in sparse_hits]
    fused_ids = rrf_fuse([dense_ids, sparse_ids])[:top_k]

    by_id = {h.id: h for h in dense_hits}
    for h in sparse_hits:
        by_id.setdefault(h.id, h)

    return [by_id[i] for i in fused_ids if i in by_id]

I used RRF here because it only needs ranks, not comparable raw scores from different retrieval systems.

Optional cross reranking

After first-stage retrieval, I optionally reranked the candidates with BAAI/bge-reranker-v2-m3.

from FlagEmbedding import FlagReranker


def cross_rerank(qtext: str, hits, max_chars: int = 1400, batch_size: int = 16):
    rr = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

    pairs = []
    keep = []
    for h in hits:
        text = (h.payload.get("text") or "")[:max_chars].replace("\n", " ")
        if text.strip():
            pairs.append([qtext, text])
            keep.append(h)

    scores = rr.compute_score(pairs, normalize=True, batch_size=batch_size)

    for h, s in zip(keep, scores):
        h.payload["_cross"] = float(s)

    keep.sort(key=lambda x: x.payload.get("_cross", 0.0), reverse=True)
    return keep

This is a second stage, not a replacement for the index. In a larger system I would keep the rerank candidate set bounded, for example by reranking only the top 20 to 50 candidates.

What the small test showed

I ran six combinations:

python test_index_and_rerank.py --index dense  --rerank none
python test_index_and_rerank.py --index dense  --rerank cross
python test_index_and_rerank.py --index sparse --rerank none
python test_index_and_rerank.py --index sparse --rerank cross
python test_index_and_rerank.py --index both   --rerank none
python test_index_and_rerank.py --index both   --rerank cross

Dense search without reranking put the holiday policy first, the English holiday sentence second, and the deadline third:

INDEX = DENSE, SEARCH = DENSE   RERANK=NONE
Query: "正月の休暇と申請締切"

score=0.727  id=2  text=年末年始の休暇規定を確認してください
score=0.664  id=7  text=Please check the holiday policy for New Year vacation.
score=0.617  id=3  text=申請の締切日は12月15日です
score=0.474  id=5  text=会議は月曜日に延期になりました
score=0.378  id=1  text=今日は天気がいい

Sparse search put the deadline first and the holiday policy second:

INDEX = SPARSE, SEARCH = SPARSE   RERANK=NONE
Query: "正月の休暇と申請締切"

score=0.097  id=3  text=申請の締切日は12月15日です
score=0.071  id=2  text=年末年始の休暇規定を確認してください
score=0.001  id=6  text=正しいPythonの環境構築方法
score=0.000  id=5  text=会議は月曜日に延期になりました

Hybrid search plus cross reranking gave this order in the same small test:

INDEX = BOTH, SEARCH = HYBRID(RRF)   RERANK=CROSS
Query: "正月の休暇と申請締切"

score=0.617  cross=0.030  id=3  text=申請の締切日は12月15日です
score=0.727  cross=0.025  id=2  text=年末年始の休暇規定を確認してください
score=0.664  cross=0.020  id=7  text=Please check the holiday policy for New Year vacation.
score=0.474  cross=0.000  id=5  text=会議は月曜日に延期になりました
score=0.001  cross=0.000  id=6  text=正しいPythonの環境構築方法

I would not read too much into the exact scores from such a small corpus. The useful observation was simpler:

• dense search captured semantic and cross-language similarity better in this test
• sparse search surfaced lexical matches such as the deadline sentence
• hybrid search let both paths contribute candidates
• reranking changed the order of the top candidates, so it should be evaluated separately from retrieval

Issues I hit

The main problems were ordinary integration issues:

• Vector dimension mismatch: if the model dimension does not match the collection's configured vector size, recreate the collection.
• Sparse vector support: use a recent qdrant-client; I used qdrant-client>=1.15.0.
• Pydantic compatibility: I pinned pydantic>=2.7,<2.12 in this environment.
• Model download size: the first BGE-M3 download needs several GB of disk space.
• Docker persistence: without -v $(pwd)/qdrant_storage:/qdrant/storage, deleting the container also removes the data.

How I would use this for real documents

For actual internal documents, I would keep the same retrieval structure and add a document ingestion pipeline:

• extract text from .txt, .md, and .pdf
• split into chunks with overlap
• store dense and sparse vectors on the same Qdrant point
• include payload fields such as path, page, chunk_index, and text
• query with dense, sparse, or hybrid mode depending on the use case
• optionally rerank a bounded candidate set

My default would be to ingest both dense and sparse vectors from the beginning. Even if the first UI only exposes semantic search, having both vectors available makes later experiments much cheaper.

Qdrant docs:

• Named vectors: https://qdrant.tech/documentation/concepts/vectors/#named-vectors
• Sparse vectors: https://qdrant.tech/documentation/concepts/vectors/#sparse-vectors

BGE-M3:

• Model: https://huggingface.co/BAAI/bge-m3
• FlagEmbedding: https://github.com/FlagOpen/FlagEmbedding

Happy to answer questions.