It was 1 AM. The project manager sent three rapid-fire voice messages in the group: “Why did our demo smart Q&A answer ‘How are remaining annual leave days calculated?’ with a team-building notice? Is this project ever going live?” I stared at Grafana—the accuracy curve was unnervingly flat, stuck at 72% for an entire week. The product manager wanted employees to query internal policies in natural language. Instead, it felt like talking to a brand-new intern: you say A, it replies B, and sometimes you just have to guess.
The background was clear: we were building a private knowledge base Q&A for a 2,000-person company. All data lived in internal PDFs, Word documents, and Confluence pages. Public network access was forbidden, so we couldn’t just throw everything at a ChatGPT-like public API. We initially used Elasticsearch with inverted indexes and keyword matching—results were poor. Then we switched to vector search (Embedding + Faiss) and saw recall briefly climb to 70%, but it stopped there. In an enterprise context, 70% means one out of every three questions is wrong. No one dares to use that.
1. Why the Conventional Approach Gets Stuck at 70%
Private knowledge base Q&A is fundamentally RAG (Retrieval-Augmented Generation): chunk documents, embed them into vectors, store them in a vector database; when a user asks, retrieve the most similar chunks, pack them into a prompt, and let the LLM generate the answer. The accuracy bottleneck is not the LLM’s generative ability—it’s that the retrieval phase already feeds irrelevant content to the model. Even the best model can’t conjure correct answers from thin air.
The typical approach hits three major pitfalls:
- One-size-fits-all chunking. Cutting rigidly at 500 characters might split “Annual leave calculation: employees with 1 year of service get 5 days, plus 1 extra day per additional year, up to 15 days” right in the middle. The first half gets retrieved, the second half falls into another chunk—the model never sees the complete rule.
- Relying solely on vector similarity. Embedding models are insensitive to numbers, abbreviations, and synonyms. “PTO” and “paid vacation” might not be close enough in vector space, causing key documents to be missed.
- No re-ranking. The top-K retrieved chunks often include noise—titles that are similar but content irrelevant. Feeding that noise directly to the LLM makes it fabricate hallucinated answers from the misleading context.
If you don’t fix these problems, swapping vector databases or tuning better LLMs is just fooling yourself. Here’s how I used LangChain + Milvus to push recall to 95% with just three core changes.
2. Why LangChain + Milvus (Instead of Faiss/Pinecone/Elasticsearch)
- Vector database: Milvus. Faiss is essentially a single-machine index library—no persistence or distributed capabilities; once data grows you have to shard manually. Pinecone and Weaviate offer great cloud services, but our client’s compliance rules mandate everything stays on‑prem. Milvus supports one‑click private deployment via Docker Compose and easily scales to billions of vectors. For enterprises this is a hard requirement.
- Orchestration layer: LangChain. Not because it’s trendy, but because its Document Loaders, Text Splitters, Retrievers, and Chain abstractions let me quickly combine hybrid search + re‑ranking without writing piles of boilerplate code. The official Milvus integration also saves time compared to hand‑rolling gRPC calls.
- Hybrid search + re‑ranking. Pure vector retrieval misses too many hits in terminology‑dense documents. I chose to fuse vector similarity + BM25 keyword scoring, then apply a Cross‑Encoder Reranker on the merged candidate set to surface the 3–5 most relevant chunks for the LLM.
3. Core Implementation
The code below demonstrates the three key changes, with all imports included so you can adapt your own knowledge base project right away.
Change 1: Semantically Intact Chunking Strategy
This solves the “hard cut truncates rules” problem. Use RecursiveCharacterTextSplitter with a separator priority suitable for Chinese documents, and set an overlap window to preserve cross‑chunk context.
from langchain.text_splitter import RecursiveCharacterTextSplitter
# 企业中文文档的分隔符层级：优先按段落，再按句子，最后按字
splitter = RecursiveCharacterTextSplitter(
separators=["\n\n", "\n", "。", "！", "？", "；", "，", " ", ""],
chunk_size=400, # 根据内部测试，400字能涵盖大部分制度条款
chunk_overlap=50, # 防止规则被切断，重叠保证连续性
length_function=len,
)
docs = splitter.split_documents(raw_documents)
Why chunk_size=400: I tested values from 200 to 800. 200 is too small and loses context during retrieval; 800 is too large and introduces noise. 400 hit the sweet spot. chunk_overlap shouldn’t be too big, otherwise the retrieved results become redundant and complex to deduplicate.
Change 2: Hybrid Retrieval + Milvus Ingestion
Since pure vector retrieval falls short, we also store the original text at index time to enable BM25 keyword search. Milvus 2.3+ supports string fields and dynamic schemas, so we write the chunk text alongside the vectors. During retrieval, use LangChain’s EnsembleRetriever to fuse vector and BM25 scores.
python
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Milvus
from langchain.retrievers import BM25Retriever, EnsembleRetriever
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
# 1. 用 BGE 中文嵌入模型，私有化部署，不走外网
embeddings = HuggingFaceEmbeddings(
model_name="BAAI/bge-large-zh-v1.5",
model_kwargs={'device': 'cuda'},
encode_kwargs={'normalize_embeddings': True}
)
# 2. 连接 Milvus（假设已通过 docker-compose 启动）
connections.connect("default", host="localhost", port="19530")
# 3. 创建集合：包含主键、文本字段、向量字段
fields = [
FieldSchema(name="pk", dtype=DataType.INT64, is_primary=True, auto_id=True),
FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=655
Top comments (0)