I'm a lawyer. I deal with contracts, case files, and internal technical documents that legally cannot leave my machine. Last year I wanted to build a RAG pipeline to search through 2,000+ internal documents I'd accumulated. Every guide I found said the same thing: call OpenAI's embedding API, store vectors in Pinecone, query through a managed retrieval service.
That works if your data is blog posts and public docs. It doesn't work when you're bound by professional confidentiality obligations.
So I built Hippo — a local search toolkit that runs BM25 out of the box and hybrid search when you need it. No cloud APIs. No ChromaDB. No jieba. One pip install.
This post is about the search side of Hippo. If you want the LLM server side, I wrote about that here.
The problem with "just call the API"
Most RAG tutorials assume you're fine sending your documents to a third-party embedding endpoint. For a lot of use cases, that's acceptable. For regulated industries — legal, medical, financial — it often isn't.
Even if you're not in a regulated field, you might want local search for simpler reasons:
- You don't want to manage API keys and billing
- You want sub-millisecond latency
- You're working offline
- You have 10,000+ documents and API costs add up
The standard local stack is ChromaDB + sentence-transformers + a bunch of glue code. It works, but it's more infrastructure than most people need for a search box.
BM25 in 30 seconds
Here's the simplest thing Hippo does:
from hippo.embedding import VectorStore
store = VectorStore("docs.db") # SQLite, mode="sparse" by default
store.add_batch([
{"text": "Pipeline parallelism splits layers across devices"},
{"text": "BM25 handles exact keyword matches"},
{"text": "Speculative decoding improves latency by 2-3x"},
])
results = store.search("how to run big models on small GPUs", top_k=5)
for doc in results:
print(f"[{doc.score:.3f}] {doc.text}")
That's it. No model download. No GPU. No network call. SQLite handles persistence, so your index survives restarts. BM25 query latency is under 1 millisecond on my M2 Mac Mini.
This alone covers a lot of use cases. If you're building a documentation search, a CLI "grep but smarter" tool, or a keyword router for an agent system, BM25 gets you 80% of the way there with zero dependencies beyond numpy.
When BM25 isn't enough
BM25 is keyword matching. It finds documents that share terms with your query. It doesn't know that "GPU" and "graphics card" mean the same thing. It doesn't handle typos or paraphrasing.
Dense embedding fixes this. You encode each document into a vector, encode the query, and measure cosine similarity. The catch: you need a model, and running it takes real compute.
Hippo's hybrid mode combines both using Reciprocal Rank Fusion (RRF). BM25 catches exact matches. Dense embedding catches semantic matches. RRF merges the rankings without needing score calibration — BM25 scores and cosine similarities live on completely different scales, but rank fusion sidesteps that problem.
from hippo.embedding import EmbeddingEngine, VectorStore
# Requires: pip install hippo-llm[embedding]
engine = EmbeddingEngine(model="nomic-embed-text") # runs locally
store = VectorStore("docs.db", mode="hybrid", embedding_engine=engine)
store.add_batch([
{"text": "Pipeline parallelism splits layers across devices"},
{"text": "BM25 handles exact keyword matches"},
], engine=engine)
results = store.search("how to run big models on small GPUs", engine=engine, top_k=5)
Same API surface. The mode parameter switches between sparse, dense, and hybrid. SQLite still handles storage.
Real accuracy numbers
I benchmarked this on 110 out-of-domain queries (legal documents indexed, technical questions asked — cross-domain stress test):
| Setup | Top-1 Accuracy | Latency |
|---|---|---|
| BM25 only | ~55% | <1ms |
| bge-small-zh (512d, dense) | 85.5% | 5ms |
| bge-m3 (1024d, dense) | 92.7% | 630ms |
| BM25 + bge-small-zh (hybrid RRF) | 91.8% | ~6ms |
The hybrid result is the interesting one. A 183MB model fused with BM25 hits 91.8% — within 1% of the 2.2GB model running standalone. The fusion effect is real: BM25 and embedding errors don't overlap much, so combining them recovers ground that either approach misses alone.
For production, I run bge-small-zh + BM25 hybrid. 5ms per query, 183MB model, runs fine on a Mac Mini with 16GB RAM.
Chinese text, no jieba
Most Chinese NLP tools assume you'll install jieba for word segmentation. Hippo has a built-in tokenizer with Chinese stop words. No extra dependency, no configuration.
store.add_batch([
{"text": "管道并行将模型层拆分到多台设备上"},
{"text": "混合搜索结合了关键词匹配和语义相似度"},
])
results = store.search("怎么在低端显卡上跑大模型", top_k=3)
This matters more than it sounds. Chinese BM25 is sensitive to tokenization quality — bad segmentation produces garbage term frequencies. I tested against jieba-based pipelines and got comparable results with the built-in tokenizer, without the dependency chain.
A full RAG pipeline
Search is half of RAG. The other half is generating an answer from retrieved context. Hippo doesn't ship a chatbot — it gives you the retrieval layer and gets out of the way.
Here's a complete RAG setup using Hippo for retrieval and any OpenAI-compatible local LLM for generation:
from hippo.embedding import EmbeddingEngine, VectorStore
import openai
# 1. Index your documents (one-time)
engine = EmbeddingEngine(model="nomic-embed-text")
store = VectorStore("knowledge.db", mode="hybrid", embedding_engine=engine)
documents = [
"Pipeline parallelism splits model layers across multiple devices via TCP.",
"Each device loads only its shard of layers, reducing memory per device.",
"The loop detector catches semantic repetition using Jaccard similarity.",
"Hybrid search combines keyword matching with semantic similarity via RRF.",
]
store.add_batch([{"text": d} for d in documents], engine=engine)
# 2. Retrieve relevant context
query = "how does hippo handle memory?"
results = store.search(query, engine=engine, top_k=2)
context = "\n".join(doc.text for doc in results)
# 3. Generate with any local LLM
client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="none")
response = client.chat.completions.create(
model="qwen3-30b-a3b-q3",
messages=[
{"role": "system", "content": f"Answer based on this context:\n{context}"},
{"role": "user", "content": query}
]
)
print(response.choices[0].message.content)
The LLM endpoint can be Hippo's own server, Ollama, vLLM, or anything that speaks OpenAI API. The retrieval layer doesn't care.
Why not just use ChromaDB?
You can. ChromaDB is a solid vector database. If you're already using it and happy, there's no reason to switch.
Hippo's VectorStore exists for the case where you want search without standing up a separate service. It's a Python class backed by SQLite. No server process, no Docker container, no client library. You instantiate it, add documents, and query. The database file is a single SQLite file you can copy, back up, or ship with your application.
For my use case — searching 2,000 internal documents on a single machine — this is the right amount of infrastructure. I don't need a distributed vector database. I need a Python import that works in 30 seconds.
There's also a debugging argument. When ChromaDB returns unexpected results, you're debugging a client-server protocol, a collection abstraction, and an embedding pipeline simultaneously. With Hippo, the entire search stack is one Python class. If something goes wrong, you set a breakpoint in VectorStore.search() and step through. The BM25 scoring function is 40 lines of math you can read.
Privacy: not a feature, a constraint
I want to be specific about what "local" means here, because the term gets abused in marketing copy.
Hippo's sparse mode (BM25 only) makes zero network calls. No telemetry, no model download, no phone-home check. You can verify this by reading the source — it's about 300 lines of Python, and the only imports are sqlite3, math, re, and numpy.
Hybrid mode downloads an embedding model from HuggingFace on first run. After that, it runs offline. The model weights live on your disk. Inference happens in-process. Your query text and document text never leave the machine.
This isn't a selling point. It's the baseline requirement for a lot of professional work. If you're a doctor searching patient records, a lawyer searching case files, or an engineer searching proprietary source code — the "just call the OpenAI API" stack was never an option for you.
I learned this the hard way. Early in my career, I used a cloud-based document analysis tool for a contract review. The tool was convenient. It also uploaded my client's entire merger agreement to a third-party server. My client was not amused. That incident is the direct origin of Hippo's architecture — every design decision starts from "what stays local?" and works backward from there.
For regulated industries, this isn't paranoia. HIPAA, GDPR, attorney-client privilege, trade secret protection — they all impose real constraints on where data can live. A search tool that requires network access to an embedding API is a compliance liability, not a productivity win.
When to pick what
| You want... | Use this |
|---|---|
| Keyword search, zero setup |
VectorStore("docs.db") — BM25, done |
| Semantic search with local model |
pip install hippo-llm[embedding], hybrid mode |
| Chinese document search | Built-in tokenizer, same API |
| Agent memory / semantic routing | Hybrid RRF, 5ms per query |
| Full RAG pipeline | Hippo retrieval + any local LLM |
Getting started
pip install hippo-llm
Then in Python:
from hippo.embedding import VectorStore
store = VectorStore("docs.db")
store.add_batch([{"text": "hello world"}])
print(store.search("hello", top_k=1))
If you need hybrid search: pip install hippo-llm[embedding] and switch to mode="hybrid".
The full README with benchmarks, API reference, and configuration options is at github.com/lawcontinue/hippo. MIT licensed. Contributions welcome.
The author is a practicing lawyer. Hippo was built to solve a real document search problem in a profession where data confidentiality isn't optional.
Top comments (0)