My knowledge base grew across thousands of markdown files. Finding one fact meant reading whole files. Keyword search found words, not meaning. And reading whole files burns tokens: tens of thousands of tokens to surface one fact. The real metric is not storage. It is tokens per answer.
Why not the usual tools?
- grep finds the file, but you still read it. Cost unchanged.
- A cloud vector DB sends private notes out and adds infra for a 2 MB corpus.
- Semantic-only search cannot do exact filters.
What I built: a local-first retrieval layer.
- SQLite FTS5 index: instant keyword search, exact filters, real file:line references.
- Local ONNX embeddings (384-dim): semantic ranking. Every hit points at a real file.
- One-command rebuild. The index is derived, so it never goes stale.
Result: 2,000 chunks, one CLI, zero servers. An answer now costs a few hundred tokens, not a sweep through thousands of files.
Start deterministic. Add semantic where keywords fail. Rebuild often.
Top comments (1)
start deterministic, add semantic where keywords fail, rebuild often is a better rule than most of what gets written about this. one FTS5 detail that pays off immediately on a notes corpus.
bm25() in FTS5 takes per-column weights, and if your chunks are one column you're leaving the biggest win on the table. split the heading path into its own column and weight it five to ten times the body. on markdown notes the heading is usually a near-perfect statement of what the chunk is about, and unweighted BM25 treats a term in an H2 exactly like a term in a parenthetical.
related trap at your scale: length normalization is doing more than you'd think across 2,000 chunks of very uneven size. a 40 line note and a 900 line note compete on different terms and short files win queries they shouldn't. worth eyeballing before you reach for the embeddings, because it presents as "semantic search would fix this" and it isn't a semantic problem.
the file:line requirement is the thing i'd defend hardest if anyone pushes back on the design. a similarity score is not a citation. you can't check it.