DEV Community

diogo
diogo

Posted on

I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests

I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests

Ok, I made a thing! LLMs have their... quirks. Namely re-reading the same notes, wandering back and forth, and spending 10x more tokens than needed because I can't be explicit.

So I built Haystack — a semantic router for a markdown vault. Instead of the agent digging through files, it queries: hybrid retrieval (embeddings + corpus-IDF term stats, RRF-fused, cross-encoder reranking via llama.cpp), all local, zero cloud, one 35 MB model, ~10 ms per query. MIT, one-command setup: github.com/diogocccarvalho/haystack.

The part I don't see elsewhere

Your corrections become regression tests. Every retrieval miss you fix becomes a permanent rule AND a QA pair. The audit reindexes and runs every pair — it exits 1 on a regression, so the same question can't silently fail twice.

And the learned rules only pin when they actually beat retrieval: a stored phrasing must outrank the corpus' own best hit for the same query. Scale-free, not absolute-threshold — so swapping embedders doesn't break your corrections. I A/B'd models against frozen QA sets and rejected two (a reranker that was 2x faster but -16pp Recall@3; a 97M embedder at -7pp). Numbers in docs/RESEARCH.md.

The numbers, honestly

Corpus Recall@3 MRR
English vault (68 pairs, nickname-style) 1.000 0.917
Portuguese vault (14 pairs, synthetic) 1.000 1.000
Foreign corpus (llama.cpp docs, 33 pairs) 1.000 0.970

The foreign-corpus row matters more than the home-vault row: the chunker generalizes beyond the corpus it was tuned on. Also: llama-server does both the embeddings (bge-small-en GGUF) and the reranking (--reranking flag on ms-marco MiniLM-L12) — one binary, tested with llama.cpp v1.0.0.

The honest part

The agent that uses Haystack daily also built most of it with me. Every gate in the repo was run by the agent against its own index — that's how the setup-script bugs got caught. And docs/SHORTCOMINGS.md is the failure catalog: ten real failures (a reranker overriding user corrections live; a rebuild that once deleted every correction; an over-context chunk killing a whole batch), each one now a design rule.

Try it

./setup.sh --start          # model + config + embeddings server, one command
python3 haystack.py reindex
python3 haystack.py query "that thing about the invoices"
Enter fullscreen mode Exit fullscreen mode

Any OpenAI-compatible endpoint works instead (Ollama, LM Studio, remote API). Python 3.11+, stdlib + numpy only — no framework, no vector DB daemon, no cloud.

The open question I'd genuinely love answers to: this was tuned entirely on my vault. I want to know how the default config holds up on other people's notes. If you run it, tell me what breaks.

Top comments (4)

Collapse
 
reidmarlow profile image
Reid Marlow •

Reading SHORTCOMINGS.md, R4 (index rebuilds wiping learned corrections) and R2 (the degraded fallback dropping pinned routes when the cross-encoder is down) are the two traps most retrieval pipelines miss until weeks in.

Keeping learned.json outside the rebuildable index store fixes the maintenance wipe, though one edge case shows up once you re-run haystack.py reindex after renaming or splitting notes in a vault. If a taught route points to a stale relative path after a file move, pinning it ahead of retrieval will surface a dead path unless the audit step verifies file existence or matches moved targets by content hash. Returning pinned + lexical_order in the reranker-down fallback keeps behavior consistent across degraded runs too.

Collapse
 
hdac2k profile image
Diogo •

You found a real issue (thanks)!
The audit function does flag that issue, when audit runs. However between audit calls we'd get dead answers.
Even on renaming, as long as the file stays on the folder, we'd boost the folder (loses accuracy though)

Collapse
 
recallrundev profile image
RecallRun •

Corrections becoming QA pairs that exit 1 on regression is a good loop. A 30-question golden set in CI caught 3 of 4 retrieval regressions we planted, so even a small frozen set earns its keep.

One question on the pins: what happens to a learned rule when the chunker changes? When we moved to heading-aware chunks on the FastAPI docs, hit@1 doubled, but every chunk boundary moved. If a rule pins a chunk id, the QA pair still runs after a re-chunk but the rule may now point at a different span. Do you key the rules to chunk ids, or to something that survives re-chunking like (file, heading)?

Collapse
 
hdac2k profile image
Diogo •

Hi! So essentially, even if we re-chunk it we use file paths as the pin. So even if we change the chunking rules it still survives. The one thing it does not survive is file renaming(loses some accuracy)