<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: diogo</title>
    <description>The latest articles on DEV Community by diogo (@diogocccarvalho).</description>
    <link>https://dev.to/diogocccarvalho</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1853134%2F185f66ad-b333-4438-abae-7292b292bc89.png</url>
      <title>DEV Community: diogo</title>
      <link>https://dev.to/diogocccarvalho</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/diogocccarvalho"/>
    <language>en</language>
    <item>
      <title>I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests</title>
      <dc:creator>diogo</dc:creator>
      <pubDate>Fri, 13 Nov 2026 22:06:33 +0000</pubDate>
      <link>https://dev.to/diogocccarvalho/i-made-my-ai-agents-notes-vault-retrievable-35-mb-embedder-llamacpp-reranker-corrections-as-4k34</link>
      <guid>https://dev.to/diogocccarvalho/i-made-my-ai-agents-notes-vault-retrievable-35-mb-embedder-llamacpp-reranker-corrections-as-4k34</guid>
      <description>&lt;h1&gt;
  
  
  I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests
&lt;/h1&gt;

&lt;p&gt;Ok, I made a thing! LLMs have their... quirks. Namely re-reading the same notes, wandering back and forth, and spending 10x more tokens than needed because I can't be explicit.&lt;/p&gt;

&lt;p&gt;So I built &lt;strong&gt;Haystack&lt;/strong&gt; — a semantic router for a markdown vault. Instead of the agent digging through files, it queries: hybrid retrieval (embeddings + corpus-IDF term stats, RRF-fused, cross-encoder reranking via llama.cpp), all local, zero cloud, one 35 MB model, ~10 ms per query. MIT, one-command setup: &lt;a href="https://github.com/diogocccarvalho/haystack" rel="noopener noreferrer"&gt;github.com/diogocccarvalho/haystack&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part I don't see elsewhere
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Your corrections become regression tests.&lt;/strong&gt; Every retrieval miss you fix becomes a permanent rule AND a QA pair. The audit reindexes and runs every pair — it exits 1 on a regression, so the same question can't silently fail twice.&lt;/p&gt;

&lt;p&gt;And the learned rules only pin when they actually beat retrieval: a stored phrasing must outrank the corpus' own best hit for the same query. Scale-free, not absolute-threshold — so swapping embedders doesn't break your corrections. I A/B'd models against frozen QA sets and rejected two (a reranker that was 2x faster but -16pp Recall@3; a 97M embedder at -7pp). Numbers in &lt;a href="https://github.com/diogocccarvalho/haystack/blob/main/docs/RESEARCH.md" rel="noopener noreferrer"&gt;docs/RESEARCH.md&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers, honestly
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Corpus&lt;/th&gt;
&lt;th&gt;Recall@3&lt;/th&gt;
&lt;th&gt;MRR&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;English vault (68 pairs, nickname-style)&lt;/td&gt;
&lt;td&gt;1.000&lt;/td&gt;
&lt;td&gt;0.917&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Portuguese vault (14 pairs, synthetic)&lt;/td&gt;
&lt;td&gt;1.000&lt;/td&gt;
&lt;td&gt;1.000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Foreign corpus (llama.cpp docs, 33 pairs)&lt;/td&gt;
&lt;td&gt;1.000&lt;/td&gt;
&lt;td&gt;0.970&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The foreign-corpus row matters more than the home-vault row: the chunker generalizes beyond the corpus it was tuned on. Also: &lt;code&gt;llama-server&lt;/code&gt; does both the embeddings (bge-small-en GGUF) and the reranking (&lt;code&gt;--reranking&lt;/code&gt; flag on ms-marco MiniLM-L12) — one binary, tested with llama.cpp v1.0.0.&lt;/p&gt;

&lt;h2&gt;
  
  
  The honest part
&lt;/h2&gt;

&lt;p&gt;The agent that uses Haystack daily also built most of it with me. Every gate in the repo was run by the agent against its own index — that's how the setup-script bugs got caught. And &lt;a href="https://github.com/diogocccarvalho/haystack/blob/main/docs/SHORTCOMINGS.md" rel="noopener noreferrer"&gt;docs/SHORTCOMINGS.md&lt;/a&gt; is the failure catalog: ten real failures (a reranker overriding user corrections live; a rebuild that once deleted every correction; an over-context chunk killing a whole batch), each one now a design rule.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;./setup.sh --start          &lt;span class="c"&gt;# model + config + embeddings server, one command&lt;/span&gt;
python3 haystack.py reindex
python3 haystack.py query &lt;span class="s2"&gt;"that thing about the invoices"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Any OpenAI-compatible endpoint works instead (Ollama, LM Studio, remote API). Python 3.11+, stdlib + numpy only — no framework, no vector DB daemon, no cloud.&lt;/p&gt;

&lt;p&gt;The open question I'd genuinely love answers to: this was tuned entirely on my vault. I want to know how the default config holds up on other people's notes. If you run it, tell me what breaks.&lt;/p&gt;

</description>
      <category>llamacpp</category>
      <category>rag</category>
      <category>opensource</category>
      <category>python</category>
    </item>
  </channel>
</rss>
