<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Bart de Goede</title>
    <description>The latest articles on DEV Community by Bart de Goede (@bartdegoede).</description>
    <link>https://dev.to/bartdegoede</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4027237%2F65297504-8585-4a54-a04b-fd7fe3382ff0.jpg</url>
      <title>DEV Community: Bart de Goede</title>
      <link>https://dev.to/bartdegoede</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/bartdegoede"/>
    <language>en</language>
    <item>
      <title>Running RAG with a (small) language model in the browser</title>
      <dc:creator>Bart de Goede</dc:creator>
      <pubDate>Tue, 13 Oct 2026 03:45:00 +0000</pubDate>
      <link>https://dev.to/bartdegoede/running-rag-with-a-small-language-model-in-the-browser-2d05</link>
      <guid>https://dev.to/bartdegoede/running-rag-with-a-small-language-model-in-the-browser-2d05</guid>
      <description>&lt;p&gt;A few weeks ago &lt;a href="https://stateofutopia.com/experiments/microllmlab/" rel="noopener noreferrer"&gt;MicroLLM lab&lt;/a&gt; was featured in &lt;a href="https://buttondown.com/hacker-newsletter/archive/812/" rel="noopener noreferrer"&gt;Hacker Newsletter&lt;/a&gt;. It shows off how you can run small language models (25 to 360 million parameters) in a browser on the users' own GPU (or CPU)&lt;sup id="fnref1"&gt;1&lt;/sup&gt;. No server, no API key, no account. Pretty nifty, so it got me thinking about the steps after &lt;a href="https://dev.to/semantic-search-in-your-browser/"&gt;semantic search I added to this blog&lt;/a&gt;: that already provides the "R" in RAG&lt;sup id="fnref2"&gt;2&lt;/sup&gt; for a question, and it does that entirely in the browser, from a 4 MB lookup table. What if a tiny model, also in the browser, read those passages and answered the question?&lt;/p&gt;

&lt;p&gt;So I built that: &lt;a href="https://dev.to/ask/"&gt;"ask my blog"&lt;/a&gt;&lt;sup id="fnref3"&gt;3&lt;/sup&gt;. Search finds the four most relevant passages, and a small language model running in your browser summarises them in two sentences. Nothing you type leaves your machine. And because "a small model summarizes things" is exactly the sort of sentence that hides a lot of made-up nonsense, I measured how often it's actually faithful to what I wrote. Turns out small models have some limitations.&lt;/p&gt;

&lt;h2&gt;
  
  
  How it works
&lt;/h2&gt;

&lt;p&gt;The page is retrieval-augmented generation (RAG) with every piece running client-side:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Retrieve.&lt;/strong&gt; Your question goes through the same static-embedding search as the homepage, but instead of rolling results up to posts, it keeps the four best-scoring &lt;em&gt;passages&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gate.&lt;/strong&gt; If the best passage scores below a threshold, the model isn't called at all and the page says it couldn't find this on the blog. The idea is to not bother if a user asks something there's no data for. More on why that only half works below.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt.&lt;/strong&gt; The passages, numbered, plus a system prompt: answer in at most two sentences, use only these passages, say so if they don't answer the question.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Generate.&lt;/strong&gt; A Web Worker runs the model with &lt;a href="https://huggingface.co/docs/transformers.js" rel="noopener noreferrer"&gt;transformers.js&lt;/a&gt; on WebGPU, or on the CPU via WebAssembly if your browser doesn't have WebGPU, and streams the answer into the page. It stops as soon as it has two complete sentences.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The passages are shown first, as cards, and the summary will sit above them. You can search without loading a model at all; you just get the passages. Loading one is an explicit click, because it means downloading a few hundred megabytes from Hugging Face (once; your browser caches it). There's also a button to clear the cache/&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fevg1bwo8rh4cw57be7s0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fevg1bwo8rh4cw57be7s0.png" alt="The download indicator: one square per 4 MB of model" width="800" height="168"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Which model?
&lt;/h2&gt;

&lt;p&gt;I wanted something that would be a reasonable click for a reader: a few hundred megabytes, not gigabytes (what if someone reads this on the bus). The candidates that have browser-ready (ONNX) builds were SmolLM2 (&lt;a href="https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct" rel="noopener noreferrer"&gt;135M&lt;/a&gt; and &lt;a href="https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct" rel="noopener noreferrer"&gt;360M&lt;/a&gt; parameters), &lt;a href="https://huggingface.co/onnx-community/gemma-3-270m-it-ONNX" rel="noopener noreferrer"&gt;Gemma 3 270M&lt;/a&gt;, &lt;a href="https://huggingface.co/onnx-community/LFM2.5-350M-ONNX" rel="noopener noreferrer"&gt;LFM2.5-350M&lt;/a&gt;, &lt;a href="https://huggingface.co/onnx-community/Qwen2.5-0.5B-Instruct" rel="noopener noreferrer"&gt;Qwen2.5-0.5B&lt;/a&gt; and &lt;a href="https://huggingface.co/onnx-community/Qwen3-0.6B-ONNX" rel="noopener noreferrer"&gt;Qwen3-0.6B&lt;/a&gt;. For comparison I added two references that would never ship on the page: &lt;a href="https://huggingface.co/onnx-community/gemma-4-E2B-it-ONNX" rel="noopener noreferrer"&gt;Gemma 4 E2B&lt;/a&gt;, the smallest of Google's new Gemma 4 models at a 3.1 GB download, and &lt;a href="https://www.anthropic.com/claude-haiku-5-5" rel="noopener noreferrer"&gt;Claude Haiku 5.5&lt;/a&gt; through the API, the "just pay someone" option.&lt;/p&gt;

&lt;p&gt;One surprise before anything ran: for several of these models the "4-bit" (q4) build is &lt;em&gt;bigger&lt;/em&gt; than the 4-bit-with-16-bit-floats (q4f16) one, because the q4 export keeps the embedding table in full 32-bit precision. SmolLM2-360M is 390 MB as q4 and 275 MB as q4f16, with identical output. Everything below uses q4f16.&lt;/p&gt;

&lt;h2&gt;
  
  
  The eval
&lt;/h2&gt;

&lt;p&gt;I reused the 30 questions from the &lt;a href="https://dev.to/semantic-search-in-your-browser/"&gt;eval I did for the semantic search post&lt;/a&gt;, rephrased as actual questions ("How did the blog use Lunr to add search?"), and added 15 questions this blog definitely &lt;em&gt;doesn't&lt;/em&gt; answer. A few of those are far off ("How do I bake sourdough bread at home?"), but most are near-misses designed to tempt a model into predicting plausible-sounding-but-wrong-tokens: "Which GPU should I buy to run Llama 70B locally?", "How does BM25 ranking improve on TF-IDF?", "How do I fine-tune Kokoro to speak in my own voice?" The blog talks &lt;em&gt;about&lt;/em&gt; these, or similar things, but doesn't have and answer for those.&lt;/p&gt;

&lt;p&gt;Every model got the same four passages and the same prompt, and Opus 5.5 graded each answer against those passages as &lt;strong&gt;faithful&lt;/strong&gt; (every claim is in the passages), &lt;strong&gt;unsupported&lt;/strong&gt; (at least one claim isn't, even if it happens to be true) or a &lt;strong&gt;non-answer&lt;/strong&gt;. I labelled 20 of those grades myself without seeing the judge's verdict; we agreed on 16 of 20 about whether an answer was faithful&lt;sup id="fnref4"&gt;4&lt;/sup&gt;. Claude also generated the question set, which I reviewed.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh1o932w540e0s265yud9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh1o932w540e0s265yud9.png" alt="Faithful answers out of 30 answerable questions, against download size" width="800" height="428"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frwft2yylvbpsdcb8sfdz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frwft2yylvbpsdcb8sfdz.png" alt="Evals" width="800" height="488"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Turns out nothing off the shelf under 350 MB is usable for this. They aren't slightly worse; they're faithful &lt;strong&gt;at most three times out of thirty&lt;/strong&gt;, and they'll happily answer almost any question you throw at them. Asked which GPU to buy for Llama 70B, with four passages about embeddings and a coding agent on a MacBook:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"...the best option to run Llama 70B locally is likely to be a NVIDIA GTX 960 or GTX 970. [...] They also offer high memory bandwidth, making them ideal for handling large amounts of data such as the 6.4 million documents required by Llama 70B." (SmolLM2-360M)&lt;/p&gt;

&lt;p&gt;"...a high-performance graphics card (GPU) such as a NVIDIA A100 series (e.g., RTX 3090, RTX 3080, 3070, 3080, 3070M, 3080M)" (Gemma 3 270M)&lt;/p&gt;

&lt;p&gt;"...popular options include RTX 3090, RTR 5900, or newer generations like TensorRT 60+ for improved compute efficiency." (LFM2.5-350M)&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Gemma 4 E2B, for the record, correctly decided that "The provided passages do not contain information about which GPU you should buy to run Llama 70B locally." Good job Google.&lt;/p&gt;

&lt;p&gt;Even on questions the blog &lt;em&gt;does&lt;/em&gt; answer, the small models are prone to garble things. Here's LFM2.5 on the most basic question about my &lt;a href="https://dev.to/bloom-filters-bit-arrays-recommendations-caches-bitcoin/"&gt;Bloom filter post&lt;/a&gt;, at 73 tokens per second on my laptop's GPU:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg4x3jpkolpoqspcu1k9q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg4x3jpkolpoqspcu1k9q.png" alt="LFM2.5-350M explaining Bloom filters" width="800" height="371"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Fast, fluent, and wrong. A Bloom filter isn't a hash table: it's a bit array, and an element counts as "probably in the set" only if every one of its &lt;em&gt;k&lt;/em&gt; hash functions lands on a bit that's set. The last clause is actually correct (a missing bit means the element was never added, which is why Bloom filters can't give false negatives), but the rest is made up. To be fair to the model, search didn't hand it the section that explains how Bloom filters work; the four passages it got were about use cases and the interactive demo. A summary can only be as good as the passages it reads. Seems like there's some work to do on the relevance scoring in my little search engine that could.&lt;/p&gt;

&lt;p&gt;Qwen3-0.6B is the interesting one. At 579 MB it's pretty thicc, and over the budget I'd set myself. It's faithful half the time, which is basically a coin flip. However, it's the smallest model that's a coin flip rather than a pretty much guaranteed miss, so that's what &lt;a href="https://dev.to/ask/"&gt;"ask my blog"&lt;/a&gt; loads by default. You can also choose three smaller models so you can see for yourself what a non-finetuned 300 MB model does with your question&lt;sup id="fnref5"&gt;5&lt;/sup&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Small models loop
&lt;/h2&gt;

&lt;p&gt;The first time I ran the small models, most of their answers weren't wrong so much as stuck. SmolLM2-135M, asked how to add OpenSearch support to a website:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"1. Custom Search: Search from your URL bar 2. Search from your URL bar 3. Search from your URL bar 4. Search from your URL bar [1] Custom Search: Search from your URL bar..."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Gemma 3 270M answered the GPU question with &lt;code&gt;I'5555555555555555555555...&lt;/code&gt; until it ran out of tokens. That's what happens when you always pick the single most likely next token and nothing discourages repeating it: tiny models fall into a groove and can't get out (I saw it happen pretty often when I was playing around with MicroLLMLab too). The easy fix is a repetition penalty (1.15 in this case, which slightly discourages tokens the model has already seen). It doesn't fix the faithfulness or quality; the models just write longer, more varied nonsense instead. All the numbers above are with the penalty set to 1.15.&lt;/p&gt;

&lt;h2&gt;
  
  
  A relevance gate to catch the obvious ones
&lt;/h2&gt;

&lt;p&gt;A relevance gate seemed like the cheap fix for made-up answers: if search can't find anything relevant, don't ask the model. It works for the far-off questions; "How do I bake sourdough bread?" scores 0.25 against the best passage on the blog, and a threshold of 0.45 stops it (which is good, because Gemma 3's answer was a bread recipe). But the near-misses score 0.54 to 0.63, &lt;em&gt;higher&lt;/em&gt; than several questions the blog actually answers, because they land on posts about the same topic. To block those, the threshold would have to reject 13 of the 30 real questions too.&lt;/p&gt;

&lt;p&gt;So the gate only catches the obviously off-topic questions, and the rest is on the model. That's the real reason size matters here: Gemma 4 and Haiku say "the passages don't cover this", and the small models never do.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is it worth renting a GPU, or paying for an API?
&lt;/h2&gt;

&lt;p&gt;Usually, yes. At current pricing (yay competition) it's almost never worth it, unless you have strong reasons not to (ie compliance, privacy, air gapped deployments, etc). As a thought exercise tho: the 3.1 GB Gemma 4 E2B matches Haiku on quality, so what would it cost to run it for readers instead of making them download it? The cheapest sensible cloud GPU for it, an NVIDIA L4 on Google Cloud, is about $0.71 an hour&lt;sup id="fnref6"&gt;6&lt;/sup&gt;, or &lt;strong&gt;$516 a month&lt;/strong&gt; to keep running. For the amount of readers this blog sees in a month, assuming each and everyone asks a question, that would be over a dollar per question.&lt;/p&gt;

&lt;p&gt;Haiku on the other hand answered all 45 eval questions for &lt;strong&gt;$0.006&lt;/strong&gt;. At 500 questions a month (a guy can dream) it would cost about seven cents. In other words, if you have a real use case, start with Haiku&lt;sup id="fnref7"&gt;7&lt;/sup&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Shorter is safer
&lt;/h2&gt;

&lt;p&gt;If two sentences from Qwen3 are only right half the time, would letting it say more help? I ran a quick experiment with Qwen3-0.6B and Gemma 4 E2B, allowing for two, four or six sentences.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7eutupu6cm2upr434sjw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7eutupu6cm2upr434sjw.png" alt="Faithful answers out of 30 when the model may use 2, 4 or 6 sentences" width="800" height="452"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Both get &lt;em&gt;less&lt;/em&gt; faithful with more room, but the smaller model falls off a cliff: Qwen3 goes from 15 to 9 to 5 faithful answers out of 30. Every extra sentence is another chance to hallucinate. At six sentences it stops declining the GPU question altogether and helpfully summarises every passage instead, before concluding that "the answer would depend on whether the user needs a high-performance GPU or a standard one."&lt;/p&gt;

&lt;p&gt;Gemma 4 slips too (28, 26, 24), but it barely uses the extra room (about 2.8 sentences when allowed 6) and never makes up an answer to a question the blog doesn't cover. Shoutout to Gemma 4 btw, this model family is pretty awesome. This is a great option for finetuning.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;p&gt;The next step would be making the small models less bad at this one job. A 300 MB model doesn't need to know everything; it needs to read four passages and either summarise them or say they don't answer the question. That sounds like a fine-tuning problem, and maybe a follow-up post.&lt;/p&gt;

&lt;p&gt;The other idea is the homepage search box: if you've already loaded a model on &lt;a href="https://dev.to/ask/"&gt;"ask my blog"&lt;/a&gt;, the homepage could quietly use it to improve search, without ever downloading anything on its own. In the meantime: &lt;a href="https://dev.to/ask/"&gt;ask my blog something&lt;/a&gt;, and check the passages before you believe the summary.&lt;/p&gt;




&lt;ol&gt;

&lt;li id="fn1"&gt;
&lt;p&gt;It was in turn inspired by &lt;a href="https://github.com/yangqi0/petitgpt" rel="noopener noreferrer"&gt;petitgpt&lt;/a&gt;. I didn't reuse any MicroLLM lab code (the repository doesn't say what license it's under); the page here is built on transformers.js directly.&amp;nbsp;↩&lt;/p&gt;
&lt;/li&gt;

&lt;li id="fn2"&gt;
&lt;p&gt;&lt;a href="https://en.wikipedia.org/wiki/Retrieval-augmented_generation" rel="noopener noreferrer"&gt;Retrieval-augmented generation&lt;/a&gt;. The TLDR is that you use a search tool or engine to find the appropriate context for a LLM to do its inference. It's like how Claude Code greps through your code, or when ChatGPT searches the web. Is this overkill for the couple of posts I have? Absolutely. Fun tho.&amp;nbsp;↩&lt;/p&gt;
&lt;/li&gt;

&lt;li id="fn3"&gt;
&lt;p&gt;Naming things is the hardest thing in computer science amirite.&amp;nbsp;↩&lt;/p&gt;
&lt;/li&gt;

&lt;li id="fn4"&gt;
&lt;p&gt;Most of the remaining disagreement was about &lt;em&gt;which kind&lt;/em&gt; of bad an answer was (I'd call rambling nonsense a non-answer; the judge calls it unsupported if it contains a single invented claim). On the faithful-or-not question the judge was stricter than I was twice, and more lenient twice. The gap between the small models and the rest is far bigger than that noise.&amp;nbsp;↩&lt;/p&gt;
&lt;/li&gt;

&lt;li id="fn5"&gt;
&lt;p&gt;Two of them only on WebGPU; see the CPU caveats further down.&amp;nbsp;↩&lt;/p&gt;
&lt;/li&gt;

&lt;li id="fn6"&gt;
&lt;p&gt;A &lt;code&gt;g2-standard-4&lt;/code&gt; instance (one L4) in us-central1, on demand, via &lt;a href="https://computeprices.com/providers/google/gpus/l4" rel="noopener noreferrer"&gt;computeprices.com&lt;/a&gt; on 12 October 2026.&amp;nbsp;↩&lt;/p&gt;
&lt;/li&gt;

&lt;li id="fn7"&gt;
&lt;p&gt;Don't be like Chipotle tho and &lt;a href="https://github.com/cyberpapiii/chipotlai-max" rel="noopener noreferrer"&gt;let just anyone&lt;/a&gt; rack up your inference bill; more so, if you have a static site, please, don't put your Anthropic/OpenAI API key in there.&amp;nbsp;↩&lt;/p&gt;
&lt;/li&gt;

&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>javascript</category>
      <category>llm</category>
      <category>rag</category>
    </item>
    <item>
      <title>Model2vec is a nifty way to distill embedding models to such a size you could use them for semantic search on a static website :-)</title>
      <dc:creator>Bart de Goede</dc:creator>
      <pubDate>Tue, 14 Jul 2026 01:06:48 +0000</pubDate>
      <link>https://dev.to/bartdegoede/model2vec-is-a-nifty-way-to-distill-embedding-models-to-such-a-size-you-could-use-them-for-semantic-3n33</link>
      <guid>https://dev.to/bartdegoede/model2vec-is-a-nifty-way-to-distill-embedding-models-to-such-a-size-you-could-use-them-for-semantic-3n33</guid>
      <description>&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/bartdegoede/client-side-semantic-search-for-your-static-site-5f84" class="crayons-story__hidden-navigation-link"&gt;Client-side semantic search for your static site&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/bartdegoede" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4027237%2F65297504-8585-4a54-a04b-fd7fe3382ff0.jpg" alt="bartdegoede profile" class="crayons-avatar__image"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/bartdegoede" class="crayons-story__secondary fw-medium m:hidden"&gt;
              Bart de Goede
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                Bart de Goede
                
              
              &lt;div id="story-author-preview-content-4137243" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/bartdegoede" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4027237%2F65297504-8585-4a54-a04b-fd7fe3382ff0.jpg" class="crayons-avatar__image" alt=""&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;Bart de Goede&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/bartdegoede/client-side-semantic-search-for-your-static-site-5f84" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Jul 14&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/bartdegoede/client-side-semantic-search-for-your-static-site-5f84" id="article-link-4137243"&gt;
          Client-side semantic search for your static site
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/rag"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;rag&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/ai"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;ai&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
            &lt;a href="https://dev.to/bartdegoede/client-side-semantic-search-for-your-static-site-5f84#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              &lt;span class="hidden s:inline"&gt;Add&amp;nbsp;Comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            13 min read
          &lt;/small&gt;
            
              &lt;span class="bm-initial crayons-icon c-btn__icon"&gt;
                

              &lt;/span&gt;
              &lt;span class="bm-success crayons-icon c-btn__icon"&gt;
                

              &lt;/span&gt;
            
        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;


</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>nlp</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Client-side semantic search for your static site</title>
      <dc:creator>Bart de Goede</dc:creator>
      <pubDate>Tue, 14 Jul 2026 01:04:15 +0000</pubDate>
      <link>https://dev.to/bartdegoede/client-side-semantic-search-for-your-static-site-5f84</link>
      <guid>https://dev.to/bartdegoede/client-side-semantic-search-for-your-static-site-5f84</guid>
      <description>&lt;p&gt;Eight years ago I &lt;a href="https://dev.to/searching-your-hugo-site-with-lunr/"&gt;added client-side search to this blog with Lunr.js&lt;/a&gt;. It creates an inverted index at build time, ships it as JSON, and matches strings in your browser. No server-side engine required. It has worked fine ever since, in the sense that it finds a post if you type a word that is actually in it.&lt;/p&gt;

&lt;p&gt;Earlier this year I wrote a &lt;a href="https://dev.to/building-a-semantic-search-engine-in-250-lines-of-python/"&gt;semantic search engine in ±250 lines of Python&lt;/a&gt; (the kind that lets you find the "London Beer Flood" when you search for "alcoholic beverage disaster in England," because it understands that beer is alcoholic and a flood is a disaster). That one needs a machine with &lt;code&gt;sentence-transformers&lt;/code&gt; installed and a few hundred megabytes of PyTorch; to serve something like that in a production server requires beefy machines with expensive RAM. Not something you run in a browser tab.&lt;/p&gt;

&lt;p&gt;So this blog had keyword search that runs anywhere and understands nothing, but no semantic search that understands things but can't run anywhere near a static site. This post is about closing that gap: &lt;strong&gt;semantic search that runs entirely in your browser, with no server and no API, where the entire model is essentially a 4 MB lookup table.&lt;/strong&gt; You can try all three of the models I benchmarked, further down, running live on your own hardware (I can't afford GPU clusters).&lt;/p&gt;

&lt;p&gt;Doing this surfaced a couple things, chief among which that the keyword search had a bug for a couple of years. TUrns out that doing proper evals is important.&lt;/p&gt;

&lt;h2&gt;
  
  
  The obvious approach costs 23 megabytes
&lt;/h2&gt;

&lt;p&gt;The models that power &lt;a href="https://dev.to/building-a-semantic-search-engine-in-250-lines-of-python/"&gt;the Python post&lt;/a&gt; (sentence-transformers like &lt;code&gt;all-MiniLM-L6-v2&lt;/code&gt;) can, in fact, run in a browser. &lt;a href="https://github.com/huggingface/transformers.js" rel="noopener noreferrer"&gt;Transformers.js&lt;/a&gt; will download an ONNX build of one and run it on WebAssembly. I benchmarked it, and on my laptop, the quantized model plus its runtime is &lt;strong&gt;23.45 MB&lt;/strong&gt; over the wire, takes about two seconds to load and become available, and then embeds a query in ±18 ms.&lt;/p&gt;

&lt;p&gt;Twenty-three megabytes is a bit rich to embed a search box query, for a blog that has 14 posts. That is roughly a dozen high-resolution photos worth of bytes to download so we can have fancy search. It works, and for some applications it is completely worth it, but it is not something I want to inflict on someone who clicked through to read about bloom filters, especially not on a mobile connection.&lt;/p&gt;

&lt;p&gt;The thing is, you do not need a transformer to embed a search query. You need a vector that's good enough, and there is a much cheaper way to get one.&lt;/p&gt;

&lt;h2&gt;
  
  
  A static embedding model is a lookup table
&lt;/h2&gt;

&lt;p&gt;The trick is a family of models called &lt;a href="https://github.com/MinishLab/model2vec" rel="noopener noreferrer"&gt;model2vec&lt;/a&gt; (the specific ones are named "potion"). They are distilled from a real sentence-transformer, but the result is not a neural network. It is a table.&lt;/p&gt;

&lt;p&gt;Here is the model's entire forward pass. Not a simplification — the whole thing, from the library's source:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;tokenize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;          &lt;span class="c1"&gt;# split into subword token ids
&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;embedding&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;         &lt;span class="c1"&gt;# look up one vector per token
&lt;/span&gt;&lt;span class="n"&gt;vector&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;    &lt;span class="c1"&gt;# average them
&lt;/span&gt;&lt;span class="n"&gt;vector&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;vector&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vector&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# normalise to unit length
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is it. Tokenize, look up a row per token, average, normalize. There is no attention, no layers, no inference. "Running the model" is a handful of array lookups and an average. For &lt;code&gt;potion-base-8M&lt;/code&gt;, the table is 29,528 tokens by 256 dimensions of float32 (about 30 MB) and it reaches &lt;a href="https://huggingface.co/minishlab/potion-retrieval-32M" rel="noopener noreferrer"&gt;81% of MiniLM's retrieval quality&lt;/a&gt; while being, definitionally, a dictionary lookup.&lt;/p&gt;

&lt;p&gt;Thirty megabytes is still too much, but a table is a much friendlier thing to shrink than a transformer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Building the index: chunking, and a cache that never earns its keep
&lt;/h2&gt;

&lt;p&gt;Embeddings are generated at build time. On my laptop, whenever I run &lt;code&gt;hugo&lt;/code&gt;, &lt;a href="https://github.com/bartdegoede/blog/blob/main/scripts/build_search_index.py" rel="noopener noreferrer"&gt;a Python script&lt;/a&gt; walks through the posts, strips out the front matter and the code blocks, splits each post into overlapping 600-character-ish chunks, embeds every chunk, and writes the vectors to a file the browser downloads when a user clicks the search input box.&lt;/p&gt;

&lt;p&gt;The chunking matters more than I expected, because of that averaging step. A static embedding is the &lt;em&gt;mean&lt;/em&gt; of its token vectors, and if you average an entire 15,000-character post into one vector, you get something that points at "generic English prose about software" and not much else. The rare, distinctive words (terms like &lt;code&gt;pydub&lt;/code&gt; or &lt;code&gt;mmh3&lt;/code&gt;) get drowned out by the hundreds of ordinary words around them. Chopping the post into smaller chunks helps keep those signals sharp. I'll come back to this, because it turns out to be the key to why some models beat others.&lt;/p&gt;

&lt;p&gt;I also built an embedding cache, keyed on a hash of each chunk's text, so that re-embedding only touches chunks that changed. This turned out to be a waste of time and tokens. Embedding every chunk of this blog is a few hundred lookups and an average; it takes about ten milliseconds. I built a cache to speed up an operation that is already pretty instantaneous (turns out 14 blog posts is not a lot of data). It would matter more for the MiniLM model, where embedding is a real neural network forward pass, but I'm not shipping that one. So the cache sits there, correct and pointless, and I've left it in as a reminder that overengineering is easy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Shrinking the table: the model's stopword list is hiding in plain sight
&lt;/h2&gt;

&lt;p&gt;The table is 30 MB because it is float32. Quantizing it to int8 makes it a quarter of the size. The catch is that you cannot just clip everything to the same scale, and understanding &lt;em&gt;why&lt;/em&gt; was one of my favorite things I learned building this.&lt;/p&gt;

&lt;p&gt;Look at what the row magnitudes actually are. If you sort every token in &lt;code&gt;potion-base-8M&lt;/code&gt; by the length of its vector, the shortest vectors (i.e. the ones closest to zero) are:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;a  .  ,  -  )  the  to  and  of  in
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the longest are:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;turkmenistan  seychelles  guantanamo  hemingway  vanuatu
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is not a coincidence and it isn't noise either. &lt;strong&gt;The model's stopword list is its row magnitudes.&lt;/strong&gt; When you average token vectors together, a word with a tiny vector barely moves the result, and a word with a big vector dominates it. The model has learned, with no stopword list and no special-casing, that a word like "the" should contribute almost nothing and a word like "guantanamo" should contribute a lot. It's kinda beautiful, and it's sitting right there in the geometry.&lt;/p&gt;

&lt;p&gt;Which is why quantization needs a &lt;em&gt;per row&lt;/em&gt; scale: each token gets its own float32 multiplier, so that the relative magnitudes survive being crushed into int8. However, turns out that it didn't matter much anyway.&lt;/p&gt;

&lt;p&gt;I measured it; I quantized the real table with a single global scale and checked how much it actually degraded the query vectors. The answer was: almost nothing. Aggregate cosine similarity against the original stayed at 0.9998. A global scale zeroes out exactly two rows in the entire 29,528-token vocabulary (&lt;code&gt;.&lt;/code&gt; and &lt;code&gt;a&lt;/code&gt;) which are precisely the two tokens the model had already decided contribute pretty much nothing. The mechanism is real; it just doesn't matter for this particular model.&lt;/p&gt;

&lt;p&gt;I kept the per-row scales anyway, because they cost 118 KB out of 4 MB and they're correct, but more like "cheap insurance" than "load-bearing". The whole int8 table, per-row scales and all, reproduces the original float32 model to a cosine of &lt;strong&gt;0.999958&lt;/strong&gt;. Good enough to ship a lookup table in the browser rather than a model.&lt;/p&gt;

&lt;h2&gt;
  
  
  WordPiece in eighty lines, and its three gotchas
&lt;/h2&gt;

&lt;p&gt;The browser has the token table, but it still has to turn your typed query into token ids the same way we did in Python, so we can look up the right rows. That means reimplementing the BERT WordPiece tokenizer in JavaScript. It's about eighty lines, and it has three gotchas that could each silently poison query vector:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;No &lt;code&gt;[CLS]&lt;/code&gt;/&lt;code&gt;[SEP]&lt;/code&gt;.&lt;/strong&gt; BERT tokenizers normally wrap your text in special marker tokens. The &lt;code&gt;tokenizer.json&lt;/code&gt; config even has a section describing how. &lt;code&gt;model2vec&lt;/code&gt; doesn't use it, but calls the tokenizer with &lt;code&gt;add_special_tokens=False&lt;/code&gt; instead. Add the markers and you're averaging in two vectors that shouldn't be there.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Unknown tokens are &lt;em&gt;deleted&lt;/em&gt;, not embedded.&lt;/strong&gt; If a word isn't in the vocabulary, &lt;code&gt;model2vec&lt;/code&gt; drops it from the sequence entirely rather than substituting an &lt;code&gt;[UNK]&lt;/code&gt; vector. So a query made entirely of gibberish produces an empty token list and a zero vector, and you have to handle that instead of dividing by zero. (In practice this is nearly unreachable; with a 29,528-token vocabulary, every single character is in the vocabulary, so the only way to trigger it is a word longer than 100 characters.)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;"strip_accents": null&lt;/code&gt; means accents &lt;em&gt;are&lt;/em&gt; stripped.&lt;/strong&gt; This one is a little nasty. The config says &lt;code&gt;strip_accents&lt;/code&gt; is null, which reads like "off." But in HuggingFace's tokenizer library, a null value inherits from the &lt;code&gt;lowercase&lt;/code&gt; setting, which is on. So &lt;code&gt;café&lt;/code&gt; becomes &lt;code&gt;cafe&lt;/code&gt;. If we'd copy the config literally, every accented query drifts.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Some frustrations and a bunch of Claude generated test strings ( &lt;code&gt;pydub&lt;/code&gt;, &lt;code&gt;café naïve&lt;/code&gt;, &lt;code&gt;C++ vs C#&lt;/code&gt;, &lt;code&gt;日本語のみ&lt;/code&gt;) later, I'm reasonably confident the &lt;a href="https://github.com/bartdegoede/blog/blob/main/assets/js/semantic/tokenizer.js" rel="noopener noreferrer"&gt;JS tokenizer&lt;/a&gt; matches the BERT WordPiece tokenizer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Search is a few hundred dot products
&lt;/h2&gt;

&lt;p&gt;With the query embedded, search is almost anticlimactic. There are a few hundred chunk vectors (one per chunk of every post). Computing the cosine similarity against all of them is a few hundred dot products of 128 numbers each, tens of thousands of multiply-adds, which a browser on a reasonably modern machine does in a fraction of a millisecond. Then we group the chunks by post, take each post's best-scoring chunk, and sort. On my machine (M1 MacBook Air) the entire query (tokenize, embed, score every chunk, rank) takes about &lt;strong&gt;0.4 milliseconds&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;In a production setting, this is where you reach for approximate-nearest-neighbour indexes (HNSW and friends), and if you have a bazillion documents you should. With a few hundred, an ANN index would be slower than the brute-force loop and much larger on disk. It's worth saying out loud because "vector search" in this case isn't a complex database system,but just a &lt;code&gt;for&lt;/code&gt; loop.&lt;/p&gt;

&lt;p&gt;There's one wrinkle worth knowing if you build one of these. The document vectors are stored as int8. Cosine similarity is invariant to a positive scale, and the document matrix uses one global scale, so the browser can dot a float32 query straight against the raw int8 bytes and get the right ranking without ever un-quantizing them. But &lt;code&gt;int8 × int8&lt;/code&gt; in JavaScript overflows silently; a dot product whose true value is three million comes back as &lt;code&gt;-64&lt;/code&gt;, no error, no warning. So it's better to accumulate into a regular float. I mention it because this was a fun one to debug&lt;sup id="fnref1"&gt;1&lt;/sup&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two search engines that are each blind in a different way
&lt;/h2&gt;

&lt;p&gt;Before adding semantic search, I figured I'd do the responsible thing and measured how bad the existing keyword search actually was, so I'd have a baseline to beat. I had Claude create a set of thirty test queries and split them into three kinds:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Exact tokens&lt;/strong&gt;: &lt;code&gt;pydub&lt;/code&gt;, &lt;code&gt;lunr&lt;/code&gt;, &lt;code&gt;mmh3&lt;/code&gt;, &lt;code&gt;papermod&lt;/code&gt;. Words that literally appear in a post.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paraphrases&lt;/strong&gt;: "find documents by what they mean instead of which words they contain." Concepts, in different words than the post uses.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Navigational&lt;/strong&gt;: "how do I add search to a static site." Broad intent.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Keyword search would surely ace the exact tokens and fail the paraphrases. Then I ran it, and it failed &lt;code&gt;pydub&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;pydub&lt;/code&gt; is a Python library I've used in &lt;a href="https://dev.to/use-google-cloud-text-to-speech-to-create-an-audio-version-of-your-blog-posts/"&gt;a previous post&lt;/a&gt;. The word is right there in the text-to-speech post. And Fuse.js, the fuzzy-search library this blog uses, returned nothing. It turned out my configuration told Fuse to only score matches near the &lt;em&gt;start&lt;/em&gt; of a field; a setting called &lt;code&gt;location&lt;/code&gt;, with a &lt;code&gt;distance&lt;/code&gt; window of 1000 characters. &lt;code&gt;pydub&lt;/code&gt; first appears about 3,700 characters into that post, well outside the window, so as far as my search box was concerned it did not exist. The same was true for &lt;code&gt;mmh3&lt;/code&gt;, and for any distinctive word that happened to appear deep in a long post. The words that &lt;em&gt;did&lt;/em&gt; work only worked because they were in a title.&lt;/p&gt;

&lt;p&gt;My keyword search had been quietly broken for years, and I only found out because I was about to benchmark against it. Derp.&lt;/p&gt;

&lt;p&gt;And the two engines turn out to be blind in exactly complementary ways:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;keyword finds it&lt;/th&gt;
&lt;th&gt;semantic finds it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;pydub&lt;/code&gt; (exact token, out of vocabulary)&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"find documents by what they mean" (paraphrase)&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Keyword search is perfect on the exact tokens and returns &lt;em&gt;literally nothing&lt;/em&gt; for all ten paraphrases; not bad results, just zero results. Semantic search is the mirror image: it nails the paraphrases and stumbles on the proper nouns, because &lt;code&gt;pydub&lt;/code&gt; isn't in its vocabulary and gets shattered into meaningless subword fragments. Neither is better. They fail on disjoint sets.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reciprocal Rank Fusion, or: how to merge different search result sets
&lt;/h2&gt;

&lt;p&gt;If each engine catches what the other misses, you want both. The problem is combining them. Keyword search returns a fuzzy-match &lt;em&gt;distance&lt;/em&gt;; vector search returns a &lt;em&gt;cosine similarity&lt;/em&gt;. These live on different scales that mean different things, and averaging them is meaningless; it'd be like adding a temperature to a weight.&lt;/p&gt;

&lt;p&gt;The answer here is &lt;a href="https://medium.com/@devalshah1619/mathematical-intuition-behind-reciprocal-rank-fusion-rrf-explained-in-2-mins-002df0cc5e2a" rel="noopener noreferrer"&gt;Reciprocal Rank Fusion&lt;/a&gt;. Run queries against both indices, throw away the scores entirely and keep only the &lt;em&gt;ranks&lt;/em&gt;. Each document's "fused score" is the sum, over both engines, of &lt;code&gt;1 / (k + rank)&lt;/code&gt;, with &lt;code&gt;k&lt;/code&gt; conventionally 60. A document ranked first by one engine and unranked by the other scores &lt;code&gt;1/61&lt;/code&gt;. A document ranked third by &lt;em&gt;both&lt;/em&gt; scores &lt;code&gt;1/63 + 1/63&lt;/code&gt;, which is larger. The &lt;code&gt;k&lt;/code&gt; flattens the top of the curve so that "both engines quite liked this" beats "one engine loved it", which is exactly what you want when one engine is blind to paraphrase and the other to proper nouns. It's about ten lines of code, it has a single parameter with a sensible default, and it never has to compare the two incomparable scores. When you search this blog now, that's what runs: keyword and semantic, fused by rank. This is easily extensible to many sources of ranked lists.&lt;/p&gt;

&lt;h2&gt;
  
  
  The benchmark: it'll have to run on your hardware
&lt;/h2&gt;

&lt;p&gt;I benchmarked three query encoders: the model2vec lookup table I've been describing, MiniLM q8 through transformers.js, and &lt;a href="https://github.com/soycaporal/ternlight" rel="noopener noreferrer"&gt;ternlight&lt;/a&gt;, a clever BitNet-style ternary model compiled to WebAssembly that originally sent me down this whole path.&lt;/p&gt;

&lt;p&gt;On my website there is &lt;a href="https://bart.degoe.de/semantic-search-in-your-browser/#the-benchmark-itll-have-to-run-on-your-hardware" rel="noopener noreferrer"&gt;an interactive benchmark&lt;/a&gt; that'll download different query encoders and run embeddings if applicable on our device. I've included my benchmarks below, and a GIF of it running, but definitely check it out for yourself!&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Query encoder&lt;/th&gt;
&lt;th&gt;Over the wire&lt;/th&gt;
&lt;th&gt;Embed a query*&lt;/th&gt;
&lt;th&gt;Retrieval quality†&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;potion&lt;/strong&gt; (model2vec lookup) — &lt;em&gt;what I shipped&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4.21 MB&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~0.3 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;on par with MiniLM&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MiniLM q8 (transformers.js)&lt;/td&gt;
&lt;td&gt;23.45 MB&lt;/td&gt;
&lt;td&gt;~8 ms&lt;/td&gt;
&lt;td&gt;baseline&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ternlight base (BitNet-style ternary, WASM)&lt;/td&gt;
&lt;td&gt;7.17 MB&lt;/td&gt;
&lt;td&gt;between the two&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;*Timings are from my laptop; the live widget measures them on &lt;em&gt;your&lt;/em&gt; device. †Recall@1 over 30 labeled queries.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faay1wrfd0k85uricnu4d.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faay1wrfd0k85uricnu4d.gif" alt=" " width="720" height="688"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;⚡ &lt;strong&gt;These three run live in your browser, on your own hardware.&lt;/strong&gt; I've included the static table above for your reference; you can see a 4 MB lookup table keep pace with a 23 MB transformer on your machine on my website. &lt;strong&gt;&lt;a href="https://bart.degoe.de/semantic-search-in-your-browser/#the-benchmark-itll-have-to-run-on-your-hardware" rel="noopener noreferrer"&gt;Run the interactive benchmark →&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;On my laptop the story is pretty clear: the lookup table is &lt;strong&gt;4.2 MB and embeds a query in a third of a millisecond&lt;/strong&gt;; MiniLM is &lt;strong&gt;23.5 MB and 18 milliseconds&lt;/strong&gt;; ternlight sits in between. And in retrieval quality, measured over those thirty labeled queries, the lookup table scores just as well as MiniLM. &lt;em&gt;Just as well&lt;/em&gt;. At a fifth of the size and roughly fifty times the speed.&lt;/p&gt;

&lt;p&gt;Caveat on that claim, though; thirty queries is a small sample and I nearly fooled myself with it. Twice.&lt;/p&gt;

&lt;h2&gt;
  
  
  The metric couldn't see anything
&lt;/h2&gt;

&lt;p&gt;Initially, the headline metric was &lt;code&gt;recall@3&lt;/code&gt;: did a relevant post make the top three? On a thirteen-post corpus, "top three of thirteen" is not a very high bar (random guessing clears it about a quarter of the time) and it turned out that almost every model cleared it on almost every query. Five different configurations tied at exactly 0.978. &lt;code&gt;Recall@3&lt;/code&gt; could not tell a 4 MB lookup table apart from a 23 MB transformer.&lt;/p&gt;

&lt;p&gt;Worse, it produced confident nonsense. Pure semantic search scored a &lt;em&gt;perfect&lt;/em&gt; 1.000 on the exact-token queries; it apparently found &lt;code&gt;pydub&lt;/code&gt; every time. Except it didn't:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pydub  →  1. Free SSL on GitHub Pages       (wrong)
          2. Let's Encrypt on GitHub Pages   (wrong)
          3. Text-to-Speech with pydub       (correct)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The right post is third, behind two completely unrelated ones, and &lt;code&gt;recall@3&lt;/code&gt; scored that as a hit. The metric was rewarding the model for landing the right answer in a bucket wide enough that landing it there meant almost nothing. Switching to recall@&lt;strong&gt;1&lt;/strong&gt; ("is the first result correct?") separated the models more cleanly and told a different story. Lesson learned: a metric is worthless if the metric can't see the thing you care about. I only caught it because the results looked &lt;em&gt;too&lt;/em&gt; good.&lt;/p&gt;

&lt;h2&gt;
  
  
  What shipped, and how to steal it
&lt;/h2&gt;

&lt;p&gt;The search box on &lt;a href="https://dev.to/"&gt;the homepage&lt;/a&gt; now runs keyword, semantic, and hybrid search, with a toggle so you can compare and watch them disagree. Type &lt;code&gt;pydub&lt;/code&gt; and flip to semantic mode to see it get the answer wrong; flip to hybrid to see it get it right again. The whole thing is a 4 MB lookup table, a tiny document index, and about 300 lines of dependency-free JavaScript, lazy-loaded only when you focus the search box so the page itself pays nothing.&lt;/p&gt;

&lt;p&gt;The build pipeline (the chunking, the quantization, the eval harness, all of it) is a Python package you can point at your own site:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;static-site-search-eval
sss-eval build &lt;span class="nt"&gt;--corpus&lt;/span&gt; content/post &lt;span class="nt"&gt;--outdir&lt;/span&gt; static/search &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--model&lt;/span&gt; minishlab/potion-base-8M &lt;span class="nt"&gt;--dims&lt;/span&gt; 128 &lt;span class="nt"&gt;--chunk-size&lt;/span&gt; 600
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://github.com/bartdegoede/static-site-search-eval" rel="noopener noreferrer"&gt;code is on GitHub&lt;/a&gt;, the thirty eval queries are in there too so you can try it out for yourself. If you run the benchmark above on a phone over cellular, you'll likely feel the 23 MB, and understand why the lookup table was worth the trouble.&lt;/p&gt;




&lt;ol&gt;

&lt;li id="fn1"&gt;
&lt;p&gt;All time classic &lt;a href="https://www.destroyallsoftware.com/talks/wat" rel="noopener noreferrer"&gt;https://www.destroyallsoftware.com/talks/wat&lt;/a&gt;&amp;nbsp;↩&lt;/p&gt;
&lt;/li&gt;

&lt;/ol&gt;

</description>
      <category>rag</category>
      <category>ai</category>
    </item>
  </channel>
</rss>
