<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: RecallRun</title>
    <description>The latest articles on DEV Community by RecallRun (@recallrundev).</description>
    <link>https://dev.to/recallrundev</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160628%2F626bba7e-f376-4429-beb0-6b0aa0331e52.webp</url>
      <title>DEV Community: RecallRun</title>
      <link>https://dev.to/recallrundev</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/recallrundev"/>
    <language>en</language>
    <item>
      <title>OpenAI Python SDK errors, retries and timeouts: what it retries for you and what it can't</title>
      <dc:creator>RecallRun</dc:creator>
      <pubDate>Sat, 14 Nov 2026 06:31:47 +0000</pubDate>
      <link>https://dev.to/recallrundev/openai-python-sdk-errors-retries-and-timeouts-what-it-retries-for-you-and-what-it-cant-3gf5</link>
      <guid>https://dev.to/recallrundev/openai-python-sdk-errors-retries-and-timeouts-what-it-retries-for-you-and-what-it-cant-3gf5</guid>
      <description>&lt;p&gt;Network calls fail. Rate limits get hit, servers have bad minutes, requests are malformed, keys expire.&lt;/p&gt;

&lt;p&gt;The OpenAI Python SDK already &lt;strong&gt;retries the retryable errors for you&lt;/strong&gt; (connection problems, 408, 409, 429 and 5xx) with exponential backoff. Your job is three things: set sensible &lt;strong&gt;timeouts&lt;/strong&gt;, handle the errors that retrying cannot fix, and log the &lt;strong&gt;request id&lt;/strong&gt; so a failure can be traced later.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. The exception hierarchy
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;openai.APIError
├── APIConnectionError          -&amp;gt; retried automatically
│   └── APITimeoutError         -&amp;gt; retried automatically
└── APIStatusError
    ├── 400 BadRequestError          -&amp;gt; fix the request
    ├── 401 AuthenticationError      -&amp;gt; fix the config
    ├── 403 PermissionDeniedError    -&amp;gt; fix the config
    ├── 404 NotFoundError            -&amp;gt; fix the request
    ├── 422 UnprocessableEntityError -&amp;gt; fix the request
    ├── 429 RateLimitError           -&amp;gt; retried automatically
    └── 5xx InternalServerError      -&amp;gt; retried automatically
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  2. Automatic retries, observed
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;fake_client&lt;/code&gt; below is a small stub that scripts HTTP responses, so you can watch the retry logic without spending tokens or waiting for a real outage.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;fake_openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fake_client&lt;/span&gt;

&lt;span class="c1"&gt;# a 429 then a 500, then success: the SDK retries with backoff and you never see the errors&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fake_client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;script&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Recovered."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;responses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"gpt-5.6-luna"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"hello"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"| HTTP requests made:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# more failures than retries: the error reaches your code&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fake_client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;script&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"message"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Rate limit reached for requests"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;responses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"gpt-5.6-luna"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"hello"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;RateLimitError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"| request id:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="s2"&gt;"| requests made:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# a 400 is NOT retried: sending the same bad request again would fail again&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fake_client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;script&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"message"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Invalid value for 'model'"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;responses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"gpt-5.6-luna"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"hello"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;BadRequestError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"| requests made:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Recovered. | HTTP requests made: 3
RateLimitError 429 | request id: req_fake_8 | requests made: 3
BadRequestError 400 Error code: 400 - {'error': {'message': "Invalid value for 'model'", 'type': 'fake_error', 'code': None}} | requests made: 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three requests for one call in the first case, and the 400 went out exactly once.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Timeouts
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;fake_openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fake_client&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fake_client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;script&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"delay"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;2.0&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"too late"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;responses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"gpt-5.6-luna"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"summarise this long report"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;APITimeoutError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"timed out:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# per-call override for a known-slow request (e.g. high reasoning effort)&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fake_client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;script&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"delay"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.8&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"done"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;with_options&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;2.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;responses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"gpt-5.6-luna"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"hard question"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"with a longer timeout:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;timed out: APITimeoutError
with a longer timeout: done
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The default timeout is 10 minutes.&lt;/strong&gt; That suits long reasoning calls, not a web request where a user is waiting. Set a timeout per use case (interactive: 30-60 s; background: longer), stream interactive responses so users see progress, and remember a retried timeout multiplies the wait: worst case = timeout × (retries + 1).&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  4. A handling pattern for applications
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;

&lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getLogger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"assistant"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;responses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="no"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_output_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;800&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;RateLimitError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"We're busy right now. Please try again in a minute."&lt;/span&gt;      &lt;span class="c1"&gt;# already retried by the SDK&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;APITimeoutError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"That took too long. Try a shorter question."&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;APIConnectionError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exception&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"network problem reaching OpenAI"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"Temporary connection problem."&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;BadRequestError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"bad request %s: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;          &lt;span class="c1"&gt;# a bug in OUR code&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"Sorry, something went wrong."&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;openai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;AuthenticationError&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;PermissionDeniedError&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;critical&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"OpenAI credentials rejected"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                        &lt;span class="c1"&gt;# page someone&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"incomplete"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;warning&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"incomplete response %s: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;incomplete_details&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Order matters: &lt;code&gt;APITimeoutError&lt;/code&gt; is a subclass of &lt;code&gt;APIConnectionError&lt;/code&gt;, so catch it first.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Error&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;th&gt;Action&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;401 / 403&lt;/td&gt;
&lt;td&gt;bad, revoked or under-privileged key&lt;/td&gt;
&lt;td&gt;alert; never retry in a loop&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;400 / 422&lt;/td&gt;
&lt;td&gt;your request is invalid (parameters, schema, too long)&lt;/td&gt;
&lt;td&gt;fix the code; log the request id&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;404&lt;/td&gt;
&lt;td&gt;unknown model, file or response id&lt;/td&gt;
&lt;td&gt;check names and ids (models get retired)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;429&lt;/td&gt;
&lt;td&gt;rate limit or quota exceeded&lt;/td&gt;
&lt;td&gt;retried automatically; if it persists, cut concurrency or raise limits; check billing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5xx, connection, timeout&lt;/td&gt;
&lt;td&gt;transient server or network trouble&lt;/td&gt;
&lt;td&gt;retried automatically; degrade gracefully if it persists&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Recap
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The SDK retries&lt;/strong&gt; connection errors and 408/409/429/5xx (&lt;code&gt;max_retries&lt;/code&gt;, default 2) with backoff.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Other 4xx errors&lt;/strong&gt; mean fix your request; retrying won't help.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Set timeouts per use case&lt;/strong&gt;; the default is long.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Log &lt;code&gt;request_id&lt;/code&gt;&lt;/strong&gt; and check &lt;code&gt;status == "incomplete"&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is part 12 of a free 20-part OpenAI Python SDK guide. The full series (streaming, structured outputs, function calling, evals, cost) is at &lt;a href="https://recallrun.dev/guides/openai-sdk/?utm_source=devto&amp;amp;utm_medium=crosspost&amp;amp;utm_campaign=guide-errors" rel="noopener noreferrer"&gt;recallrun.dev/guides/openai-sdk&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>python</category>
      <category>openai</category>
      <category>ai</category>
      <category>backend</category>
    </item>
    <item>
      <title>Streaming LLM answers from FastAPI with SSE: sources at 52 ms, and one time.sleep that pushed it to 3.3 s</title>
      <dc:creator>RecallRun</dc:creator>
      <pubDate>Fri, 06 Nov 2026 15:49:36 +0000</pubDate>
      <link>https://dev.to/recallrundev/streaming-llm-answers-from-fastapi-with-sse-sources-at-52-ms-and-one-timesleep-that-pushed-it-to-528n</link>
      <guid>https://dev.to/recallrundev/streaming-llm-answers-from-fastapi-with-sse-sources-at-52-ms-and-one-timesleep-that-pushed-it-to-528n</guid>
      <description>&lt;p&gt;A RAG answer takes seconds to finish. Users judge the wait by when something appears. So: what does streaming change in a FastAPI endpoint, and what breaks it?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; send the retrieved sources as the first SSE event, then the tokens. In my test the first byte arrived after 52 ms instead of 1,664 ms, and the first token after 453 ms. The total time stayed the same (1,673 ms vs 1,664 ms). The most common way to break it is a blocking call inside an &lt;code&gt;async&lt;/code&gt; generator: with 5 clients the median first byte rose to 3,280 ms.&lt;/p&gt;

&lt;p&gt;Tested on a 2 vCPU GitHub runner (CPU only), Python 3.12, fastapi 0.161.1, starlette 1.9.0, uvicorn 0.54.0, httpx 0.28.1, real uvicorn server on localhost. Retrieval (50 ms), LLM first token (400 ms) and token rate (60 tokens x 20 ms) are simulated with asyncio.sleep, so absolute numbers are not what a real model gives.&lt;/p&gt;

&lt;h2&gt;
  
  
  Setup
&lt;/h2&gt;

&lt;p&gt;The endpoint is a plain &lt;a href="https://fastapi.tiangolo.com/advanced/custom-response/" rel="noopener noreferrer"&gt;&lt;code&gt;StreamingResponse&lt;/code&gt;&lt;/a&gt; with &lt;code&gt;media_type="text/event-stream"&lt;/code&gt; and hand-formatted events. Recent FastAPI versions also ship a dedicated &lt;code&gt;EventSourceResponse&lt;/code&gt; in &lt;a href="https://fastapi.tiangolo.com/reference/sse/" rel="noopener noreferrer"&gt;&lt;code&gt;fastapi.sse&lt;/code&gt;&lt;/a&gt;; I did not test it, and the manual version works the same way on older releases.&lt;/p&gt;

&lt;p&gt;The stream has three kinds of event:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;sources&lt;/code&gt;: one event with the retrieved chunks (sent right after retrieval).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;token&lt;/code&gt;: one event per token.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;done&lt;/code&gt;: a final marker so the client knows the stream ended cleanly.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The comparison endpoint does the same work and returns one JSON body at the end. A client script (httpx) times the first byte, the &lt;code&gt;sources&lt;/code&gt; event, the first &lt;code&gt;token&lt;/code&gt; event and the end. Each case ran in a real uvicorn process.&lt;/p&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;All times in milliseconds (median over runs; last column is the worst first byte).&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Case&lt;/th&gt;
&lt;th&gt;First byte&lt;/th&gt;
&lt;th&gt;Sources&lt;/th&gt;
&lt;th&gt;First token&lt;/th&gt;
&lt;th&gt;Total&lt;/th&gt;
&lt;th&gt;Worst first byte&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Non-streaming JSON, 1 client&lt;/td&gt;
&lt;td&gt;1664&lt;/td&gt;
&lt;td&gt;-&lt;/td&gt;
&lt;td&gt;-&lt;/td&gt;
&lt;td&gt;1664&lt;/td&gt;
&lt;td&gt;1666&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SSE sources-first, 1 client&lt;/td&gt;
&lt;td&gt;52&lt;/td&gt;
&lt;td&gt;52&lt;/td&gt;
&lt;td&gt;453&lt;/td&gt;
&lt;td&gt;1673&lt;/td&gt;
&lt;td&gt;53&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SSE sources-first, 20 concurrent&lt;/td&gt;
&lt;td&gt;71&lt;/td&gt;
&lt;td&gt;71&lt;/td&gt;
&lt;td&gt;472&lt;/td&gt;
&lt;td&gt;1715&lt;/td&gt;
&lt;td&gt;86&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SSE, 5 concurrent (control)&lt;/td&gt;
&lt;td&gt;54&lt;/td&gt;
&lt;td&gt;54&lt;/td&gt;
&lt;td&gt;455&lt;/td&gt;
&lt;td&gt;1685&lt;/td&gt;
&lt;td&gt;56&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SSE + blocking &lt;code&gt;time.sleep&lt;/code&gt;, 5 concurrent&lt;/td&gt;
&lt;td&gt;3280&lt;/td&gt;
&lt;td&gt;3280&lt;/td&gt;
&lt;td&gt;3681&lt;/td&gt;
&lt;td&gt;4894&lt;/td&gt;
&lt;td&gt;6510&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SSE + &lt;code&gt;GZipMiddleware&lt;/code&gt;, 1 client&lt;/td&gt;
&lt;td&gt;52&lt;/td&gt;
&lt;td&gt;52&lt;/td&gt;
&lt;td&gt;453&lt;/td&gt;
&lt;td&gt;1673&lt;/td&gt;
&lt;td&gt;53&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What the numbers mean
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Streaming moves the first visible byte, not the finish.&lt;/strong&gt; First byte improved by 1,612 ms. The first token showed up 401 ms after the sources, which is the simulated model delay. Total time did not improve.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sources first is nearly free.&lt;/strong&gt; Retrieval is done before generation starts, so the user sees the evidence 52 ms in while the model is still thinking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;20 concurrent streams were fine&lt;/strong&gt; (71 ms median first byte), as long as the event loop is not blocked.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Blocking the loop is the real hazard.&lt;/strong&gt; The &lt;code&gt;time.sleep&lt;/code&gt; variant does the same simulated work with a blocking call. With 5 clients the median first byte was 3,280 ms, about 61 times the control (54 ms), and the worst was 6,510 ms. One client's stream stalls everyone else's. In real code the culprit is a sync SDK call, a CPU-heavy rerank or a blocking DB driver inside &lt;code&gt;async def&lt;/code&gt;. Use the async client, or run sync work in a thread (&lt;code&gt;run_in_threadpool&lt;/code&gt;, &lt;code&gt;asyncio.to_thread&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GZip did not buffer the stream here.&lt;/strong&gt; Starlette's &lt;code&gt;GZipMiddleware&lt;/code&gt; leaves &lt;code&gt;text/event-stream&lt;/code&gt; alone, and all 62 events arrived as separate chunks, with first byte unchanged at 52 ms. Other middleware, or a proxy, may behave differently.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Caveats
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The model and retrieval are fakes. A real provider adds network jitter and a longer, variable first-token time; the shape of the gain (first byte early, total unchanged) should hold, the numbers will not.&lt;/li&gt;
&lt;li&gt;Localhost has no proxy. nginx buffers responses by default; the &lt;code&gt;X-Accel-Buffering: no&lt;/code&gt; header below asks it not to, but I did not test it.&lt;/li&gt;
&lt;li&gt;I did not test client disconnects. In production, stop the upstream LLM call when the client leaves, or you pay for tokens nobody reads.&lt;/li&gt;
&lt;li&gt;The blocking case is deliberately extreme (every token blocks). Shorter blocks hurt less but still add to every stream's latency.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Copyable code
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;fastapi&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;FastAPI&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;fastapi.responses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;StreamingResponse&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;sse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;data: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="nd"&gt;@app.get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/ask&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;events&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="n"&gt;chunks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;retrieve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;             &lt;span class="c1"&gt;# async vector search
&lt;/span&gt;        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="nf"&gt;sse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sources&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;           &lt;span class="c1"&gt;# evidence first
&lt;/span&gt;        &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;llm_stream&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;  &lt;span class="c1"&gt;# async client, never blocking
&lt;/span&gt;            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="nf"&gt;sse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;token&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="nf"&gt;sse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;done&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;StreamingResponse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nf"&gt;events&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
        &lt;span class="n"&gt;media_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text/event-stream&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Cache-Control&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no-cache&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;X-Accel-Buffering&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;retrieve&lt;/code&gt; and &lt;code&gt;llm_stream&lt;/code&gt; are your own functions. The full test server and benchmark script are on the &lt;a href="https://recallrun.dev/posts/fastapi-sse-streaming-llm/" rel="noopener noreferrer"&gt;original post&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule of thumb
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Send &lt;code&gt;sources&lt;/code&gt; as the first event, then tokens, then an explicit &lt;code&gt;done&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Everything inside the generator must be &lt;code&gt;await&lt;/code&gt;-able or pushed to a thread.&lt;/li&gt;
&lt;li&gt;Measure first byte under 5-20 concurrent clients, not just one, and put that number in CI.&lt;/li&gt;
&lt;li&gt;Test through your real proxy before shipping; localhost results do not cover buffering.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you want to check the sources the model cites afterwards, I wrote up &lt;a href="https://recallrun.dev/posts/validate-llm-citations-fastapi/" rel="noopener noreferrer"&gt;validating LLM citations in FastAPI&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>fastapi</category>
      <category>python</category>
      <category>llm</category>
      <category>rag</category>
    </item>
    <item>
      <title>RAG evals that fail CI: a 30-question golden set caught 3 of 4 retrieval regressions</title>
      <dc:creator>RecallRun</dc:creator>
      <pubDate>Wed, 04 Nov 2026 07:28:48 +0000</pubDate>
      <link>https://dev.to/recallrundev/rag-evals-that-fail-ci-a-30-question-golden-set-caught-3-of-4-retrieval-regressions-1ml</link>
      <guid>https://dev.to/recallrundev/rag-evals-that-fail-ci-a-30-question-golden-set-caught-3-of-4-retrieval-regressions-1ml</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://recallrun.dev/posts/rag-evaluation-golden-set-ci/?utm_source=devto&amp;amp;utm_campaign=golden-ci" rel="noopener noreferrer"&gt;RecallRun&lt;/a&gt;, where the full scripts and raw output are attached.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;How do you stop a retrieval change from silently making your RAG worse? Put a small golden set in CI and fail the build when metrics drop below a committed baseline.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; four files. On 30 questions it flagged 3 of 4 accidental regressions I injected. The one it passed was a real quality change that stayed inside the tolerance I set (hit@5 fell 3.3 points). One regression (chunks of 200 characters) &lt;em&gt;raised&lt;/em&gt; hit@1 and MRR while cutting hit@5 by 13.3 points, so gating on a single metric would have missed it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Setup
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Retriever under test:&lt;/strong&gt; heading-aware chunks (800 characters) with the heading path prepended, ranked by BM25. Same chunker as in &lt;a href="https://dev.to/recallrundev/rag-chunking-measured-heading-aware-chunks-doubled-hit1-on-the-fastapi-docs-5hh7"&gt;my chunking benchmark&lt;/a&gt;, and the 30 questions are the paraphrase set from &lt;a href="https://dev.to/recallrundev/hybrid-search-in-python-bm25-vectors-with-reciprocal-rank-fusion-alpha-swept-0-to-1-4oj7"&gt;the hybrid search post&lt;/a&gt;. BM25 only, so CI needs no model download and runs in about 0.2 s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Corpus:&lt;/strong&gt; FastAPI English docs (151 Markdown files). Python 3.12, pytest 9.1, stdlib only.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Golden set:&lt;/strong&gt; &lt;code&gt;golden.jsonl&lt;/code&gt;, one line per question: &lt;code&gt;{"question": ..., "expected_doc": ...}&lt;/code&gt;. A hit means the expected doc appears in the top-k documents.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Metrics:&lt;/strong&gt; hit@1, hit@5, MRR (cut off at rank 10).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gate:&lt;/strong&gt; &lt;code&gt;pytest&lt;/code&gt; fails if any metric is more than 0.05 below &lt;code&gt;baseline.json&lt;/code&gt;. With 30 questions, one question is 0.033, so one flipped answer passes and two fail.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Regressions injected:&lt;/strong&gt; each is a one-field config change: smaller chunks, larger chunks, no heading prefix, and a tokenizer bug (lowercasing removed).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I did not test a citation-rate metric here. The harness only scores retrieval.&lt;/p&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;Differences are in percentage points versus baseline, computed from the unrounded values.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;th&gt;hit@1&lt;/th&gt;
&lt;th&gt;hit@5&lt;/th&gt;
&lt;th&gt;MRR&lt;/th&gt;
&lt;th&gt;Gate&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Baseline&lt;/td&gt;
&lt;td&gt;0.367&lt;/td&gt;
&lt;td&gt;0.933&lt;/td&gt;
&lt;td&gt;0.583&lt;/td&gt;
&lt;td&gt;pass&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;chunk size 800 to 200&lt;/td&gt;
&lt;td&gt;0.500 (+13.3)&lt;/td&gt;
&lt;td&gt;0.800 (-13.3)&lt;/td&gt;
&lt;td&gt;0.628 (+4.5)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;fail&lt;/strong&gt; (hit@5)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;chunk size 800 to 3000&lt;/td&gt;
&lt;td&gt;0.367 (0.0)&lt;/td&gt;
&lt;td&gt;0.900 (-3.3)&lt;/td&gt;
&lt;td&gt;0.601 (+1.8)&lt;/td&gt;
&lt;td&gt;pass&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;heading prefix off&lt;/td&gt;
&lt;td&gt;0.300 (-6.7)&lt;/td&gt;
&lt;td&gt;0.867 (-6.7)&lt;/td&gt;
&lt;td&gt;0.513 (-7.0)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;fail&lt;/strong&gt; (all 3)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;lowercase off&lt;/td&gt;
&lt;td&gt;0.400 (+3.3)&lt;/td&gt;
&lt;td&gt;0.733 (-20.0)&lt;/td&gt;
&lt;td&gt;0.553 (-3.0)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;fail&lt;/strong&gt; (hit@5)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What the numbers mean
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The 200-character case is why you gate on several metrics.&lt;/strong&gt; Small chunks make the best chunk more specific, so the right doc more often lands at rank 1 (+13.3 points). But 6 questions dropped out of the top 5 entirely. If your gate were "MRR must not fall", this change would pass and ship. My reading of the cause: short chunks split the answer away from the words the question uses, so some docs have no chunk that scores at all. I did not test that explanation directly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The tokenizer bug moved MRR only 3.0 points&lt;/strong&gt; yet lost 20.0 points of hit@5, which is 6 more questions failing. An average hid it; the per-metric threshold did not.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Removing the heading prefix hit every metric&lt;/strong&gt; by roughly 7 points, so it failed all three tests at once.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3000-character chunks passed the gate&lt;/strong&gt; with a 3.3-point hit@5 drop. That is one question. It sits inside the tolerance on purpose, and it also shows the cost: a 30-question set cannot separate a real 3-point drop from noise.&lt;/p&gt;

&lt;h2&gt;
  
  
  Caveats
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;30 questions is a smoke test. Every question is worth 3.3 points, so differences under about 5 points are not reliable. Grow the set from real user queries and from every bug report.&lt;/li&gt;
&lt;li&gt;The questions and corpus are the ones I used to tune earlier posts, so the baseline is optimistic.&lt;/li&gt;
&lt;li&gt;The expected answer is a document, not a passage. Coarse, but stable while you re-chunk.&lt;/li&gt;
&lt;li&gt;BM25 only. A dense or hybrid retriever adds nondeterminism (model versions, ANN settings), so pin those and expect to widen the tolerance.&lt;/li&gt;
&lt;li&gt;The injected regressions are mine; real ones are less tidy.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;The retriever config is a frozen dataclass, so a regression is a one-field diff. The harness and the gate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# harness.py
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="n"&gt;HERE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dirname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__file__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;load_golden&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;HERE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;golden.jsonl&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;retriever&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;golden&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;golden&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;ranked&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;retriever&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;question&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;rank&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ranked&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;index&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected_doc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected_doc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ranked&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
        &lt;span class="n"&gt;h1&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;rank&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;h5&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;rank&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;rank&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;rr&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;rank&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;rank&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;question&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;question&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected_doc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rank&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rank&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;golden&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hit@1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;h1&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hit@5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;h5&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mrr&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rr&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# test_golden.py
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Fail the build if any retrieval metric drops below the committed baseline.
Update baseline.json deliberately (in the same PR, reviewed) when you improve the pipeline.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pytest&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;rag&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Retriever&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Config&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;harness&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;load_golden&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;evaluate&lt;/span&gt;

&lt;span class="n"&gt;HERE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dirname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__file__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;TOLERANCE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.05&lt;/span&gt;   &lt;span class="c1"&gt;# 30 questions = 0.033 each: one flipped answer passes, two fail
&lt;/span&gt;&lt;span class="n"&gt;CONFIG&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;RAG_CONFIG_OVERRIDE&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;

&lt;span class="nd"&gt;@pytest.fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scope&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;module&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;result&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Retriever&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;CONFIG&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;load_golden&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="nd"&gt;@pytest.mark.parametrize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;metric&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hit@1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hit@5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mrr&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;test_metric_not_regressed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;
    &lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;HERE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;baseline.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)))[&lt;/span&gt;&lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;misses&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;question&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rank&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rank&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;TOLERANCE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &amp;lt; baseline &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; - &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;TOLERANCE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;misses&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; questions missed top-5, e.g. &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;misses&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;baseline.json&lt;/code&gt; is committed and holds &lt;code&gt;{"hit@1": 0.3667, "hit@5": 0.9333, "mrr": 0.5828}&lt;/code&gt;. The workflow:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# .github/workflows/rag-eval.yml&lt;/span&gt;
&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rag-eval&lt;/span&gt;
&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;pull_request&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;paths&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rag/**"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;golden.jsonl"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;baseline.json"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;prompts/**"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;eval&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v4&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/setup-python@v5&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{&lt;/span&gt; &lt;span class="nv"&gt;python-version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;3.12"&lt;/span&gt; &lt;span class="pi"&gt;}&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pip install pytest&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;bash fetch_corpus.sh&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pytest -q test_golden.py&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Rule of thumb
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Commit a golden set and a baseline; fail the build on any single metric dropping, not on an average.&lt;/li&gt;
&lt;li&gt;Set tolerance from your set size: at least two questions' worth, so one flip is not a red build.&lt;/li&gt;
&lt;li&gt;Print the missed questions in the failure message so the fix starts from a list.&lt;/li&gt;
&lt;li&gt;Raise the baseline only in the same reviewed PR that improves the pipeline.&lt;/li&gt;
&lt;li&gt;Trigger the job on changes to chunking, embeddings, prompts and the golden set itself.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The full &lt;code&gt;rag.py&lt;/code&gt;, &lt;code&gt;golden.jsonl&lt;/code&gt;, the demo runner and raw &lt;code&gt;results.txt&lt;/code&gt; are attached to &lt;a href="https://recallrun.dev/posts/rag-evaluation-golden-set-ci/?utm_source=devto&amp;amp;utm_campaign=golden-ci" rel="noopener noreferrer"&gt;the original post&lt;/a&gt;. If you gate RAG changes differently, I'd like to hear what metric you trust.&lt;/p&gt;

</description>
      <category>rag</category>
      <category>python</category>
      <category>testing</category>
      <category>ai</category>
    </item>
    <item>
      <title>Hybrid search in Python: BM25 + vectors with Reciprocal Rank Fusion, alpha swept 0 to 1</title>
      <dc:creator>RecallRun</dc:creator>
      <pubDate>Sat, 31 Oct 2026 06:29:21 +0000</pubDate>
      <link>https://dev.to/recallrundev/hybrid-search-in-python-bm25-vectors-with-reciprocal-rank-fusion-alpha-swept-0-to-1-4oj7</link>
      <guid>https://dev.to/recallrundev/hybrid-search-in-python-bm25-vectors-with-reciprocal-rank-fusion-alpha-swept-0-to-1-4oj7</guid>
      <description>&lt;p&gt;Should you fuse BM25 with vector search, and what weight should each side get? I ran both retrievers separately and fused them with Reciprocal Rank Fusion (RRF) across three different query styles, sweeping the fusion weight from pure vector to pure BM25.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; no single retriever wins everywhere. Pure BM25 beat pure vectors by 49 points of MRR on exact code identifiers (0.772 vs 0.279). Pure vectors beat pure BM25 by 24 points on hand-written paraphrase questions (0.768 vs 0.566). An RRF blend with 80% weight on BM25 rank was the only setting that stayed within 9 points of the best score on &lt;em&gt;every&lt;/em&gt; query type.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Corpus:&lt;/strong&gt; the English FastAPI docs, 151 Markdown files, chunked with the heading-aware-plus-heading-path chunker from &lt;a href="https://recallrun.dev/posts/rag-chunking-strategies-measured/?utm_source=devto&amp;amp;utm_campaign=hybrid" rel="noopener noreferrer"&gt;my chunking benchmark&lt;/a&gt;, the strategy that scored best there. 2,713 chunks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BM25:&lt;/strong&gt; the same from-scratch implementation as the chunking post (k1=1.5, b=0.75), reused unchanged.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Vectors:&lt;/strong&gt; &lt;code&gt;sentence-transformers/all-MiniLM-L6-v2&lt;/code&gt;, 384 dimensions, cosine similarity via normalized dot product. A small, CPU-friendly model (&lt;a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2" rel="noopener noreferrer"&gt;model card&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fusion:&lt;/strong&gt; &lt;code&gt;score(d) = alpha / (60 + rank_bm25(d)) + (1 - alpha) / (60 + rank_vec(d))&lt;/code&gt;. Standard RRF with &lt;code&gt;k=60&lt;/code&gt;, the constant from the &lt;a href="https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf" rel="noopener noreferrer"&gt;original Cormack, Clarke and Buettcher paper&lt;/a&gt;, with a weight &lt;code&gt;alpha&lt;/code&gt; instead of a flat 50/50 sum. &lt;code&gt;alpha=1.0&lt;/code&gt; is pure BM25, &lt;code&gt;alpha=0.0&lt;/code&gt; is pure vector search. Each retriever contributes its own top 50 candidates; a document missing from one side gets rank 200 for that side's term.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Three query sets, on purpose:&lt;/strong&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Heading queries (773):&lt;/strong&gt; section headings used verbatim. They overlap with the indexed heading-path prefix by construction, so this is an easy benchmark, kept for continuity, not as the main result.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Identifier queries (305):&lt;/strong&gt; inline-code spans like &lt;code&gt;response_model&lt;/code&gt; or &lt;code&gt;BackgroundTasks&lt;/code&gt; that appear in exactly one chunk. This tests exact-token recall, the case hybrid search is usually sold on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paraphrase queries (30):&lt;/strong&gt; questions I wrote by hand, like "How do I run code after sending the response to the client?" for the background-tasks page, avoiding the target page's own vocabulary. A hit counts any chunk from the target file. Small sample, so treat the percentages as approximate.&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;MRR by alpha (0.0 = pure vector, 1.0 = pure BM25):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;alpha&lt;/th&gt;
&lt;th&gt;heading MRR&lt;/th&gt;
&lt;th&gt;identifier MRR&lt;/th&gt;
&lt;th&gt;paraphrase MRR&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0.0&lt;/td&gt;
&lt;td&gt;0.491&lt;/td&gt;
&lt;td&gt;0.279&lt;/td&gt;
&lt;td&gt;0.768&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.2&lt;/td&gt;
&lt;td&gt;0.581&lt;/td&gt;
&lt;td&gt;0.471&lt;/td&gt;
&lt;td&gt;0.794&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.4&lt;/td&gt;
&lt;td&gt;0.642&lt;/td&gt;
&lt;td&gt;0.569&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.806&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.5&lt;/td&gt;
&lt;td&gt;0.679&lt;/td&gt;
&lt;td&gt;0.696&lt;/td&gt;
&lt;td&gt;0.751&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.6&lt;/td&gt;
&lt;td&gt;0.710&lt;/td&gt;
&lt;td&gt;0.735&lt;/td&gt;
&lt;td&gt;0.731&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;0.8&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.782&lt;/td&gt;
&lt;td&gt;0.753&lt;/td&gt;
&lt;td&gt;0.722&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.9&lt;/td&gt;
&lt;td&gt;0.813&lt;/td&gt;
&lt;td&gt;0.768&lt;/td&gt;
&lt;td&gt;0.678&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.0&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.859&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.772&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.566&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;hit@1 / hit@5 for the identifier and paraphrase sets:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;alpha&lt;/th&gt;
&lt;th&gt;identifier hit@1&lt;/th&gt;
&lt;th&gt;identifier hit@5&lt;/th&gt;
&lt;th&gt;paraphrase hit@1&lt;/th&gt;
&lt;th&gt;paraphrase hit@5&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0.0&lt;/td&gt;
&lt;td&gt;0.207&lt;/td&gt;
&lt;td&gt;0.397&lt;/td&gt;
&lt;td&gt;0.667&lt;/td&gt;
&lt;td&gt;0.933&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.4&lt;/td&gt;
&lt;td&gt;0.515&lt;/td&gt;
&lt;td&gt;0.656&lt;/td&gt;
&lt;td&gt;0.667&lt;/td&gt;
&lt;td&gt;0.967&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.8&lt;/td&gt;
&lt;td&gt;0.662&lt;/td&gt;
&lt;td&gt;0.889&lt;/td&gt;
&lt;td&gt;0.533&lt;/td&gt;
&lt;td&gt;0.933&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.0&lt;/td&gt;
&lt;td&gt;0.692&lt;/td&gt;
&lt;td&gt;0.889&lt;/td&gt;
&lt;td&gt;0.367&lt;/td&gt;
&lt;td&gt;0.867&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What the numbers mean
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;BM25 wins big on identifiers, vectors win big on paraphrases, by more than I guessed.&lt;/strong&gt; MiniLM's hit@1 on identifier queries was 0.207. A 384-dim sentence model wasn't trained to preserve rare exact tokens like &lt;code&gt;BackgroundTasks&lt;/code&gt;; it embeds the gist of the sentence around it, and one API reference page looks a lot like another. BM25's IDF weighting does the opposite: a rare token dominates the score.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pure BM25 also won the heading benchmark, and that number is misleading.&lt;/strong&gt; The chunker prepends each chunk's heading path to the indexed text, so querying with the heading is close to a substring search, BM25's best case. Real users don't type section titles. The 30 paraphrase questions are the fairer read on "does semantic search help", and there pure vectors beat pure BM25 by 24 MRR points.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;No fixed alpha is best for all three.&lt;/strong&gt; alpha=0.4 is best for paraphrases, alpha=1.0 for identifiers and headings. Taking the worst-case regret across the three query types, the minimum is at &lt;strong&gt;alpha=0.8&lt;/strong&gt;: 1.9 points below the best identifier score, 8.4 below the best paraphrase score, 7.7 below the best heading score. alpha=0.5, the "obvious" 50/50 choice, loses 18 points on headings.&lt;/p&gt;

&lt;h2&gt;
  
  
  Honest caveats
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The paraphrase set is small and I wrote it.&lt;/strong&gt; 30 questions is enough to see a 24-point gap, not enough to trust the third decimal place.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The heading benchmark favors whatever puts the query text into the index&lt;/strong&gt;, BM25's easiest case.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One small local embedding model.&lt;/strong&gt; Larger or API-hosted embedding models could close some of the identifier gap. I didn't test that.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;RRF's k=60 wasn't swept here.&lt;/strong&gt; Only alpha was. (I did sweep k in &lt;a href="https://recallrun.dev/posts/rrf-k-parameter-explained/?utm_source=devto&amp;amp;utm_campaign=hybrid" rel="noopener noreferrer"&gt;a follow-up&lt;/a&gt;.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Single corpus.&lt;/strong&gt; FastAPI's docs are unusually well structured. A messier corpus would likely show a smaller BM25 edge on identifiers.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The fusion code
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;K_RRF&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rrf_fuse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bm25_ranked_ids&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vec_ranked_ids&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;bm25_ranked_ids, vec_ranked_ids: lists of doc ids, best first.
    alpha=1.0 is pure BM25, alpha=0.0 is pure vector.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;rank_bm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;doc_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;doc_id&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bm25_ranked_ids&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
    &lt;span class="n"&gt;rank_vec&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;doc_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;doc_id&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vec_ranked_ids&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
    &lt;span class="n"&gt;fallback&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bm25_ranked_ids&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vec_ranked_ids&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;
    &lt;span class="n"&gt;candidates&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rank_bm&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rank_vec&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;scored&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;alpha&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;K_RRF&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;rank_bm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;K_RRF&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;rank_vec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;candidates&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;scored&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;scored&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run BM25 and your vector index independently, each returning its own top-N (50 was enough here), then fuse. No score normalization needed: that's the point of using ranks instead of raw scores, which differ in scale between BM25 and cosine similarity.&lt;/p&gt;

&lt;p&gt;Latency on 2 vCPUs over 2,713 chunks: BM25 1.82 ms median, vector encode+search 23.99 ms. A hybrid query costs roughly what the vector half costs.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd do in your pipeline
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Don't default to alpha=0.5.&lt;/strong&gt; On this corpus it lost more than any other tested value on at least one query type. Start from 0.7-0.8 (BM25-leaning) and tune on a real query log.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If your users type API names, error codes or config keys, weight BM25 heavily.&lt;/strong&gt; Small embedding models are bad at exact tokens; that's structural, not a tuning problem.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If your users type full questions, don't skip vectors.&lt;/strong&gt; BM25 alone lost 24 MRR points on plain paraphrases.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure your own alpha&lt;/strong&gt; with a &lt;a href="https://recallrun.dev/posts/rag-evaluation-golden-set-ci/?utm_source=devto&amp;amp;utm_campaign=hybrid" rel="noopener noreferrer"&gt;golden-set eval in CI&lt;/a&gt; built from real traffic instead of a guess.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The full script (&lt;code&gt;bench_hybrid.py&lt;/code&gt;), raw results and corpus fetch script are on the &lt;a href="https://recallrun.dev/posts/hybrid-search-bm25-rrf-python/?utm_source=devto&amp;amp;utm_campaign=hybrid" rel="noopener noreferrer"&gt;original post&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I run experiments like this every week at RecallRun. If you want to build RAG systems end to end, the AI Engineering course starts free: &lt;a href="https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=hybrid" rel="noopener noreferrer"&gt;https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=hybrid&lt;/a&gt;&lt;/p&gt;

</description>
      <category>rag</category>
      <category>python</category>
      <category>search</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>RAG chunking, measured: heading-aware chunks doubled hit@1 on the FastAPI docs</title>
      <dc:creator>RecallRun</dc:creator>
      <pubDate>Sat, 31 Oct 2026 06:27:58 +0000</pubDate>
      <link>https://dev.to/recallrundev/rag-chunking-measured-heading-aware-chunks-doubled-hit1-on-the-fastapi-docs-5hh7</link>
      <guid>https://dev.to/recallrundev/rag-chunking-measured-heading-aware-chunks-doubled-hit1-on-the-fastapi-docs-5hh7</guid>
      <description>&lt;p&gt;Chunking is the first decision in a RAG pipeline and the one people tune least. Most projects take the default of their framework, often fixed-size chunks with overlap, and move on. I wanted numbers, so I ran five strategies over a real documentation set and measured how often retrieval finds the right section.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; splitting at headings first, then packing paragraphs, raised hit@1 from 0.39 to 0.71. Adding the heading path to each chunk raised it again to 0.79.&lt;/p&gt;

&lt;h2&gt;
  
  
  The test
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Corpus:&lt;/strong&gt; the English FastAPI docs, 151 Markdown files with 1.45 million characters. Real technical writing, with headings, code blocks and admonitions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Queries:&lt;/strong&gt; every h2/h3 heading with at least 300 characters of body and 2+ words. That gave &lt;strong&gt;773 queries&lt;/strong&gt;, such as "Declare request example data" or "Dependencies with yield and except".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Correct answer:&lt;/strong&gt; a retrieved chunk counts as a hit when at least half of its text lies inside the section that heading introduces.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retriever:&lt;/strong&gt; BM25 (k1=1.5, b=0.75). It is deterministic and needs no API key, so anyone can reproduce the numbers exactly.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Strategies
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Fixed 800/100:&lt;/strong&gt; 800-character windows with 100 characters of overlap. The classic default.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fixed 530/100:&lt;/strong&gt; the same, but sized to match the heading-aware chunks, as a control.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paragraph-packed 800:&lt;/strong&gt; greedily pack blank-line-separated paragraphs up to 800 characters. This is roughly what "recursive" splitters do.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Heading-aware:&lt;/strong&gt; split at every heading first, so a chunk never contains two sections, then paragraph-pack within the section.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Heading-aware + heading path:&lt;/strong&gt; the same chunks, with &lt;code&gt;Page title &amp;gt; Section &amp;gt; Subsection&lt;/code&gt; prepended to the text that gets indexed.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Strategy&lt;/th&gt;
&lt;th&gt;Chunks&lt;/th&gt;
&lt;th&gt;Avg chars&lt;/th&gt;
&lt;th&gt;Crosses a section&lt;/th&gt;
&lt;th&gt;Splits a code block&lt;/th&gt;
&lt;th&gt;hit@1&lt;/th&gt;
&lt;th&gt;hit@5&lt;/th&gt;
&lt;th&gt;MRR&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fixed 800/100&lt;/td&gt;
&lt;td&gt;2,142&lt;/td&gt;
&lt;td&gt;770&lt;/td&gt;
&lt;td&gt;58.8%&lt;/td&gt;
&lt;td&gt;7.3%&lt;/td&gt;
&lt;td&gt;0.389&lt;/td&gt;
&lt;td&gt;0.611&lt;/td&gt;
&lt;td&gt;0.477&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fixed 530/100 (size-matched)&lt;/td&gt;
&lt;td&gt;3,446&lt;/td&gt;
&lt;td&gt;516&lt;/td&gt;
&lt;td&gt;47.5%&lt;/td&gt;
&lt;td&gt;6.1%&lt;/td&gt;
&lt;td&gt;0.481&lt;/td&gt;
&lt;td&gt;0.766&lt;/td&gt;
&lt;td&gt;0.598&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Paragraph-packed 800&lt;/td&gt;
&lt;td&gt;1,737&lt;/td&gt;
&lt;td&gt;835&lt;/td&gt;
&lt;td&gt;57.2%&lt;/td&gt;
&lt;td&gt;3.6%&lt;/td&gt;
&lt;td&gt;0.375&lt;/td&gt;
&lt;td&gt;0.576&lt;/td&gt;
&lt;td&gt;0.455&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Heading-aware&lt;/td&gt;
&lt;td&gt;2,713&lt;/td&gt;
&lt;td&gt;534&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.0%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2.1%&lt;/td&gt;
&lt;td&gt;0.712&lt;/td&gt;
&lt;td&gt;0.933&lt;/td&gt;
&lt;td&gt;0.804&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Heading-aware + heading path&lt;/td&gt;
&lt;td&gt;2,713&lt;/td&gt;
&lt;td&gt;534&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.0%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2.1%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.785&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.966&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.859&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What the numbers say
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;More than half of fixed-size chunks straddle two sections.&lt;/strong&gt; A chunk that is half "Install" and half "Configuration" is a mediocre match for both. That shows up directly as lower retrieval scores.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Smaller chunks help, but not as much as structure.&lt;/strong&gt; The size-matched control improved on the 800-character default (hit@5 0.61 → 0.77), which confirms part of the gain is simply size. But heading-aware chunks of the &lt;em&gt;same&lt;/em&gt; average size scored 0.93. The remaining gap comes from respecting section boundaries.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Paragraph packing alone didn't help.&lt;/strong&gt; It halves the number of broken code blocks, which is nice for answer quality, but its chunks cross sections just as often as fixed windows do.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The heading path is cheap recall.&lt;/strong&gt; A chunk from the middle of a long section often doesn't repeat the words of its title. Prepending &lt;code&gt;Tutorial &amp;gt; Security &amp;gt; OAuth2 scopes&lt;/code&gt; gives every chunk that context. It cost nothing and added 7 points of hit@1.&lt;/p&gt;

&lt;h2&gt;
  
  
  Honest caveats
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Heading-as-query favors heading-aware chunking.&lt;/strong&gt; Real user questions are phrased differently from section titles. The direction of the result should hold, but expect smaller gaps with real queries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The hit rule rewards staying inside one section.&lt;/strong&gt; That is the point I'm testing, but a chunk that crosses sections and still contains the answer counts as a miss here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BM25, not embeddings.&lt;/strong&gt; Dense retrievers are more forgiving of wording. Section-crossing chunks still dilute an embedding across two topics, so I expect the same ranking, but run it on your own stack.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Markdown makes this easy.&lt;/strong&gt; For PDFs you first need to recover headings (font size, bold, numbering). That extraction step is worth the effort.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A heading-aware chunker in 30 lines
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="n"&gt;HEADING&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;^(#{1,3}) (.+?)\s*(\{.*\})?$&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;M&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;SIZE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;800&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Yield (start, end, &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;Page &amp;gt; Section &amp;gt; Sub&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;) for each heading.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;HEADING&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;finditer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;stack&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;ms&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ms&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;level&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;stack&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;stack&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;level&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
        &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &amp;gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Greedily pack blank-line paragraphs up to SIZE characters.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;finditer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;\n\s*\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;SIZE&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;
        &lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:])&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;]):&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;embed_text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;section&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Store &lt;code&gt;text&lt;/code&gt; for display and citations. Embed and index &lt;code&gt;embed_text&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd do in your pipeline
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Never let a chunk cross a heading.&lt;/strong&gt; It is the single biggest win in this test.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prepend the heading path to what you embed and index&lt;/strong&gt;, not to what you show the user.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep chunks around 500 characters&lt;/strong&gt; for technical docs, then tune with your own eval set.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure it.&lt;/strong&gt; Generate queries from your own headings like I did here, and you have a free retrieval regression test in under an hour.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The full benchmark script (BM25 from scratch, all five strategies, metrics, about 90 lines of standard-library Python) is on the &lt;a href="https://recallrun.dev/posts/rag-chunking-strategies-measured/?utm_source=devto&amp;amp;utm_campaign=chunking" rel="noopener noreferrer"&gt;original post&lt;/a&gt;. The follow-up, which fuses these chunks' BM25 ranks with vectors, is &lt;a href="https://recallrun.dev/posts/hybrid-search-bm25-rrf-python/?utm_source=devto&amp;amp;utm_campaign=chunking" rel="noopener noreferrer"&gt;hybrid search with RRF, alpha swept 0 to 1&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I run experiments like this every week at RecallRun. If you want to build RAG systems end to end, the AI Engineering course starts free: &lt;a href="https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=chunking" rel="noopener noreferrer"&gt;https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=chunking&lt;/a&gt;&lt;/p&gt;

</description>
      <category>rag</category>
      <category>python</category>
      <category>llm</category>
      <category>performance</category>
    </item>
    <item>
      <title>Make your RAG say 'I don't know': a score threshold, measured</title>
      <dc:creator>RecallRun</dc:creator>
      <pubDate>Sun, 25 Oct 2026 11:06:23 +0000</pubDate>
      <link>https://dev.to/recallrundev/make-your-rag-say-i-dont-know-a-score-threshold-measured-1k6c</link>
      <guid>https://dev.to/recallrundev/make-your-rag-say-i-dont-know-a-score-threshold-measured-1k6c</guid>
      <description>&lt;p&gt;A RAG system always gets something back from the index. Ask it about cooking and it returns the five nearest chunks of API docs, and the LLM then tries to be helpful. So: can a simple score threshold on retrieval tell "the docs cover this" from "they don't"?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; yes, mostly. On the FastAPI docs, the top-1 cosine score separated the two groups well (AUC 0.986), and a threshold of 0.40 refused 24 of 30 unanswerable questions while answering 29 of 30 answerable ones. The leftovers are near-domain questions, which no threshold fully fixes. Raw BM25 scores did worse, and BM25 almost never returns an empty result.&lt;/p&gt;

&lt;h2&gt;
  
  
  Setup
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Corpus:&lt;/strong&gt; 152 files of the FastAPI docs, split into 2,808 heading-aware chunks with the heading path prefixed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrievers:&lt;/strong&gt; MiniLM-L6-v2 cosine similarity, and BM25 over the same chunks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;30 answerable queries:&lt;/strong&gt; hand-written questions, each matching one docs page (for example "How do I set a cookie on the response?").&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;30 unanswerable queries:&lt;/strong&gt; 15 near-domain (pagination, rate limiting, gRPC, Terraform, Prometheus, Selenium...) and 15 unrelated (egg boiling, football, tides...). I checked by keyword search that the near-domain topics are not documented pages, though a few terms are mentioned in passing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decision score:&lt;/strong&gt; the top-1 chunk's score. Refuse when it is below the threshold.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;Top-1 cosine score by group:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Group&lt;/th&gt;
&lt;th&gt;Median&lt;/th&gt;
&lt;th&gt;Min&lt;/th&gt;
&lt;th&gt;Max&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Answerable (30)&lt;/td&gt;
&lt;td&gt;0.619&lt;/td&gt;
&lt;td&gt;0.393&lt;/td&gt;
&lt;td&gt;0.818&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unanswerable, near-domain (15)&lt;/td&gt;
&lt;td&gt;0.385&lt;/td&gt;
&lt;td&gt;0.256&lt;/td&gt;
&lt;td&gt;0.508&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unanswerable, unrelated (15)&lt;/td&gt;
&lt;td&gt;0.218&lt;/td&gt;
&lt;td&gt;0.175&lt;/td&gt;
&lt;td&gt;0.298&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;How well each score separates answerable from unanswerable (AUC, 1.0 is perfect, 0.5 is chance):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;AUC&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Cosine top-1&lt;/td&gt;
&lt;td&gt;0.986&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BM25 top-1, stopwords removed&lt;/td&gt;
&lt;td&gt;0.927&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BM25 top-1, raw&lt;/td&gt;
&lt;td&gt;0.910&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cosine gap between top-1 and top-2&lt;/td&gt;
&lt;td&gt;0.649&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Cosine threshold sweep:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Threshold&lt;/th&gt;
&lt;th&gt;Answerable kept&lt;/th&gt;
&lt;th&gt;Near-domain refused&lt;/th&gt;
&lt;th&gt;Unrelated refused&lt;/th&gt;
&lt;th&gt;All unanswerable refused&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0.30&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;13%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;57%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.35&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;20%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;60%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.40&lt;/td&gt;
&lt;td&gt;97% (29/30)&lt;/td&gt;
&lt;td&gt;60%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;80% (24/30)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.45&lt;/td&gt;
&lt;td&gt;93% (28/30)&lt;/td&gt;
&lt;td&gt;73%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;87% (26/30)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.50&lt;/td&gt;
&lt;td&gt;90% (27/30)&lt;/td&gt;
&lt;td&gt;93%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;97% (29/30)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.55&lt;/td&gt;
&lt;td&gt;73%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;I also split each group in half, picked the threshold on one half and scored the other. Requiring every answerable calibration query to be kept gave a threshold of 0.393. On the held-out half it kept 15 of 15 answerable queries and refused 11 of 14 unanswerable ones (79%).&lt;/p&gt;

&lt;h2&gt;
  
  
  What the numbers mean
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Unrelated questions are easy.&lt;/strong&gt; Their best score (0.298) is below the worst answerable score (0.393). Any threshold between 0.30 and 0.39 catches all of them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Near-domain questions are the real problem.&lt;/strong&gt; Their scores run up to 0.508, which overlaps the lowest answerable queries. The highest-scoring one was "How do I expose Prometheus metrics from my service?" at 0.508, matched to the OpenTelemetry page. Arguably that page is a partial answer, so this one is a labeling grey zone as much as a miss.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The cost of tightening is visible.&lt;/strong&gt; Going from 0.40 to 0.50 refuses 5 more unanswerable queries but also loses 2 more answerable ones.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BM25 is a weaker gate.&lt;/strong&gt; Its raw scores depend on query length and term rarity, so one cutoff does not transfer. It also never returned an empty candidate list for any of the 60 queries, because words like "how" and "do" match something. With stopwords removed, only 5 of the 30 unanswerable queries came back empty. "Check for empty results" alone is not a refusal strategy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The top-1 vs top-2 gap is nearly useless here&lt;/strong&gt; (AUC 0.649).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Caveats
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;60 queries is small. One query moves a percentage by about 3 points.&lt;/li&gt;
&lt;li&gt;I wrote both query sets myself, and the unanswerable ones are cleaner than real traffic.&lt;/li&gt;
&lt;li&gt;Thresholds are specific to this model and corpus. Recalibrate whenever you change the embedding model or re-chunk.&lt;/li&gt;
&lt;li&gt;This tests retrieval only. A model can still hallucinate when the retrieved chunks are on topic but do not contain the answer; that needs an answer-level check such as citation validation.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Copyable code
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;should_refuse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query_vec&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;chunk_matrix&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;threshold&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.40&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Vectors must be L2-normalised. Returns (refuse, top_score, top_index).&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;sims&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;chunk_matrix&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="n"&gt;query_vec&lt;/span&gt;
    &lt;span class="n"&gt;top&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;argmax&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sims&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sims&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;threshold&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sims&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="n"&gt;top&lt;/span&gt;

&lt;span class="n"&gt;REFUSAL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I couldn&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;t find this in the documentation.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pick the threshold from your own labeled set: take the highest value that still answers at least 95-100% of your known-answerable queries, then check what it refuses.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule of thumb
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Build a set of at least 50 answerable and 50 unanswerable queries from your own logs, including near-domain ones.&lt;/li&gt;
&lt;li&gt;Use the top-1 cosine score, not BM25, as the gate, and set the threshold just under your lowest answerable score.&lt;/li&gt;
&lt;li&gt;Put the threshold in a test, so a model or chunking change that moves the score scale fails CI.&lt;/li&gt;
&lt;li&gt;Expect to refuse most, not all, of the near-domain questions. Treat the rest with answer-level checks.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The script and raw output are on the &lt;a href="https://recallrun.dev/posts/empty-retrieval-refusal/?utm_source=devto&amp;amp;utm_campaign=rag-idk" rel="noopener noreferrer"&gt;original post&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;I run experiments like this every week at RecallRun. If you want to build RAG systems end to end, the AI Engineering course starts free: &lt;a href="https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=rag-idk" rel="noopener noreferrer"&gt;https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=rag-idk&lt;/a&gt;&lt;/p&gt;

</description>
      <category>rag</category>
      <category>python</category>
      <category>llm</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Metadata filtering in vector search: post-filter loses results, measured</title>
      <dc:creator>RecallRun</dc:creator>
      <pubDate>Thu, 15 Oct 2026 19:19:58 +0000</pubDate>
      <link>https://dev.to/recallrundev/metadata-filtering-in-vector-search-post-filter-loses-results-measured-4gm2</link>
      <guid>https://dev.to/recallrundev/metadata-filtering-in-vector-search-post-filter-loses-results-measured-4gm2</guid>
      <description>&lt;p&gt;Multi-tenant RAG needs a filter: “only search documents this user may see.” Vector indexes are built for nearest neighbours, not for “nearest neighbours among these 1,000.” So where does the filter go: after the search, inside it, or before it?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; post-filtering silently returns fewer results as the filter gets more selective. In my test, a filter that keeps 1% of the data returned 1.9 results out of 10 when I fetched the top 100 and filtered. In-graph filtering fixed recall but got slow. For selective filters, exact search over the filtered subset was the fastest and always correct.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;100,000 vectors, each with a tenant id. I ran two layouts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;random&lt;/strong&gt;: the tenant is independent of content. Think “documents from many customers about the same product.”&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;topical&lt;/strong&gt;: each tenant owns whole topic clusters, and its queries land in its own region.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Selectivity is the fraction of the corpus the filter keeps: about 50%, 10%, 1% and 0.1%. Ground truth is exact search restricted to the tenant’s vectors, so recall@10 here means “how many of the true filtered top 10 did the method return.”&lt;/p&gt;

&lt;p&gt;Methods:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;post-filter&lt;/strong&gt;: ask HNSW for the top &lt;em&gt;k’&lt;/em&gt; (10, 100 or 1000), drop results that fail the filter, keep up to 10.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;in-graph filter&lt;/strong&gt;: hnswlib’s &lt;code&gt;filter&lt;/code&gt; argument, evaluated while the graph is traversed (&lt;a href="https://github.com/nmslib/hnswlib" rel="noopener noreferrer"&gt;hnswlib&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;pre-filter exact&lt;/strong&gt;: select the allowed rows, then do a brute-force dot product. I timed two versions: build the subset per query (mask + gather) and keep a per-tenant matrix prebuilt (partitioned).
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;post_filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mask&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fetch&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;knn_query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;mask&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;]][:&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;in_graph&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mask&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;knn_query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;filter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mask&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;pre_filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;X_tenant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ids_tenant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;  &lt;span class="c1"&gt;# X_tenant = X[ids_tenant], built once
&lt;/span&gt;    &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;X_tenant&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;
    &lt;span class="n"&gt;top&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;argpartition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)[:&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids_tenant&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;argsort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;])]])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Results: random layout
&lt;/h2&gt;

&lt;p&gt;Recall@10 / average results returned (max 10) / median latency.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Selectivity&lt;/th&gt;
&lt;th&gt;post k=10&lt;/th&gt;
&lt;th&gt;post k=100&lt;/th&gt;
&lt;th&gt;post k=1000 (ef=1000)&lt;/th&gt;
&lt;th&gt;in-graph filter&lt;/th&gt;
&lt;th&gt;pre-filter (partitioned)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;50%&lt;/td&gt;
&lt;td&gt;0.553 / 5.5 / 0.21 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 0.24 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 1.94 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 0.44 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 1.38 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10%&lt;/td&gt;
&lt;td&gt;0.181 / 1.8 / 0.20 ms&lt;/td&gt;
&lt;td&gt;0.908 / 9.1 / 0.22 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 1.83 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 2.19 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 0.16 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1%&lt;/td&gt;
&lt;td&gt;0.110 / 1.1 / 0.20 ms&lt;/td&gt;
&lt;td&gt;0.191 / 1.9 / 0.22 ms&lt;/td&gt;
&lt;td&gt;0.884 / 9.2 / 1.88 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 17.03 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 0.03 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.1%&lt;/td&gt;
&lt;td&gt;0.100 / 1.0 / 0.14 ms&lt;/td&gt;
&lt;td&gt;0.105 / 1.0 / 0.16 ms&lt;/td&gt;
&lt;td&gt;0.214 / 2.1 / 1.79 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 74.22 ms&lt;/td&gt;
&lt;td&gt;1.000 / 10 / 0.01 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Post-filtering behaves like a coin flip with the selectivity as the odds: fetch 10, keep about &lt;code&gt;selectivity × 10&lt;/code&gt;. The query is never told that most of its neighbours are off-limits. Even fetching 1,000 candidates returned only 2.1 results per query at 0.1%.&lt;/p&gt;

&lt;p&gt;In-graph filtering never lost recall, but its cost grows as the filter tightens, from 0.44 ms at 50% to 74.22 ms at 0.1%. The traversal has to wander through many forbidden nodes to find allowed ones. At 0.1% it took 74.22 ms, while the exact scan of the 100-vector subset took 0.01 ms.&lt;/p&gt;

&lt;h2&gt;
  
  
  Results: topical layout
&lt;/h2&gt;

&lt;p&gt;When tenants own their own regions of the space, the problem mostly disappears.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Selectivity&lt;/th&gt;
&lt;th&gt;post k=10&lt;/th&gt;
&lt;th&gt;in-graph filter&lt;/th&gt;
&lt;th&gt;pre-filter (partitioned)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;50%&lt;/td&gt;
&lt;td&gt;1.000 / 0.20 ms&lt;/td&gt;
&lt;td&gt;1.000 / 0.22 ms&lt;/td&gt;
&lt;td&gt;1.000 / 1.34 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10%&lt;/td&gt;
&lt;td&gt;1.000 / 0.20 ms&lt;/td&gt;
&lt;td&gt;1.000 / 0.29 ms&lt;/td&gt;
&lt;td&gt;1.000 / 0.16 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1%&lt;/td&gt;
&lt;td&gt;1.000 / 0.12 ms&lt;/td&gt;
&lt;td&gt;1.000 / 0.61 ms&lt;/td&gt;
&lt;td&gt;1.000 / 0.03 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.1%&lt;/td&gt;
&lt;td&gt;1.000 / 0.07 ms&lt;/td&gt;
&lt;td&gt;1.000 / 0.16 ms&lt;/td&gt;
&lt;td&gt;1.000 / 0.01 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Recall is 1.000 for every method. The query’s nearest neighbours are already inside the allowed set, so the filter rejects almost nothing. This is the trap: post-filtering looks fine in a demo where a user asks about their own topic, then fails when a filter is independent of the query. Examples are a date range, a document type, or a tenant whose content overlaps with everyone else’s. Real tenants are somewhere between my two layouts, so test with your own data.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the numbers mean
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Post-filter is only safe when the filter is loose or correlated with the query.&lt;/strong&gt; Below about 10% selectivity on independent filters it breaks, and it breaks quietly: fewer results, not an error. The 0.1% row returned a single result on average.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Over-fetching helps, but needs a lot.&lt;/strong&gt; To reach 0.884 recall at 1% selectivity I had to fetch 1,000 and raise &lt;code&gt;ef&lt;/code&gt; to 1,000, which cost 1.88 ms versus 0.22 ms.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;In-graph filtering is the default to reach for between ~10% and 100%&lt;/strong&gt;, where it stayed between 0.22 and 2.19 ms (and at 10% the exact partitioned scan was faster, 0.16 ms).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Below ~1%, scan the subset.&lt;/strong&gt; At 1% the subset has about 1,000 vectors, so a dot product takes 0.03 ms. This is the same reasoning as in &lt;a href="//../do-you-need-a-vector-database/"&gt;Do you need a vector database?&lt;/a&gt;: exact search is cheap at small sizes.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Caveats
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Synthetic Gaussian clusters, 128 dimensions, one index configuration. Real embeddings will have different structure. Treat the shape of the curves as the result, not the exact milliseconds.&lt;/li&gt;
&lt;li&gt;One tenant per setting, 200 queries each. Latency is a median on a shared 2 vCPU machine.&lt;/li&gt;
&lt;li&gt;The partitioned pre-filter times exclude building the per-tenant matrices. It needs memory for a second copy of the vectors, or an index per tenant.&lt;/li&gt;
&lt;li&gt;hnswlib’s filter is a Python callback here. A native filter in a database engine will have different costs. The qualitative trend (slower as selectivity falls) is reported for hnswlib in the &lt;a href="https://arxiv.org/abs/2507.11907" rel="noopener noreferrer"&gt;SIEVE paper&lt;/a&gt;, which also notes that hnswlib applies the filter during search.&lt;/li&gt;
&lt;li&gt;My first run gave a misleading 32 ms for exact search over 50k vectors. The cause was a float64 query array: &lt;code&gt;np.sqrt(D)&lt;/code&gt; returns a float64 scalar, and under NumPy 2 this upcast the query, so each dot product converted the whole float32 matrix. I fixed it by casting queries to float32 and reran everything. Check dtypes if your brute-force numbers look too slow.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Rule of thumb
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Estimate selectivity per query: allowed rows divided by total rows.&lt;/li&gt;
&lt;li&gt;Above ~50%, post-filter with a modest over-fetch is fine.&lt;/li&gt;
&lt;li&gt;Between ~10% and 50%, use the index’s in-graph filter.&lt;/li&gt;
&lt;li&gt;Below ~1%, or when the filter is a hard permission boundary, search the filtered subset exactly (or give each tenant its own index).&lt;/li&gt;
&lt;li&gt;Always log how many results survive the filter. A silent drop from 10 to 1 is the failure to catch. A test that asserts &lt;code&gt;len(results) == k&lt;/code&gt; on an independent filter would have caught it here.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For the basics of embeddings and indexes, see the free lesson &lt;a href="//../../learn/ai-engineering/06-vector-search.html"&gt;Embeddings &amp;amp; vector databases&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://recallrun.dev/posts/rag-metadata-filtering/" rel="noopener noreferrer"&gt;RecallRun&lt;/a&gt;.&lt;/em&gt; I run experiments like this every week at RecallRun. If you want to build RAG systems end to end, the AI Engineering course starts free: &lt;a href="https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=metadata-filter" rel="noopener noreferrer"&gt;https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=metadata-filter&lt;/a&gt;&lt;/p&gt;

</description>
      <category>rag</category>
      <category>vectordatabase</category>
      <category>python</category>
      <category>performance</category>
    </item>
    <item>
      <title>Do you need a vector database? Brute-force search benchmarked from 10k to 1M vectors</title>
      <dc:creator>RecallRun</dc:creator>
      <pubDate>Wed, 07 Oct 2026 11:01:14 +0000</pubDate>
      <link>https://dev.to/recallrundev/do-you-need-a-vector-database-brute-force-search-benchmarked-from-10k-to-1m-vectors-6fk</link>
      <guid>https://dev.to/recallrundev/do-you-need-a-vector-database-brute-force-search-benchmarked-from-10k-to-1m-vectors-6fk</guid>
      <description>&lt;p&gt;Most RAG tutorials start by installing a vector database. For a lot of projects, that is a dependency you don't need yet. I measured how long plain exact search takes as the corpus grows, and what an HNSW index buys you in return.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; below about 100,000 chunks, a numpy matrix multiply is fast enough for almost any RAG app. Between 100k and 1M it depends on your traffic. Past that, use an index.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;Every embedding is normalized, so cosine similarity is a dot product. Exact search is one matrix-vector product and a partial sort:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;top_k&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;X&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ndarray&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ndarray&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ndarray&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;X: (n, d) normalized float32 matrix. q: (d,) normalized query.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;scores&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;X&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;
    &lt;span class="n"&gt;idx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;argpartition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)[:&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;argsort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I timed this against &lt;code&gt;faiss.IndexFlatIP&lt;/code&gt; (also exact) for three common embedding sizes: 384 (MiniLM-class models), 768 (BERT-base-class) and 1536 (OpenAI &lt;code&gt;text-embedding-3-small&lt;/code&gt; default). The machine is deliberately small: 2 vCPUs, the size of a cheap cloud instance.&lt;/p&gt;

&lt;h2&gt;
  
  
  Exact search latency
&lt;/h2&gt;

&lt;p&gt;Single-query latency, median of 50 queries:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Dim&lt;/th&gt;
&lt;th&gt;Vectors&lt;/th&gt;
&lt;th&gt;RAM for vectors&lt;/th&gt;
&lt;th&gt;numpy&lt;/th&gt;
&lt;th&gt;faiss Flat&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;384&lt;/td&gt;
&lt;td&gt;10,000&lt;/td&gt;
&lt;td&gt;15 MB&lt;/td&gt;
&lt;td&gt;0.35 ms&lt;/td&gt;
&lt;td&gt;0.78 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;384&lt;/td&gt;
&lt;td&gt;100,000&lt;/td&gt;
&lt;td&gt;154 MB&lt;/td&gt;
&lt;td&gt;3.5 ms&lt;/td&gt;
&lt;td&gt;8.3 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;384&lt;/td&gt;
&lt;td&gt;500,000&lt;/td&gt;
&lt;td&gt;768 MB&lt;/td&gt;
&lt;td&gt;31 ms&lt;/td&gt;
&lt;td&gt;66 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;384&lt;/td&gt;
&lt;td&gt;1,000,000&lt;/td&gt;
&lt;td&gt;1.5 GB&lt;/td&gt;
&lt;td&gt;62 ms&lt;/td&gt;
&lt;td&gt;122 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;768&lt;/td&gt;
&lt;td&gt;100,000&lt;/td&gt;
&lt;td&gt;307 MB&lt;/td&gt;
&lt;td&gt;10.6 ms&lt;/td&gt;
&lt;td&gt;22.6 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;768&lt;/td&gt;
&lt;td&gt;500,000&lt;/td&gt;
&lt;td&gt;1.5 GB&lt;/td&gt;
&lt;td&gt;57 ms&lt;/td&gt;
&lt;td&gt;120 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;768&lt;/td&gt;
&lt;td&gt;1,000,000&lt;/td&gt;
&lt;td&gt;3.1 GB&lt;/td&gt;
&lt;td&gt;114 ms&lt;/td&gt;
&lt;td&gt;240 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1536&lt;/td&gt;
&lt;td&gt;10,000&lt;/td&gt;
&lt;td&gt;61 MB&lt;/td&gt;
&lt;td&gt;1.2 ms&lt;/td&gt;
&lt;td&gt;3.1 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1536&lt;/td&gt;
&lt;td&gt;100,000&lt;/td&gt;
&lt;td&gt;614 MB&lt;/td&gt;
&lt;td&gt;20 ms&lt;/td&gt;
&lt;td&gt;51 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1536&lt;/td&gt;
&lt;td&gt;500,000&lt;/td&gt;
&lt;td&gt;3.1 GB&lt;/td&gt;
&lt;td&gt;107 ms&lt;/td&gt;
&lt;td&gt;248 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three things stand out:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Latency scales linearly with &lt;code&gt;n × d&lt;/code&gt;.&lt;/strong&gt; It is a memory-bandwidth problem. Doubling either the corpus or the dimension doubles the time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory is the real limit, not speed.&lt;/strong&gt; 1M vectors at 1536 dims is 6 GB of float32 before any metadata. I skipped that row: it won't fit comfortably on a 7 GB machine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Plain numpy beat faiss Flat for single queries here&lt;/strong&gt;, by about 2×. Faiss is built for batches; one query at a time pays overhead that numpy's direct matrix-vector call doesn't. If you batch queries, test again before assuming this holds.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What an HNSW index buys you
&lt;/h2&gt;

&lt;p&gt;Next, I built &lt;code&gt;hnswlib&lt;/code&gt; indexes (M=16, ef_construction=200) and measured query time and &lt;strong&gt;recall@10&lt;/strong&gt;: the fraction of the true top 10 the index actually returns.&lt;/p&gt;

&lt;p&gt;I used two synthetic datasets, because recall depends heavily on how your data is shaped:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Uniform:&lt;/strong&gt; random Gaussian directions. This is the worst case for any approximate index: no structure to exploit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Clustered:&lt;/strong&gt; 2,000 tight clusters. Real embeddings sit between these two, usually much closer to clustered, since documents about the same topic land near each other.&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Data&lt;/th&gt;
&lt;th&gt;Dim&lt;/th&gt;
&lt;th&gt;Vectors&lt;/th&gt;
&lt;th&gt;Build time&lt;/th&gt;
&lt;th&gt;ef=32&lt;/th&gt;
&lt;th&gt;ef=64&lt;/th&gt;
&lt;th&gt;ef=128&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;clustered&lt;/td&gt;
&lt;td&gt;384&lt;/td&gt;
&lt;td&gt;100k&lt;/td&gt;
&lt;td&gt;23 s&lt;/td&gt;
&lt;td&gt;0.10 ms, recall 0.954&lt;/td&gt;
&lt;td&gt;0.17 ms, 0.995&lt;/td&gt;
&lt;td&gt;0.34 ms, 1.000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;clustered&lt;/td&gt;
&lt;td&gt;768&lt;/td&gt;
&lt;td&gt;100k&lt;/td&gt;
&lt;td&gt;48 s&lt;/td&gt;
&lt;td&gt;0.22 ms, 0.878&lt;/td&gt;
&lt;td&gt;0.36 ms, 0.985&lt;/td&gt;
&lt;td&gt;0.68 ms, 1.000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;uniform&lt;/td&gt;
&lt;td&gt;384&lt;/td&gt;
&lt;td&gt;100k&lt;/td&gt;
&lt;td&gt;37 s&lt;/td&gt;
&lt;td&gt;0.16 ms, 0.129&lt;/td&gt;
&lt;td&gt;0.31 ms, 0.206&lt;/td&gt;
&lt;td&gt;0.55 ms, 0.281&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;uniform&lt;/td&gt;
&lt;td&gt;768&lt;/td&gt;
&lt;td&gt;100k&lt;/td&gt;
&lt;td&gt;69 s&lt;/td&gt;
&lt;td&gt;0.34 ms, 0.058&lt;/td&gt;
&lt;td&gt;0.63 ms, 0.105&lt;/td&gt;
&lt;td&gt;1.07 ms, 0.172&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;uniform&lt;/td&gt;
&lt;td&gt;384&lt;/td&gt;
&lt;td&gt;500k&lt;/td&gt;
&lt;td&gt;279 s&lt;/td&gt;
&lt;td&gt;0.19 ms, 0.034&lt;/td&gt;
&lt;td&gt;0.35 ms, 0.057&lt;/td&gt;
&lt;td&gt;0.67 ms, 0.099&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;HNSW queries are &lt;strong&gt;10 to 100 times faster&lt;/strong&gt; than exact search at these sizes. The cost:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Build time.&lt;/strong&gt; 500k vectors took over 4.5 minutes on 2 cores. You pay this again on every full re-index, such as after switching embedding models.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recall is not guaranteed.&lt;/strong&gt; On structured data, &lt;code&gt;ef=64&lt;/code&gt; gave near-perfect recall. On structureless data, the same settings returned less than a quarter of the true neighbors. Your data will be somewhere in between, so &lt;strong&gt;measure recall on your own embeddings&lt;/strong&gt; before trusting the defaults.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where the time actually goes in a RAG request
&lt;/h2&gt;

&lt;p&gt;A typical RAG request spends 1 to 5 seconds waiting for the LLM to generate. Next to that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;3.5 ms (100k × 384, exact) is invisible.&lt;/li&gt;
&lt;li&gt;60 to 110 ms (1M exact) is noticeable only if you care about p99, or serve many queries per second on the same box.&lt;/li&gt;
&lt;li&gt;Exact search gives perfect recall for free, which removes one variable when you debug bad answers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A rough sizing rule: one PDF page is usually 2 to 3 chunks. &lt;strong&gt;100k chunks is roughly 30,000 to 50,000 pages.&lt;/strong&gt; Many internal "chat with our docs" projects never get there.&lt;/p&gt;

&lt;h2&gt;
  
  
  My rule of thumb
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Your corpus&lt;/th&gt;
&lt;th&gt;What I'd use&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;under 100k chunks&lt;/td&gt;
&lt;td&gt;numpy (or pgvector with no index). Store vectors in a &lt;code&gt;.npy&lt;/code&gt; file or Postgres.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100k to 1M chunks&lt;/td&gt;
&lt;td&gt;exact search is still fine at low QPS. Add HNSW when p95 latency or QPS demands it, and measure recall.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;over 1M chunks, or many tenants&lt;/td&gt;
&lt;td&gt;a proper ANN index (pgvector HNSW, Qdrant, Milvus, OpenSearch), plus filtering and sharding.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Most importantly, &lt;strong&gt;keep the vector store behind one interface&lt;/strong&gt; so you can swap numpy for pgvector later without touching the rest of the pipeline. That one seam is worth more than picking the "right" database on day one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproduce it
&lt;/h2&gt;

&lt;p&gt;The exact-search benchmark:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;faiss&lt;/span&gt;
&lt;span class="n"&gt;faiss&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;omp_set_num_threads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;rng&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;default_rng&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;norm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;linalg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;keepdims&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;384&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;768&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1536&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;100_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;500_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1_000_000&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mf"&gt;3.2e9&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt;          &lt;span class="c1"&gt;# skip what won't fit in RAM
&lt;/span&gt;        &lt;span class="n"&gt;X&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;standard_normal&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float32&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;Q&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;standard_normal&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float32&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;Q&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
            &lt;span class="n"&gt;sc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;X&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;
            &lt;span class="n"&gt;idx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;argpartition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)[:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
            &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;median&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="mf"&gt;1e3&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; ms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For HNSW, build with &lt;code&gt;hnswlib.Index(space="ip", dim=d)&lt;/code&gt;, &lt;code&gt;init_index(max_elements=n, ef_construction=200, M=16)&lt;/code&gt;, and compare &lt;code&gt;knn_query&lt;/code&gt; results against the exact top 10 to get recall.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Caveat:&lt;/strong&gt; these are synthetic vectors on one small machine. Latency for exact search doesn't depend on what the vectors mean, so those numbers transfer well. HNSW recall does depend on your data, which is exactly why you should measure it yourself.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Test setup: 2 vCPU Intel Xeon @ 2.1 GHz, 7 GB RAM, Python 3.11, numpy 2.4.4, faiss-cpu 1.15.1, hnswlib. Synthetic normalized float32 vectors. Median of 50 (exact) or 200 (HNSW) single queries.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;I run experiments like this every week at RecallRun. If you want to build RAG systems end to end, the AI Engineering course starts free: &lt;a href="https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=vector-db" rel="noopener noreferrer"&gt;https://recallrun.dev/learn/ai-engineering/?utm_source=devto&amp;amp;utm_campaign=vector-db&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>ai</category>
      <category>database</category>
      <category>rag</category>
    </item>
  </channel>
</rss>
