<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Muhammad Hussnain Jamil</title>
    <description>The latest articles on DEV Community by Muhammad Hussnain Jamil (@hussnainoffical).</description>
    <link>https://dev.to/hussnainoffical</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4213274%2Fcadbb6f0-998f-4eaa-baaf-c09dcf1beb94.jpg</url>
      <title>DEV Community: Muhammad Hussnain Jamil</title>
      <link>https://dev.to/hussnainoffical</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/hussnainoffical"/>
    <language>en</language>
    <item>
      <title>LLM Inference Cost in 2026: Why Cost per Finished Task Beats Cost per Token</title>
      <dc:creator>Muhammad Hussnain Jamil</dc:creator>
      <pubDate>Tue, 10 Nov 2026 07:30:23 +0000</pubDate>
      <link>https://dev.to/hussnainoffical/llm-inference-cost-in-2026-why-cost-per-finished-task-beats-cost-per-token-22k2</link>
      <guid>https://dev.to/hussnainoffical/llm-inference-cost-in-2026-why-cost-per-finished-task-beats-cost-per-token-22k2</guid>
      <description>&lt;p&gt;Most conversations about LLM cost start and end with a price table: dollars per million input tokens, dollars per million output tokens, pick the cheapest model that passes a demo. That table is necessary, but it measures the wrong unit. Your users do not buy tokens. They want a support ticket triaged, an invoice extracted, a contract clause flagged or a pull request reviewed.&lt;/p&gt;

&lt;p&gt;The unit that matters is &lt;strong&gt;cost per finished task&lt;/strong&gt;: everything you paid, across every call, retry, tool loop and fallback, divided by the number of tasks that actually reached your quality bar. Latency works the same way. Users feel the time until the task is done, not the time to the first token of one call.&lt;/p&gt;

&lt;p&gt;This article is for engineers and tech leads running LLM features in production, with a focus on Germany and the EU. It covers pricing mechanics, why cheaper tokens can mean pricier tasks, measurement, the main levers, self-hosting break-even and EU hosting, and ends with a checklist.&lt;/p&gt;

&lt;p&gt;All prices below were checked on 6 November 2026 and will change. Epoch AI found that the price of reaching a fixed performance level has fallen between 9x and 900x per year, depending on the milestone [1]. Treat every price here as an example, and any routing decision based on today's price table as short-lived.&lt;/p&gt;

&lt;h2&gt;
  
  
  How token pricing actually works in late 2026
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Input versus output
&lt;/h3&gt;

&lt;p&gt;Output tokens cost several times more than input tokens. OpenAI lists gpt-6.1-sol at $2.00 per million input tokens and $10.00 per million output tokens, and gpt-6-luna at $0.10 and $0.50 [2]. Anthropic lists Claude Sonnet 5.5 at $2 and $10, and Claude Opus 5.5 at $4 and $20 [3]. In all four cases output costs 5x input.&lt;/p&gt;

&lt;p&gt;With a 5:1 price ratio, output dominates the bill as soon as a task produces more than one output token for every five input tokens. Many benchmark sites assume a fixed mix. Artificial Analysis, for example, blends input and output prices at a 3:1 ratio [4]. That is a reasonable default for comparing models, but your workload has its own ratio, and you should compute it from your logs instead of borrowing someone else's.&lt;/p&gt;

&lt;p&gt;Token counts also depend on language. OpenAI's rule of thumb is about 4 characters or three quarters of a word per token in English, and it notes that non-English text can cost more per character [5]. If your product serves German users, measure token counts on German prompts and German outputs, not on English test fixtures.&lt;/p&gt;

&lt;p&gt;Some input is invisible in your code. When you pass tools to Claude models, Anthropic adds a tool-use system prompt of 286 tokens for the listed models, on top of the tool schemas themselves [3]. Every tool definition you send is input you pay for on every call.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cached input
&lt;/h3&gt;

&lt;p&gt;All three large API vendors now discount repeated prompt prefixes, but the mechanics differ.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;OpenAI&lt;/strong&gt; caches automatically. The minimum cacheable prompt is 1,024 tokens for GPT-5.6 and later, the default cache lifetime is 30 minutes (12 hours can be requested in the Responses API), and cache writes on these models cost more than standard input [6]. Cached input for gpt-6.1-sol is listed at $0.10 against $2.00 for uncached input [2].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Anthropic&lt;/strong&gt; uses explicit &lt;code&gt;cache_control&lt;/code&gt; breakpoints. A 5-minute cache write costs 1.25x the base input price, a 1-hour write costs 2x, and a cache read costs 0.1x, or 0.05x on Claude Opus 5.5 and Sonnet 5.5 [3]. Prompts below a model-specific minimum (between 1,024 and 4,096 tokens for the models listed) are processed without caching, and no error is returned [7].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google Gemini&lt;/strong&gt; caches implicitly on Gemini 2.5 and newer models, and lists a 90% discount on cached tokens plus prorated storage for caching [8].&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The break-even point for Anthropic-style pricing is easy to compute. With a 5-minute cache, two requests sharing a prefix cost 1.25 + 0.1 = 1.35 units of prefix price instead of 2.0, so caching pays off on the first reuse. With a 1-hour cache, two requests cost 2.0 + 0.1 = 2.1 units, which is more than 2.0, so you need at least two reuses before the longer TTL pays off.&lt;/p&gt;

&lt;p&gt;Caches are prefix matches, so small changes break them. Anthropic documents that modifying tool definitions invalidates the entire cache, and that changing thinking settings invalidates message-level cache entries [7]. Changing the thinking budget or the effort level between requests also invalidates cache breakpoints [9][10]. OpenAI's guidance is to put stable instructions and shared reference material first and dynamic content such as timestamps or user-specific text at the end [6].&lt;/p&gt;

&lt;h3&gt;
  
  
  Reasoning tokens
&lt;/h3&gt;

&lt;p&gt;Reasoning ("thinking") tokens are the most common source of surprise on an invoice. OpenAI states that reasoning tokens are not visible through the API but still occupy context and are billed as output tokens, reported as &lt;code&gt;output_tokens_details.reasoning_tokens&lt;/code&gt; [11]. Anthropic bills the full internal thinking as output tokens even when you only receive a summary, and states that the billed output count does not match the visible token count [10].&lt;/p&gt;

&lt;p&gt;Two details matter for cost control:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;OpenAI recommends reserving at least 25,000 tokens for reasoning and outputs when you start experimenting with reasoning models [11]. That is the order of magnitude a single call can consume.&lt;/li&gt;
&lt;li&gt;Anthropic notes that &lt;code&gt;max_tokens&lt;/code&gt; is a hard cap per request, but in a tool-use loop each request has its own &lt;code&gt;max_tokens&lt;/code&gt;, so it "doesn't bound the whole turn's spend" [10]. A task-level budget has to live in your orchestrator.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Batch, flex and paying for speed
&lt;/h3&gt;

&lt;p&gt;If a task does not need an answer within seconds, you can usually halve its price.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Anthropic's Batch API gives a 50% discount on input and output tokens [3].&lt;/li&gt;
&lt;li&gt;Gemini's Batch API runs at 50% of standard cost with a 24-hour target turnaround, and Gemini Flex inference also costs 50% of standard with a target of 1 to 15 minutes [8].&lt;/li&gt;
&lt;li&gt;OpenAI's Flex/Batch prices are half the standard rates, for example $1.00 input and $5.00 output for gpt-6.1-sol [2].&lt;/li&gt;
&lt;li&gt;Azure OpenAI Global Batch and Data Zone Batch deployments are 50% cheaper than the corresponding standard deployments, with a 24-hour target and no real-time SLA [12].&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The opposite direction exists too. OpenAI renamed Priority processing to Fast mode on 30 July 2026; Fast mode for gpt-6.1-sol is listed at $4.00 input and $20.00 output, and an Ultrafast tier at $12.00 and $60.00 [2]. Gemini Priority inference costs 75% to 100% more than standard [8]. Latency is now an explicit line item.&lt;/p&gt;

&lt;h3&gt;
  
  
  Long context and region multipliers
&lt;/h3&gt;

&lt;p&gt;Prompt length can change the per-token price. Claude Haiku 5.5 charges higher prices for any request whose prompt exceeds 100,000 tokens, counting cache reads and writes toward that length, while other Claude 4.6 and later models charge the same rate across the full 1M-token window [3].&lt;/p&gt;

&lt;p&gt;Data residency also costs money. OpenAI charges a 10% uplift on regional processing endpoints for models released on or after 5 March 2026 [2]. Anthropic applies a 1.1x multiplier to all token categories when you request US-only inference with &lt;code&gt;inference_geo&lt;/code&gt; [3]. If you need regional processing, add that multiplier to your cost model from day one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a cheaper model per token can cost more per task
&lt;/h2&gt;

&lt;p&gt;Four mechanisms turn a low token price into a high task price.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;More calls.&lt;/strong&gt; Agentic tasks are loops: the model plans, calls a tool, reads the result and continues. Every step resends a growing context. Anthropic reported that its agents typically use about 4x more tokens than chat interactions, and multi-agent systems about 15x more [13]. A model that needs six steps where another needs four pays for that context six times.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Longer reasoning.&lt;/strong&gt; Reasoning models can spend many tokens on easy inputs. A study of o1-like models found they used on average 1,953% more tokens than conventional models to answer simple problems, and that a mitigation reduced token output by 48.6% on MATH500 while maintaining accuracy for QwQ-32B-Preview [14].&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Retries.&lt;/strong&gt; If the output fails validation (broken JSON, a missing field, an answer that fails a business rule), you pay again. When OpenAI introduced Structured Outputs, it reported that the new model scored 100% on its complex JSON schema evals while gpt-4-0613 scored less than 40% [15]. Every percentage point of format failure is a retry you pay for.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Failure handling.&lt;/strong&gt; Some tasks fail after all retries and go to a stronger model or a human. Kapoor et al. showed that agents with similar accuracy can differ in cost by almost two orders of magnitude; on HumanEval, a simple baseline reached about 93% accuracy for about $2.45 in total, while a more complex agent architecture cost about $134.50 for 88% [16]. Accuracy and cost have to be evaluated together.&lt;/p&gt;

&lt;h3&gt;
  
  
  A formula for cost per finished task
&lt;/h3&gt;

&lt;p&gt;Start with the cost of a single model call, using per-million-token prices:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;c_call = ( N_in * p_in
         + N_cache_read * p_cache_read
         + N_cache_write * p_cache_write
         + N_out * p_out ) / 1,000,000 * m_region

N_out includes reasoning tokens.
m_region is 1.0, or 1.1 for the regional uplifts quoted above.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An attempt is the sequence of calls needed to try the task once, plus any validation calls:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;C_attempt = sum of c_call over all calls in the attempt (+ tool and validation costs)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If an attempt succeeds with probability &lt;code&gt;s&lt;/code&gt; and you allow at most &lt;code&gt;k&lt;/code&gt; attempts, the expected number of attempts is a truncated geometric series:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;E[attempts] = (1 - (1 - s)^k) / s
P(all k attempts fail) = (1 - s)^k
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If failed tasks are escalated at cost &lt;code&gt;E&lt;/code&gt; (a stronger model, or a person's time), the expected cost per task is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;C_task = C_attempt * (1 - (1 - s)^k) / s  +  (1 - s)^k * E
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If failed tasks are simply dropped, divide total spend by the number of successful tasks instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;C_finished = C_attempt * E[attempts] / (1 - (1 - s)^k)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Worked example (illustrative)
&lt;/h3&gt;

&lt;p&gt;The numbers below are &lt;strong&gt;illustrative assumptions&lt;/strong&gt;, not measurements. They show how the formula behaves. Prices are the OpenAI list prices quoted above; the token counts and success rates are invented to make the mechanics visible.&lt;/p&gt;

&lt;p&gt;Task: classify a support ticket and extract structured fields, using a small agent loop with tools. Both models share a 6,000-token static prefix (system prompt plus tool definitions) that is read from cache on every call. Cache writes are ignored because they amortise across many tasks. Each model gets at most two attempts.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Assumption&lt;/th&gt;
&lt;th&gt;Model A (gpt-6.1-sol prices)&lt;/th&gt;
&lt;th&gt;Model B (gpt-6.1-terra prices)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Price per 1M tokens: input / cached / output&lt;/td&gt;
&lt;td&gt;$2.00 / $0.10 / $10.00&lt;/td&gt;
&lt;td&gt;$1.00 / $0.05 / $5.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Calls per attempt&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Uncached input tokens per call&lt;/td&gt;
&lt;td&gt;2,000&lt;/td&gt;
&lt;td&gt;2,500&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cached input tokens per call&lt;/td&gt;
&lt;td&gt;6,000&lt;/td&gt;
&lt;td&gt;6,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Output tokens per call (visible + reasoning)&lt;/td&gt;
&lt;td&gt;300 + 800&lt;/td&gt;
&lt;td&gt;300 + 1,500&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Success probability per attempt &lt;code&gt;s&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;0.90&lt;/td&gt;
&lt;td&gt;0.75&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Per call, Model A costs 2,000 x $2 + 6,000 x $0.10 + 1,100 x $10, divided by one million, which is $0.0156. Four calls make $0.0624 per attempt. Model B costs $0.0025 + $0.0003 + $0.0090 = $0.0118 per call, and six calls make $0.0708 per attempt.&lt;/p&gt;

&lt;p&gt;With &lt;code&gt;k = 2&lt;/code&gt;, Model A needs (1 - 0.1^2) / 0.9 = 1.10 expected attempts and Model B needs (1 - 0.25^2) / 0.75 = 1.25. Expected spend per task is $0.0686 for A and $0.0885 for B. Model A finishes 99% of tasks and Model B finishes 93.75%, so the cost per finished task is &lt;strong&gt;$0.0693 for A and $0.0944 for B&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Model B is half the price per token and about 36% more expensive per finished task. It also sends 6.25% of tasks to the escalation path instead of 1%, so any non-zero escalation cost &lt;code&gt;E&lt;/code&gt; widens the gap by 0.0525 x &lt;code&gt;E&lt;/code&gt; per task. On latency, B generates about 13,500 output tokens per task in expectation against about 4,840 for A, so at equal decode speed it spends roughly 2.8x as long generating.&lt;/p&gt;

&lt;p&gt;Change the assumptions and the answer can flip. That is the point: &lt;code&gt;s&lt;/code&gt;, the number of calls and the reasoning length are properties of your traffic, and you can only get them from your own logs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measuring cost and latency per task
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Call-level latency metrics
&lt;/h3&gt;

&lt;p&gt;NVIDIA's benchmarking guide gives standard definitions [17]:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Time to first token (TTFT):&lt;/strong&gt; time to process the prompt and generate the first token. It generally includes queueing, prefill and network latency, and grows with prompt length.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;End-to-end latency:&lt;/strong&gt; TTFT plus generation time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Inter-token latency (ITL) or time per output token (TPOT):&lt;/strong&gt; the average gap between output tokens during decoding.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tokens per second:&lt;/strong&gt; per user (output length divided by end-to-end latency) or per system (all output tokens across concurrent requests).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The same guide describes the core trade-off: raising concurrency increases system throughput until the GPU saturates, while per-user speed falls and requests beyond the batch limit queue, which increases TTFT [17].&lt;/p&gt;

&lt;p&gt;Public leaderboards help with shortlisting. Artificial Analysis tests endpoints eight times a day with prompts of roughly 100, 1k and 10k tokens and reports the median over 14 days [4]. Those numbers are not your SLO: your prompts, your region and your concurrency differ.&lt;/p&gt;

&lt;h3&gt;
  
  
  Task-level latency
&lt;/h3&gt;

&lt;p&gt;For a task, measure wall-clock time from the user's request to the validated result. That includes every sequential model call, tool execution, retries and queueing in your own services. Report p50 and p95 per task type. Do not add up per-call p95 values to estimate task p95; percentiles do not add, so measure the task directly.&lt;/p&gt;

&lt;p&gt;For reasoning models, also track time to the first &lt;em&gt;visible&lt;/em&gt; token. Reasoning tokens are generated before the answer, so a fast TTFT on the wire can still mean a long wait for the user.&lt;/p&gt;

&lt;p&gt;If you self-host with vLLM, the engine exposes Prometheus histograms for &lt;code&gt;vllm:time_to_first_token_seconds&lt;/code&gt;, &lt;code&gt;vllm:inter_token_latency_seconds&lt;/code&gt;, &lt;code&gt;vllm:e2e_request_latency_seconds&lt;/code&gt; and &lt;code&gt;vllm:request_queue_time_seconds&lt;/code&gt;, plus gauges for &lt;code&gt;vllm:num_requests_waiting&lt;/code&gt; and &lt;code&gt;vllm:kv_cache_usage_perc&lt;/code&gt; and a &lt;code&gt;vllm:prefix_cache_hits&lt;/code&gt; counter [18]. Queue time is the number to watch when p95 TTFT drifts upward.&lt;/p&gt;

&lt;h3&gt;
  
  
  What to log for every task
&lt;/h3&gt;

&lt;p&gt;Log one record per task, not just per call, with: task type, task ID, model per call, uncached input, cache read and cache write tokens, output and reasoning tokens, number of calls, number of attempts, final success flag, cost, TTFT of the first call and wall time. Do not log prompt text in the same pipeline. Token counts are enough for cost analysis, and keeping user content out of metrics makes GDPR reviews much simpler.&lt;/p&gt;

&lt;p&gt;A minimal Python version, with an adapter for Anthropic's usage fields (where &lt;code&gt;input_tokens&lt;/code&gt; excludes cache reads and writes, and &lt;code&gt;output_tokens&lt;/code&gt; is the inclusive total that includes thinking [7][10]):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;dataclasses&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;

&lt;span class="c1"&gt;# USD per 1M tokens. Copy from your provider's pricing page; these will go stale.&lt;/span&gt;
&lt;span class="no"&gt;PRICES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"model-a"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"in"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;2.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cache_read"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cache_write"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;2.50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"out"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;10.00&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"model-b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"in"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cache_read"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.05&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cache_write"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.25&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"out"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;5.00&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Call&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;in_tok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;            &lt;span class="c1"&gt;# uncached input&lt;/span&gt;
    &lt;span class="n"&gt;cache_read_tok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;cache_write_tok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;out_tok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;           &lt;span class="c1"&gt;# billed output, reasoning included&lt;/span&gt;
    &lt;span class="n"&gt;reasoning_tok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;ttft_s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;
    &lt;span class="n"&gt;total_s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;

    &lt;span class="nd"&gt;@classmethod&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;from_anthropic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ttft_s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;total_s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;details&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"output_tokens_details"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;cls&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"input_tokens"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"cache_read_input_tokens"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                   &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"cache_creation_input_tokens"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"output_tokens"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                   &lt;span class="n"&gt;details&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"thinking_tokens"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ttft_s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;total_s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;cost&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;PRICES&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;in_tok&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"in"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cache_read_tok&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"cache_read"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
                &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cache_write_tok&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"cache_write"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;out_tok&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"out"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e6&lt;/span&gt;

&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Task&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;
    &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;uuid4&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;hex&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;finish&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;success&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;inp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;in_tok&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cache_read_tok&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cache_write_tok&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;rec&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="s2"&gt;"task_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"kind"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"success"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;success&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"calls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"cost_usd"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cost&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"out_tok"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;out_tok&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"reasoning_tok"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reasoning_tok&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"cache_hit_ratio"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cache_read_tok&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;inp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;inp&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"wall_s"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rec&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# ship to your log pipeline; never log prompt text here&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;rec&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;summarize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;done&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"success"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;wall&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"wall_s"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"cost_per_finished_task"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"cost_usd"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"success_rate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"p50_s"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;median&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wall&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"p95_s"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;quantiles&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wall&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wall&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;wall&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Retries of the same task should append to the same &lt;code&gt;Task&lt;/code&gt;, so their cost lands on the task that caused them. For other providers, write a second &lt;code&gt;from_...&lt;/code&gt; adapter and check in the docs whether the reported input count already includes cached tokens.&lt;/p&gt;

&lt;h2&gt;
  
  
  The levers, roughly in order of effort
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Cut tokens you do not need
&lt;/h3&gt;

&lt;p&gt;Output is the expensive side, so start there.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Bound reasoning.&lt;/strong&gt; Anthropic recommends lowering the effort level before trying prompt-based steering, and treats &lt;code&gt;max_tokens&lt;/code&gt; as the hard cap on thinking plus answer [10]. OpenAI exposes &lt;code&gt;reasoning.effort&lt;/code&gt; with model-dependent values from &lt;code&gt;none&lt;/code&gt; to &lt;code&gt;max&lt;/code&gt; [11]. Run your eval set at two or three effort levels and pick the lowest one that holds quality. The overthinking results above suggest that easy inputs often do not need the highest setting [14].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Constrain the format.&lt;/strong&gt; Structured outputs with a strict schema remove a whole class of retries [15]. Budget for a one-time delay: OpenAI notes that the first request with a new schema incurs extra latency, typically under 10 seconds, after which the processed schema is cached [15].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ask for less.&lt;/strong&gt; Short field values, no restated input, no unread explanations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Send fewer tools.&lt;/strong&gt; Each tool definition is input on every call. Group tools per route so each route has a small, stable set, because changing tool definitions also invalidates the cache [7].&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Make prompt caching work
&lt;/h3&gt;

&lt;p&gt;Order prompts as static to dynamic: system instructions, tool definitions, shared reference text, then conversation history, then the new user input [6]. Keep model settings such as thinking effort stable within a conversation [10]. Track the cache hit ratio per task type (cache read tokens divided by total input tokens), and alert when it drops after a deploy, because a small template change can silently move a timestamp to the top of the prompt.&lt;/p&gt;

&lt;p&gt;Self-hosted engines offer the same idea. vLLM's automatic prefix caching reuses the KV cache of earlier requests that share a prefix, but it only reduces prefill time, not decode time, so it helps most with long shared contexts and short answers [19]. SGLang uses RadixAttention for KV cache reuse, and its paper reports up to 6.4x higher throughput than other inference systems on workloads such as agent control, few-shot benchmarks and multi-turn chat [20].&lt;/p&gt;

&lt;h3&gt;
  
  
  Move asynchronous work to batch or flex
&lt;/h3&gt;

&lt;p&gt;Nightly enrichment, document backfills, evaluation runs and report generation rarely need second-level latency. Moving them to batch or flex tiers halves token cost at all three major vendors and on Azure [3][8][2][12]. Build a small queue abstraction so a task can declare "needed within 24 hours" and the platform picks the tier.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cache whole responses where it is safe
&lt;/h3&gt;

&lt;p&gt;Exact-match response caching (same normalised input, same model, same prompt version) is safe and cheap. Semantic caching, which reuses answers for &lt;em&gt;similar&lt;/em&gt; inputs via embeddings, is riskier. The GPTCache documentation itself warns about false positives on cache hits [21]. Use it only where a near-duplicate answer is acceptable, and include the similarity threshold in your evals.&lt;/p&gt;

&lt;h3&gt;
  
  
  Route and cascade
&lt;/h3&gt;

&lt;p&gt;A router sends easy requests to a small model and hard ones to a large model. A cascade tries the small model first and escalates if a check fails. FrugalGPT reported matching the best individual model with up to 98% cost reduction using cascades and related strategies [22]. RouteLLM reported cost reductions of over 2x in some cases without lowering response quality, evaluated on MT Bench, MMLU and GSM8K [23]. Kapoor et al. cut variable cost by 53% for GPT-3.5 and 41% for Llama-3-70B on HotPotQA by optimising cost and accuracy jointly [16].&lt;/p&gt;

&lt;p&gt;Put the cascade into the formula before you ship it. An escalated request pays for the failed small-model attempt and adds its latency, so a cascade only wins if the small model resolves enough traffic on its own.&lt;/p&gt;

&lt;h3&gt;
  
  
  Distil or fine-tune for narrow tasks
&lt;/h3&gt;

&lt;p&gt;For a stable, high-volume task, a small specialised model can beat a large general one. In "Distilling Step-by-Step", a fine-tuned 770M-parameter T5 model outperformed few-shot prompted 540B PaLM using only 80% of the available data on a benchmark [24]. Distillation is a fixed cost that pays back with volume. In a different setting, Kapoor et al. found that a one-time optimisation cost was outweighed by inference savings after roughly 1,300 inferences [16]; compute the same payback for your own task.&lt;/p&gt;

&lt;h3&gt;
  
  
  Serving-engine levers if you self-host
&lt;/h3&gt;

&lt;p&gt;If you run open-weight models, the serving engine decides how much work each GPU hour does.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Continuous batching.&lt;/strong&gt; Orca introduced iteration-level scheduling, which adds and removes requests from the batch at each decoding step, and reported 36.9x throughput over FasterTransformer at the same latency for a GPT-3 175B model [25]. TensorRT-LLM's in-flight batching applies the same idea, processing context and generation phases together to keep GPUs busy [26].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paged KV cache.&lt;/strong&gt; PagedAttention targets KV cache memory that is wasted by fragmentation and redundant duplication, and vLLM reported 2x to 4x higher throughput than FasterTransformer and Orca at the same latency [27].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chunked prefill.&lt;/strong&gt; vLLM splits long prefills and schedules them alongside decodes. Lower &lt;code&gt;max_num_batched_tokens&lt;/code&gt; values (for example 2048) improve inter-token latency, while higher values improve TTFT and throughput [28]. When KV cache memory runs out, vLLM preempts requests and recomputes them later, which hurts end-to-end latency; the docs list remedies such as increasing &lt;code&gt;gpu_memory_utilization&lt;/code&gt; or lowering &lt;code&gt;max_num_seqs&lt;/code&gt; [28].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prefill/decode disaggregation.&lt;/strong&gt; DistServe places prefill and decode on different GPUs to remove interference between them, and reported 4.48x more requests or 10.2x tighter SLOs under TTFT and TPOT constraints [29]. TensorRT-LLM offers disaggregated serving as a beta feature [26].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Quantization.&lt;/strong&gt; vLLM's FP8 docs claim a 2x reduction in model memory and up to 1.6x throughput, with W8A8 officially supported on Hopper and Ada Lovelace GPUs [30]. Red Hat's large evaluation of quantized Llama 3.1 models found full accuracy recovery for FP8 across its tasks, close to 99% average recovery on OpenLLM Leaderboard v2, about 1.8x average speedup for 8-bit models in server workloads, and 2.4x average single-stream speedup for 4-bit weight-only models, which it recommends for latency-critical use [31]. AWQ, a low-bit weight-only method, protects about 1% of salient weights and reports more than 3x speedup over the Hugging Face FP16 implementation [32]. TensorRT-LLM supports FP8 on H100 and later and FP4 weights on B200 [26]. Always rerun your own task eval after quantizing; the Red Hat study saw more variability on 8B models than on 70B and 405B [31].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Speculative decoding.&lt;/strong&gt; A small draft model proposes tokens and the large model verifies them. The original method gave 2x to 3x acceleration on T5-XXL with identical outputs and no retraining [33]. EAGLE-3 reports speedup ratios up to 6.5x, but a 1.38x throughput gain in SGLang at batch size 64 [34]. NVIDIA measured 3.55x for Llama 3.3 70B with a 1B draft on one H200, at batch size 1 [35]. The pattern is consistent: speculative decoding is a latency lever that shrinks as batches grow.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Faster constrained decoding.&lt;/strong&gt; SGLang's compressed finite state machine decodes several tokens at once when the schema allows only one path, and reported up to 2x lower latency and 2.5x higher throughput for JSON decoding [36].&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Self-hosting versus API: the break-even
&lt;/h2&gt;

&lt;p&gt;The economics of self-hosting reduce to one variable most teams underestimate: utilisation.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cost per 1M output tokens = P_gpu_hour / (T * 3600 * u / 1,000,000)

break-even utilisation u* = P_gpu_hour / (T * 3600 / 1,000,000 * P_api_per_1M)

P_gpu_hour   price of the GPU (or node) per hour
T            sustained output tokens per second at your latency SLO
u            fraction of hours the GPU actually serves at that rate
P_api_per_1M managed API price per 1M output tokens for an equivalent model
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;T&lt;/code&gt; should be measured at the concurrency where your p95 latency still meets the SLO, not at peak throughput, because the throughput curve flattens while latency keeps rising [17].&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Illustrative calculation, with assumptions.&lt;/strong&gt; Scaleway lists a single H100 PCIe 80 GB instance in Paris at €2.52 per hour, rising to €2.73 per hour from 1 July [37]. For comparison, Lambda lists $4.59 per GPU hour for an HGX H100 effective 16 November 2026 [38]. Assume €2.73 per hour and an &lt;em&gt;assumed&lt;/em&gt; sustained throughput of 1,500 output tokens per second for an open-weight model at your SLO (measure this; it varies widely by model, quantization, prompt length and engine).&lt;/p&gt;

&lt;p&gt;At 100% utilisation that GPU produces 5.4 million output tokens per hour, or about €0.51 per million. At 30% utilisation the same GPU costs about €1.69 per million. A managed EU option, STACKIT AI Model Serving, lists gpt-oss-120b and Llama 3.3 70B Instruct FP8 at €0.65 per million output tokens [39]. Under these assumptions, u* is about 78%: the GPU must stay busy 78% of all hours just to match the managed price, before you pay for engineering time, on-call, monitoring, model updates and a second GPU for redundancy (which halves utilisation per GPU at the same traffic). Note also that this simple formula charges everything to output tokens; long prompts consume GPU time for prefill as well.&lt;/p&gt;

&lt;p&gt;Self-hosting is easiest to justify when traffic is high and steady, the model is small or fine-tuned for one task, or data rules exclude third-party APIs. For spiky or low-volume traffic, run the same formula against a managed endpoint plus batch tiers before buying GPU hours.&lt;/p&gt;

&lt;h2&gt;
  
  
  EU and GDPR-relevant hosting options
&lt;/h2&gt;

&lt;p&gt;Data residency is not GDPR compliance: you still need a data processing agreement, a lawful basis, retention rules and transfer assessments. Residency narrows where prompts are processed and stored, which simplifies reviews. Documented options include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;OpenAI API with European data residency.&lt;/strong&gt; Announced on 5 February 2025: eligible customers create a new Project with Europe selected as the region, requests are handled in-region with zero data retention, and existing Projects cannot be converted [40]. Regional processing carries a 10% uplift for models released on or after 5 March 2026 [2].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Azure OpenAI Data Zone deployments.&lt;/strong&gt; Data Zone EU processes data only within the Microsoft-defined zone, which follows the EU Data Boundary and can include EFTA countries such as Norway and Switzerland; data at rest stays in the designated Azure geography; Data Zone Batch gives 50% savings [12].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Amazon Bedrock geographic cross-region inference.&lt;/strong&gt; Requests to a geography-bound profile such as EU stay within that geography, but prompts and outputs can move between Regions inside it, and data stored for abuse detection is stored in the destination Region [41].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Anthropic's first-party API.&lt;/strong&gt; The pricing page documents &lt;code&gt;inference_geo&lt;/code&gt; with a US-only option at 1.1x and global routing as the default [3]. If you need EU-only processing for Claude models, check the cloud-provider routes above and confirm the details with your provider.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;European providers.&lt;/strong&gt; STACKIT AI Model Serving offers open-weight models in its EU01 region with euro token pricing, and states that data and queries are neither stored nor used to train models [39]. Scaleway rents H100 GPUs in a Paris data centre for self-hosting [37].&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Put the residency multiplier into &lt;code&gt;m_region&lt;/code&gt; in your cost formula, and keep your own logs residency-aware too: if your cost logger stores prompt text in a US-hosted observability tool, the API region no longer matters much.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist: a recipe for cost per finished task
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Define "finished" per task type.&lt;/strong&gt; Write down the validation rule (schema valid, business checks pass, human accepted) that turns an attempt into a success.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Log per task, not per call.&lt;/strong&gt; Use the fields above: tokens by category, reasoning tokens, calls, attempts, success, cost, TTFT and wall time. No prompt text in metrics.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compute the baseline.&lt;/strong&gt; Cost per finished task, success rate, p50 and p95 wall time, cache hit ratio and output-to-input token ratio for each task type.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bound the loop.&lt;/strong&gt; Set a task-level token and step budget in the orchestrator, because per-request &lt;code&gt;max_tokens&lt;/code&gt; does not bound an agent loop [10].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tune reasoning effort on your eval set.&lt;/strong&gt; Pick the lowest level that holds quality.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use strict structured outputs&lt;/strong&gt; wherever the result is parsed by code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fix prompt layout for caching.&lt;/strong&gt; Static first, dynamic last, stable tool sets and settings. Alert on cache hit ratio drops.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Move async work to batch or flex.&lt;/strong&gt; Anything that can wait minutes or hours.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Add exact-match response caching&lt;/strong&gt; for repeated inputs; use semantic caching only with measured false-hit rates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test a router or cascade&lt;/strong&gt; against the formula, including the cost and latency of escalations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Consider distillation&lt;/strong&gt; for stable, high-volume narrow tasks, and compute the payback volume.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Self-host only after the break-even math,&lt;/strong&gt; using throughput measured at your SLO and realistic utilisation, including redundancy and people.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Choose hosting by data rules first,&lt;/strong&gt; then add the residency multiplier to the cost model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Re-run the numbers monthly.&lt;/strong&gt; Prices and models change fast [1]; yesterday's cheapest route may not be today's.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The price per token is an input to your cost model, not the output. Measure the task, and the right model, tier and hosting choice usually becomes clear from your own data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;p&gt;All sources accessed 6 November 2026.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Epoch AI, LLM inference price trends (data insight). &lt;a href="https://epoch.ai/data-insights/llm-inference-price-trends" rel="noopener noreferrer"&gt;https://epoch.ai/data-insights/llm-inference-price-trends&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenAI, API Pricing. &lt;a href="https://developers.openai.com/api/docs/pricing" rel="noopener noreferrer"&gt;https://developers.openai.com/api/docs/pricing&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic, Claude API Pricing. &lt;a href="https://platform.claude.com/docs/en/about-claude/pricing" rel="noopener noreferrer"&gt;https://platform.claude.com/docs/en/about-claude/pricing&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Hugging Face blog, "Bringing the Artificial Analysis LLM Performance Leaderboard to Hugging Face". &lt;a href="https://huggingface.co/blog/leaderboard-artificial-analysis" rel="noopener noreferrer"&gt;https://huggingface.co/blog/leaderboard-artificial-analysis&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenAI Help Center, "What are tokens and how to count them?". &lt;a href="https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them" rel="noopener noreferrer"&gt;https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenAI, Prompt caching guide. &lt;a href="https://developers.openai.com/docs/guides/prompt-caching" rel="noopener noreferrer"&gt;https://developers.openai.com/docs/guides/prompt-caching&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic, Prompt caching documentation. &lt;a href="https://platform.claude.com/docs/en/build-with-claude/prompt-caching" rel="noopener noreferrer"&gt;https://platform.claude.com/docs/en/build-with-claude/prompt-caching&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Google, Gemini API optimization and inference (Batch, Flex, Priority, caching). &lt;a href="https://ai.google.dev/gemini-api/docs/optimization" rel="noopener noreferrer"&gt;https://ai.google.dev/gemini-api/docs/optimization&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic, Extended thinking documentation. &lt;a href="https://platform.claude.com/docs/build-with-claude/extended-thinking" rel="noopener noreferrer"&gt;https://platform.claude.com/docs/build-with-claude/extended-thinking&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic, Steering thinking and cost documentation. &lt;a href="https://platform.claude.com/docs/en/build-with-claude/thinking-steering-and-cost" rel="noopener noreferrer"&gt;https://platform.claude.com/docs/en/build-with-claude/thinking-steering-and-cost&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenAI, Reasoning models guide. &lt;a href="https://developers.openai.com/api/docs/guides/reasoning" rel="noopener noreferrer"&gt;https://developers.openai.com/api/docs/guides/reasoning&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Microsoft Learn, Azure OpenAI deployment types. &lt;a href="https://learn.microsoft.com/en-us/Azure/ai-foundry/openai/how-to/deployment-types" rel="noopener noreferrer"&gt;https://learn.microsoft.com/en-us/Azure/ai-foundry/openai/how-to/deployment-types&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic Engineering, "How we built our multi-agent research system" (2025). &lt;a href="https://www.anthropic.com/engineering/multi-agent-research-system" rel="noopener noreferrer"&gt;https://www.anthropic.com/engineering/multi-agent-research-system&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Chen et al., "Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs", arXiv:2412.21187. &lt;a href="https://arxiv.org/html/2412.21187v2" rel="noopener noreferrer"&gt;https://arxiv.org/html/2412.21187v2&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenAI, "Introducing Structured Outputs in the API". &lt;a href="https://openai.com/blog/introducing-structured-outputs-in-the-api" rel="noopener noreferrer"&gt;https://openai.com/blog/introducing-structured-outputs-in-the-api&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Kapoor et al., "AI Agents That Matter", arXiv:2407.01502. &lt;a href="https://arxiv.org/html/2407.01502v1" rel="noopener noreferrer"&gt;https://arxiv.org/html/2407.01502v1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;NVIDIA Technical Blog, "LLM Inference Benchmarking: Fundamental Concepts". &lt;a href="https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts" rel="noopener noreferrer"&gt;https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;vLLM documentation, Production Metrics. &lt;a href="https://docs.vllm.ai/en/stable/usage/metrics/" rel="noopener noreferrer"&gt;https://docs.vllm.ai/en/stable/usage/metrics/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;vLLM documentation, Automatic Prefix Caching. &lt;a href="https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/" rel="noopener noreferrer"&gt;https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs", NeurIPS 2024. &lt;a href="https://proceedings.neurips.cc/paper_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract.html" rel="noopener noreferrer"&gt;https://proceedings.neurips.cc/paper_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GPTCache documentation. &lt;a href="https://gpt-cache-test.readthedocs.io/en/latest/index.html" rel="noopener noreferrer"&gt;https://gpt-cache-test.readthedocs.io/en/latest/index.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Chen, Zaharia, Zou, "FrugalGPT", arXiv:2305.05176. &lt;a href="https://arxiv.org/abs/2305.05176" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2305.05176&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Ong et al., "RouteLLM: Learning to Route LLMs with Preference Data", arXiv:2406.18665. &lt;a href="https://arxiv.org/html/2406.18665v3" rel="noopener noreferrer"&gt;https://arxiv.org/html/2406.18665v3&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Hsieh et al., "Distilling Step-by-Step!", arXiv:2305.02301. &lt;a href="https://arxiv.org/html/2305.02301v2" rel="noopener noreferrer"&gt;https://arxiv.org/html/2305.02301v2&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI 2022. &lt;a href="https://usenix.org/conference/osdi22/presentation/yu" rel="noopener noreferrer"&gt;https://usenix.org/conference/osdi22/presentation/yu&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;NVIDIA, TensorRT-LLM Overview. &lt;a href="https://nvidia.github.io/TensorRT-LLM/overview.html" rel="noopener noreferrer"&gt;https://nvidia.github.io/TensorRT-LLM/overview.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", arXiv:2309.06180. &lt;a href="https://huggingface.co/papers/2309.06180" rel="noopener noreferrer"&gt;https://huggingface.co/papers/2309.06180&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;vLLM documentation, Optimization and Tuning. &lt;a href="https://docs.vllm.ai/en/latest/configuration/optimization/" rel="noopener noreferrer"&gt;https://docs.vllm.ai/en/latest/configuration/optimization/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving", arXiv:2401.09670. &lt;a href="https://huggingface.co/papers/2401.09670" rel="noopener noreferrer"&gt;https://huggingface.co/papers/2401.09670&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;vLLM documentation, FP8 W8A8 quantization. &lt;a href="https://docs.vllm.ai/en/v0.8.4/features/quantization/fp8.html" rel="noopener noreferrer"&gt;https://docs.vllm.ai/en/v0.8.4/features/quantization/fp8.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Red Hat Developer, "We ran over half a million evaluations on quantized LLMs". &lt;a href="https://developers.redhat.com/articles/2024/10/17/we-ran-over-half-million-evaluations-quantized-llms" rel="noopener noreferrer"&gt;https://developers.redhat.com/articles/2024/10/17/we-ran-over-half-million-evaluations-quantized-llms&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;MIT HAN Lab, AWQ: Activation-aware Weight Quantization. &lt;a href="https://hanlab.mit.edu/projects/awq" rel="noopener noreferrer"&gt;https://hanlab.mit.edu/projects/awq&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Leviathan, Kalman, Matias, "Fast Inference from Transformers via Speculative Decoding", ICML 2023. &lt;a href="https://proceedings.mlr.press/v202/leviathan23a.html" rel="noopener noreferrer"&gt;https://proceedings.mlr.press/v202/leviathan23a.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Li et al., "EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test", NeurIPS 2025. &lt;a href="https://neurips.cc/virtual/2025/poster/119930" rel="noopener noreferrer"&gt;https://neurips.cc/virtual/2025/poster/119930&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;NVIDIA Technical Blog, "Boost Llama 3.3 70B Inference Throughput 3x with NVIDIA TensorRT-LLM Speculative Decoding". &lt;a href="https://developer.nvidia.com/blog/boost-llama-3-3-70b-inference-throughput-3x-with-nvidia-tensorrt-llm-speculative-decoding/" rel="noopener noreferrer"&gt;https://developer.nvidia.com/blog/boost-llama-3-3-70b-inference-throughput-3x-with-nvidia-tensorrt-llm-speculative-decoding/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LMSYS blog, "Fast JSON Decoding for Local LLMs with Compressed Finite State Machine". &lt;a href="https://www.lmsys.org/blog/2024-02-05-compressed-fsm" rel="noopener noreferrer"&gt;https://www.lmsys.org/blog/2024-02-05-compressed-fsm&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Scaleway, H100 PCIe GPU instances. &lt;a href="https://scaleway.com/en/h100-pcie-try-it-now" rel="noopener noreferrer"&gt;https://scaleway.com/en/h100-pcie-try-it-now&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Lambda, GPU Cloud Pricing. &lt;a href="https://lambda.com/pricing" rel="noopener noreferrer"&gt;https://lambda.com/pricing&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;STACKIT, AI Model Serving. &lt;a href="https://stackit.com/en/products/data-ai/stackit-ai-model-serving" rel="noopener noreferrer"&gt;https://stackit.com/en/products/data-ai/stackit-ai-model-serving&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenAI, "Introducing data residency in Europe" (5 Feb 2025). &lt;a href="https://openai.com/blog/introducing-data-residency-in-europe" rel="noopener noreferrer"&gt;https://openai.com/blog/introducing-data-residency-in-europe&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;AWS, Amazon Bedrock geographic cross-Region inference. &lt;a href="https://docs.aws.amazon.com/bedrock/latest/userguide/geographic-cross-region-inference.html" rel="noopener noreferrer"&gt;https://docs.aws.amazon.com/bedrock/latest/userguide/geographic-cross-region-inference.html&lt;/a&gt;
&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>llm</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>devops</category>
    </item>
    <item>
      <title>How to Evaluate RAG Retrieval in 2026: Question Sets, Metrics That Mislead, and a Practical Recipe</title>
      <dc:creator>Muhammad Hussnain Jamil</dc:creator>
      <pubDate>Fri, 06 Nov 2026 05:41:05 +0000</pubDate>
      <link>https://dev.to/hussnainoffical/how-to-evaluate-rag-retrieval-in-2026-question-sets-metrics-that-mislead-and-a-practical-recipe-2nh8</link>
      <guid>https://dev.to/hussnainoffical/how-to-evaluate-rag-retrieval-in-2026-question-sets-metrics-that-mislead-and-a-practical-recipe-2nh8</guid>
      <description>&lt;p&gt;Many RAG failures get blamed on "the LLM" when the generator never had a chance: the passage it needed was not in the context, or it was buried under near-duplicates. Retrieval deserves its own evaluation. It is also easy to get wrong, because the score depends on what you call "relevant", how you built the questions, which k you chose and how many questions you asked.&lt;/p&gt;

&lt;p&gt;This article covers the whole chain as it stands in late 2026: labelled question sets (including synthetic ones and their traps), the metrics and where each misleads, statistical noise, lexical versus paraphrased questions, evaluating chunking, hybrid search and rerankers, LLM-as-judge and its documented biases, end-to-end metrics, current tools and benchmarks, and a recipe you can apply this week.&lt;/p&gt;

&lt;p&gt;Numbers of my own come from small, reproducible experiments in my public lab repo, &lt;a href="https://github.com/Hussnainoffical/applied-llm-lab" rel="noopener noreferrer"&gt;applied-llm-lab&lt;/a&gt;. They show how metrics behave; they are not a benchmark.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why evaluate retrieval on its own
&lt;/h2&gt;

&lt;p&gt;Retrieval quality changes answer quality in ways the generator cannot fix. Google researchers defined &lt;em&gt;sufficient context&lt;/em&gt; as context that "contains all the necessary information to provide a definitive answer" and found that insufficient context can make models &lt;em&gt;less&lt;/em&gt; likely to abstain: for one model the incorrect-answer rate rose from 10.2% with no context to 66.1% with insufficient context [2]. A weak retriever can actively produce confident wrong answers.&lt;/p&gt;

&lt;p&gt;Serious RAG work measures the retriever directly. Anthropic's Contextual Retrieval write-up used "1 minus recall@20" and reported failure rates falling from 5.7% to 3.7% with contextual embeddings, to 2.9% with contextual BM25 added, and to 1.9% with reranking [1], all measured before any answer was generated.&lt;/p&gt;

&lt;p&gt;Retrieval-only metrics are also cheaper, deterministic once labels are fixed, and they localise problems: if recall@20 is fine but answers are wrong, look at the prompt or generator; if recall@20 is poor, no prompt will save you.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step zero: decide what "relevant" means
&lt;/h2&gt;

&lt;p&gt;Every retrieval metric compares a ranked list against a set of relevant items. Before choosing a metric, decide what the unit of relevance is:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Unit&lt;/th&gt;
&lt;th&gt;Example label&lt;/th&gt;
&lt;th&gt;Good for&lt;/th&gt;
&lt;th&gt;Weakness&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Document ID&lt;/td&gt;
&lt;td&gt;&lt;code&gt;doc_17&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Classic IR benchmarks, small docs&lt;/td&gt;
&lt;td&gt;Says nothing about chunking&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chunk ID&lt;/td&gt;
&lt;td&gt;&lt;code&gt;doc_17#chunk_3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Fast regression tests on one fixed index&lt;/td&gt;
&lt;td&gt;Labels break when you re-chunk&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Text span / fact&lt;/td&gt;
&lt;td&gt;"Each chunk keeps the file name and page number as metadata"&lt;/td&gt;
&lt;td&gt;Comparing chunkers, rerankers, hybrid setups&lt;/td&gt;
&lt;td&gt;Needs span matching logic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claims / nuggets&lt;/td&gt;
&lt;td&gt;List of atomic facts from a reference answer&lt;/td&gt;
&lt;td&gt;Multi-fact questions, end-to-end analysis&lt;/td&gt;
&lt;td&gt;Needs an LLM or human to check entailment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tokens&lt;/td&gt;
&lt;td&gt;Exact character ranges of relevant text&lt;/td&gt;
&lt;td&gt;Measuring chunk efficiency&lt;/td&gt;
&lt;td&gt;More engineering; unusual in tools&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Chroma's chunking study notes that document-level benchmarks like MTEB and BEIR "cannot take chunking into account" and measures token-wise precision, recall and intersection over union instead [18]. RAGChecker works at claim level: claim recall is the share of ground-truth claims entailed by the retrieved chunks [19].&lt;/p&gt;

&lt;p&gt;In my lab a chunk counts as the answer chunk only if it contains the &lt;em&gt;whole&lt;/em&gt; answer fact. With fixed-size chunks some facts are cut in half, so for those questions &lt;em&gt;no&lt;/em&gt; chunk can be relevant and the miss is guaranteed. Relevance definition and chunker interact, and the metric reflects both.&lt;/p&gt;

&lt;h2&gt;
  
  
  Building the labelled question set
&lt;/h2&gt;

&lt;p&gt;Questions come from three sources, in decreasing order of realism: real user queries from logs or tickets, subject-matter experts, and LLM-generated (synthetic) questions.&lt;/p&gt;

&lt;p&gt;A useful record holds the question, the relevant span(s) or claims, optional graded relevance, a short reference answer, and tags such as &lt;code&gt;lexical&lt;/code&gt; / &lt;code&gt;paraphrased&lt;/code&gt;, &lt;code&gt;single-hop&lt;/code&gt; / &lt;code&gt;multi-hop&lt;/code&gt;, &lt;code&gt;has-identifier&lt;/code&gt; and &lt;code&gt;unanswerable&lt;/code&gt;. Tags let you see that a change helped one slice and hurt another. Label spans or claims rather than chunk IDs if you will ever compare chunkers: chunk-ID labels silently become wrong when you re-chunk.&lt;/p&gt;

&lt;h3&gt;
  
  
  Synthetic question sets: useful, but biased in known ways
&lt;/h3&gt;

&lt;p&gt;Generation is now a standard feature. Ragas builds a knowledge graph from your documents, then synthesizers produce single-hop or multi-hop, specific or abstract queries with configurable length and style [9]. Langfuse documents routes via Ragas, DeepEval's synthesizer, or a custom loop writing one question per chunk [17]. Chroma's chunking study shows careful filtering: queries whose evidence excerpts were not exact matches in the corpus were rejected, and near-duplicate or weakly related queries were removed with embedding thresholds [18].&lt;/p&gt;

&lt;p&gt;The research on trusting synthetic sets is nuanced:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Synthetic queries do not look like real ones.&lt;/strong&gt; A September 2026 study found authentic queries averaged 6.8 words versus 15.7 for synthetic ones, real users concentrated on a few topics while synthetic questions spread evenly, and optimising on synthetic queries picked a hybrid retriever costing up to 8 times the latency of the fastest configuration, whose sparse half did not help short real queries [29].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Synthetic collections inflate absolute scores and favour their own model family.&lt;/strong&gt; Rahmani et al. found GPT-4 more lenient than human assessors on borderline passages, scores generally higher on synthetic data, and systems from the generator's model family favoured; the bias matters most for absolute scores and less for relative comparisons [27]. Their earlier SIGIR 2024 paper showed synthetic test collections &lt;em&gt;can&lt;/em&gt; be used reliably for retrieval evaluation [26].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Synthetic sets rank retrievers better than generators.&lt;/strong&gt; Van Elburg et al. report that synthetic benchmarks "reliably rank the RAGs varying in terms of retriever configuration" but "do not consistently produce reliable RAG rankings when comparing generator architectures" [28].&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two more traps follow from how per-chunk generation works. A question written while looking at one chunk tends to reuse its vocabulary, which flatters lexical retrievers (measure this on your own set; see below). And it is answerable from exactly one chunk by construction, so multi-chunk and unanswerable questions are under-represented.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Practical stance:&lt;/strong&gt; use synthetic questions for coverage and for comparing retriever configurations, keep a smaller human-written or log-derived set as the arbiter, and report them separately. If they disagree about the winner, believe the real one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The metrics, precisely
&lt;/h2&gt;

&lt;p&gt;Here are the definitions most tools use. Let &lt;code&gt;rank&lt;/code&gt; be the 1-based position of a relevant item, &lt;code&gt;k&lt;/code&gt; the cutoff, and &lt;code&gt;R&lt;/code&gt; the set of relevant items for one question. Every metric is averaged over questions.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Per-question definition&lt;/th&gt;
&lt;th&gt;Answers the question&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Hit rate@k (success@k)&lt;/td&gt;
&lt;td&gt;1 if any relevant item is in the top k, else 0&lt;/td&gt;
&lt;td&gt;"Did we find anything useful?"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recall@k&lt;/td&gt;
&lt;td&gt;relevant items in top k / |R|&lt;/td&gt;
&lt;td&gt;"How much of what is needed did we find?"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Precision@k&lt;/td&gt;
&lt;td&gt;relevant items in top k / k&lt;/td&gt;
&lt;td&gt;"How much of the context is useful?"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MRR@k&lt;/td&gt;
&lt;td&gt;1 / rank of the first relevant item (0 if none in top k)&lt;/td&gt;
&lt;td&gt;"How high is the first useful item?"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;nDCG@k&lt;/td&gt;
&lt;td&gt;DCG@k / ideal DCG@k, with DCG = Σ gain / log2(rank + 1)&lt;/td&gt;
&lt;td&gt;"Are the most relevant items near the top?" (supports graded relevance)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context recall (LLM)&lt;/td&gt;
&lt;td&gt;reference-answer claims supported by the retrieved context / all claims&lt;/td&gt;
&lt;td&gt;Recall without labelled chunks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context precision (LLM)&lt;/td&gt;
&lt;td&gt;rank-weighted precision over chunks judged relevant&lt;/td&gt;
&lt;td&gt;Ranking quality without labelled chunks&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Notes on origins and variants:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;MRR&lt;/strong&gt; comes from the TREC-8 QA track, where a question scored "the reciprocal of the rank at which the first correct response was returned," or 0 if none of the top five was correct [3].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;nDCG&lt;/strong&gt; was introduced by Järvelin and Kekäläinen for graded relevance with a rank discount [4]. With one binary relevant item it reduces to 1 / log2(rank + 1): 1.00 at rank 1, 0.63 at rank 2, 0.50 at rank 3. Haystack's nDCG evaluator uses ground-truth scores as graded relevance when present and binary relevance otherwise [15].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Recall" is overloaded.&lt;/strong&gt; Haystack's &lt;code&gt;DocumentRecallEvaluator&lt;/code&gt; has &lt;code&gt;single_hit&lt;/code&gt; (score 1 if &lt;em&gt;any&lt;/em&gt; ground-truth document is retrieved, i.e. hit rate) and &lt;code&gt;multi_hit&lt;/code&gt; (full score only if &lt;em&gt;all&lt;/em&gt; are) [14]. Say which one you mean.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Context precision is not one formula.&lt;/strong&gt; In Ragas, the LLM-based version averages precision@k over the positions of chunks judged relevant, so ranking matters; the ID-based version is plain precision without rank weighting [5]. DeepEval's contextual precision also weights by position, judges against &lt;code&gt;expected_output&lt;/code&gt;, and warns that scores from different evaluation modes are not comparable [10].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Context recall&lt;/strong&gt; (Ragas, DeepEval) splits the reference answer into claims and checks which can be attributed to the retrieved context [6][11]; Ragas also has non-LLM and ID-based variants. DeepEval's contextual &lt;em&gt;relevancy&lt;/em&gt; is reference-free: the share of statements in the retrieved context relevant to the input [12].&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  When each metric misleads
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Hit rate@k with a large k hides ranking.&lt;/strong&gt; Hit@20 can stay flat while the relevant chunk slides from rank 1 to rank 15. That matters if the generator sees only the top 5, or handles long contexts unevenly: "Lost in the Middle" found performance "is often highest when relevant information occurs at the beginning or end of the input context" [38]. Chroma, by contrast, argues LLMs are relatively insensitive to position and down-weights nDCG-style metrics for RAG [18]. Both can hold for different models and context lengths, so report a rank-sensitive metric next to hit rate instead of picking one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;MRR ignores everything after the first relevant item.&lt;/strong&gt; For a question needing three facts from three chunks, MRR rewards one at rank 1 and is blind to the other two. Use recall@k or claim-level context recall for multi-fact questions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Precision@k is capped by the number of relevant items.&lt;/strong&gt; With one relevant chunk, the best precision@5 is 0.20. Low precision matters when the extra chunks are distractors that pull the generator off course [2].&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Incomplete labels punish novel retrievers.&lt;/strong&gt; Unlabelled items count as irrelevant. BEIR found many datasets "subject to a lexical bias" from pooling with lexical systems; after missing TREC-COVID judgments were annotated, a dense retriever's nDCG@10 rose from 0.654 to 0.735, moving above BM25 [20]. Labels made by looking at BM25 results make new embedding models look worse.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Chunk-level metrics are not comparable across chunkers.&lt;/strong&gt; Larger chunks contain more facts, so they win hit rate almost by definition while wasting context. Chroma's token-level IoU exposes this: a recursive splitter at 800 tokens with 400 overlap (the chunking settings OpenAI Assistants used by default at the time of the study) scored an IoU of 1.5, against 6.9 for the same splitter at 200 tokens with no overlap, and reducing overlap improved IoU because overlap adds redundant tokens [18].&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The same metric name gives different numbers in different tools.&lt;/strong&gt; Rank-weighted versus unweighted context precision, &lt;code&gt;single_hit&lt;/code&gt; versus &lt;code&gt;multi_hit&lt;/code&gt; recall, LLM-judged versus ID-based: pin the tool, version and variant in your eval report.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the metrics behave on a toy set: my lab results
&lt;/h2&gt;

&lt;p&gt;The experiments in &lt;a href="https://github.com/Hussnainoffical/applied-llm-lab" rel="noopener noreferrer"&gt;applied-llm-lab&lt;/a&gt; build a tiny pipeline from scratch:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Day 01, chunking:&lt;/strong&gt; fixed-size chunks cut sentences in half; sentence packing kept every sentence whole at a similar chunk size.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Day 02, BM25 from scratch:&lt;/strong&gt; on 5 questions, sentence-packed chunks put the whole fact in BM25's top result 5/5 times, versus 2/5 for fixed-size chunks (4/5 with overlap).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Day 03, TF-IDF cosine with NumPy:&lt;/strong&gt; the same hit counts as BM25 (5/5 with sentence chunks). On 3 paraphrased questions both scored 0/3, so paraphrases need learned embeddings.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Day 04 added metrics: a 6-paragraph corpus about RAG pipelines and 12 questions, each asked in the document's own words ("original") and mostly in other words ("paraphrased"). The answer chunk is the one containing the whole fact. The harness core, in the same style:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;first_relevant_rank&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ranked_chunks&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fact&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;1-based rank of the first top-k chunk that holds the whole fact, or None.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;rank&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ranked_chunks&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;fact&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;()):&lt;/span&gt;  &lt;span class="c1"&gt;# normalise whitespace before matching
&lt;/span&gt;            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;rank&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;summarize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ranks&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Average hit@1, hit@3 and reciprocal rank (MRR@5) over all questions.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ranks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hit@1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ranks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hit@3&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ranks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mrr@5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ranks&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;


&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;summarize&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hit@1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hit@3&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mrr@5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;BM25 and TF-IDF cosine gave identical numbers in every row, so the table shows one value per row:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Chunking&lt;/th&gt;
&lt;th&gt;Questions&lt;/th&gt;
&lt;th&gt;hit@1&lt;/th&gt;
&lt;th&gt;hit@3&lt;/th&gt;
&lt;th&gt;MRR@5&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fixed size, 120 chars&lt;/td&gt;
&lt;td&gt;original&lt;/td&gt;
&lt;td&gt;0.67&lt;/td&gt;
&lt;td&gt;0.67&lt;/td&gt;
&lt;td&gt;0.67&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fixed size, 120 chars&lt;/td&gt;
&lt;td&gt;paraphrased&lt;/td&gt;
&lt;td&gt;0.50&lt;/td&gt;
&lt;td&gt;0.58&lt;/td&gt;
&lt;td&gt;0.53&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fixed 120 + 30 overlap&lt;/td&gt;
&lt;td&gt;original&lt;/td&gt;
&lt;td&gt;0.83&lt;/td&gt;
&lt;td&gt;0.83&lt;/td&gt;
&lt;td&gt;0.83&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fixed 120 + 30 overlap&lt;/td&gt;
&lt;td&gt;paraphrased&lt;/td&gt;
&lt;td&gt;0.42&lt;/td&gt;
&lt;td&gt;0.67&lt;/td&gt;
&lt;td&gt;0.54&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sentence packing, max 160&lt;/td&gt;
&lt;td&gt;original&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sentence packing, max 160&lt;/td&gt;
&lt;td&gt;paraphrased&lt;/td&gt;
&lt;td&gt;0.83&lt;/td&gt;
&lt;td&gt;0.83&lt;/td&gt;
&lt;td&gt;0.83&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;What the toy set shows about the metrics:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Hit@3 alone would call overlap a clear win on paraphrases&lt;/strong&gt; (0.58 to 0.67). But hit@1 dropped from 0.50 to 0.42 and MRR barely moved (0.53 to 0.54). In each of the three rank-2 cases a neighbouring overlap chunk ranked first, and in two of them it held only part of the fact. Overlap creates near-duplicates that compete with the real answer chunk: "hit rate hides ranking" in miniature.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paraphrasing costs every configuration.&lt;/strong&gt; The best chunker still lost 2 of 12. Per-question ranks from rerunning the script show both misses hinge on an abbreviation the text spells differently: "retrieval-augmented generation" versus "RAG", and "RRF" versus "reciprocal rank fusion". No shared terms, no lexical match.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two lexical scorers with different maths ranked identically.&lt;/strong&gt; On a tiny corpus with short chunks, the differences between BM25 and TF-IDF cosine do not show up.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Then the most important caveat.&lt;/p&gt;

&lt;h2&gt;
  
  
  Statistical noise: one question in 12 is 0.08
&lt;/h2&gt;

&lt;p&gt;With 12 questions, each question moves a score by 1/12 = 0.083. The gap between 0.42 and 0.50 hit@1 above is &lt;em&gt;one question&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Confidence intervals make this concrete. Using the Wilson score interval for a proportion:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Hits / questions&lt;/th&gt;
&lt;th&gt;Point estimate&lt;/th&gt;
&lt;th&gt;95% Wilson interval&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;8 / 12&lt;/td&gt;
&lt;td&gt;0.67&lt;/td&gt;
&lt;td&gt;0.39 to 0.86&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10 / 12&lt;/td&gt;
&lt;td&gt;0.83&lt;/td&gt;
&lt;td&gt;0.55 to 0.95&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12 / 12&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;td&gt;0.76 to 1.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;83 / 100&lt;/td&gt;
&lt;td&gt;0.83&lt;/td&gt;
&lt;td&gt;0.74 to 0.89&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;250 / 300&lt;/td&gt;
&lt;td&gt;0.83&lt;/td&gt;
&lt;td&gt;0.79 to 0.87&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The intervals for 8/12 and 12/12 overlap. Even the most dramatic result in my table, sentence packing beating fixed-size chunks 12/12 versus 8/12 on original questions, is not statistically convincing on its own: the four questions fixed-size missed were all hit by sentence packing, and none went the other way, which gives an exact two-sided McNemar (sign) test p-value of 2 × 0.5⁴ = 0.125.&lt;/p&gt;

&lt;p&gt;Three habits fix most of this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Use paired tests on the same questions.&lt;/strong&gt; Smucker, Allan and Carterette found "little practical difference between the randomization, bootstrap, and t tests" for IR, preferred the randomization test, and argued Wilcoxon and sign tests should be discontinued for differences between means [39]. Miller's paper on error bars for LLM evals recommends inference on question-level paired differences, clustered standard errors when questions come in related groups (say, several generated from one document), and power analysis [40].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Size the set for the difference you care about.&lt;/strong&gt; The 95% half-width for a proportion near 0.8 is roughly 1.96 × √(0.16 / n): about ±0.08 at n = 100 and ±0.04 at n = 400. A few dozen questions cannot detect a 3-point improvement.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Report counts, not only ratios.&lt;/strong&gt; "10/12" tells the reader what "0.83" hides.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A paired randomization test needs only the standard library:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;paired_randomization_test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;trials&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Two-sided p-value for mean(a) - mean(b), flipping signs of per-question differences.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;diffs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
    &lt;span class="n"&gt;observed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;diffs&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;rng&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Random&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;extreme&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;random&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;diffs&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;observed&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;trials&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;extreme&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;trials&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run on the per-question hit@1 scores of the sentence-packing and fixed-size rows above, it returns about 0.12, matching the exact test. If you prefer a library, &lt;code&gt;ranx&lt;/code&gt; provides paired Student's t-test, Fisher's randomization test and Tukey's HSD alongside its metrics [42].&lt;/p&gt;

&lt;h2&gt;
  
  
  Lexical versus paraphrased questions
&lt;/h2&gt;

&lt;p&gt;Paraphrases test the gap between how documents are written and how users ask. Lexical retrievers win on exact identifiers; BEIR found BM25 "a robust baseline" that dense models often failed to beat zero-shot [20]. Dense retrievers handle different wording but have their own limits: Weller et al. prove the number of top-k subsets a single-vector embedding can return is bounded by its dimension, and built LIMIT, on which state-of-the-art embedders fail despite very simple queries [25]. Reasoning-heavy queries break both: on BRIGHT's 1,384 real queries, a model scoring 59.0 nDCG@10 on BEIR scored 18.3, and explicit reasoning about the query added up to 12.2 points [23].&lt;/p&gt;

&lt;p&gt;For your own set:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Stratify&lt;/strong&gt; into lexical (shares key terms with the answer span), paraphrased, identifier lookups and multi-hop or reasoning questions, and report every metric per stratum.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure lexical overlap automatically&lt;/strong&gt; (share of question tokens found in the answer span) and compare the synthetic and real splits. A high-overlap set makes BM25 look better than it will be in production.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write paraphrases deliberately&lt;/strong&gt;, including abbreviations, synonyms and user jargon. My two universal misses were both abbreviation mismatches.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Evaluating chunking, hybrid search and rerankers
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Chunking.&lt;/strong&gt; Use span or claim labels so one question set fits every chunker. Report hit@k with a rank-sensitive metric and a cost measure (tokens sent to the generator, or token-level IoU [18]); otherwise bigger chunks always win. If you can, fix the context budget in &lt;em&gt;tokens&lt;/em&gt;, not in number of chunks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Hybrid search.&lt;/strong&gt; Reciprocal rank fusion scores each document as Σ 1 / (k + rank) over the input rankings, with k = 60 fixed in a pilot study where "the choice was not critical" [41]. Fusion helps only if the inputs make &lt;em&gt;different&lt;/em&gt; mistakes; my BM25 and TF-IDF ranked identically, so fusing them would add nothing. On FreshStack, a 2025 benchmark built from Stack Overflow questions and technical documentation, fusing four retrievers beat each one on most metrics and topics [24]. Report latency next to quality: the sparse half of a hybrid can help long synthetic questions but not short real ones, at several times the latency [29].&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rerankers.&lt;/strong&gt; Evaluate the stages separately: the first stage needs high recall at the candidate depth you rerank (Anthropic reranked 150 candidates down to 20 [1]), the reranker needs good nDCG@k or MRR@k at the depth the generator sees. Rerankers do not always help: on FreshStack one improved some topics and lowered scores for some dense and fusion runs on others [24].&lt;/p&gt;

&lt;h2&gt;
  
  
  LLM-as-judge for retrieval
&lt;/h2&gt;

&lt;p&gt;Labelling relevance by hand does not scale, so LLM judges are now everywhere: they power LLM-based context precision and recall, and they label pools for benchmarks.&lt;/p&gt;

&lt;p&gt;The case for them is real. Thomas et al. found LLMs as accurate as human labellers in their experiments, and LLM labelling in production at Bing proved "not just more efficient but more accurate than the status quo" [30]. UMBRELA reproduced this with GPT-4o on TREC Deep Learning 2019 to 2023 and supported assessments in the TREC 2024 RAG track [31]. The biases are real too:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bias&lt;/th&gt;
&lt;th&gt;Evidence&lt;/th&gt;
&lt;th&gt;Mitigation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Leniency / overrating&lt;/td&gt;
&lt;td&gt;Consistent overrating, often with high confidence, sensitive to passage length and lexical cues [34]; GPT-4 more lenient than humans on borderline passages [27]&lt;/td&gt;
&lt;td&gt;Calibrate on a human-labelled subset; strict, aspect-based prompts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Keyword stuffing&lt;/td&gt;
&lt;td&gt;GPT-4 with a basic prompt labelled about 26% of random nonsense passages perfectly relevant once the query string was inserted [32]&lt;/td&gt;
&lt;td&gt;Test with adversarial passages first&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Instruction injection&lt;/td&gt;
&lt;td&gt;A planted sentence claiming the passage holds the answer fooled several models [32]&lt;/td&gt;
&lt;td&gt;Treat retrieved text as untrusted; delimit it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Self-preference, circularity&lt;/td&gt;
&lt;td&gt;Generator's model family favoured [27]; using the judging LLM as a reranker distorts rankings [33]&lt;/td&gt;
&lt;td&gt;Judge from a different model family&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Position, verbosity&lt;/td&gt;
&lt;td&gt;General LLM-judge biases [35]&lt;/td&gt;
&lt;td&gt;Randomise order; judge one chunk at a time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Prompt sensitivity&lt;/td&gt;
&lt;td&gt;Quality "varies unpredictably with simple paraphrases" [30]&lt;/td&gt;
&lt;td&gt;Freeze and version prompt and model&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Alaofi et al. also found that agreement scores such as Cohen's kappa did not reliably predict how gullible a judge was [32], so "kappa looks fine" is not enough. Clarke and Dietz argue LLM-based assessment still cannot replace human assessment, especially for benchmarks meant to guide future research [33].&lt;/p&gt;

&lt;p&gt;A practical middle ground is what ARES formalised: an automatic judge at scale, corrected by a small human-labelled set through prediction-powered inference, using "only a few hundred human annotations" [36].&lt;/p&gt;

&lt;h2&gt;
  
  
  Retrieval-only versus end-to-end metrics
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Faithfulness&lt;/strong&gt; (Ragas): share of answer claims inferable from the retrieved context [7]. No reference needed, but it measures grounding, not correctness: a faithful answer from the wrong passage is still wrong.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Answer correctness&lt;/strong&gt; (Ragas): an F1-style claim overlap with the reference answer, combined with semantic similarity [8].&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nugget evaluation&lt;/strong&gt; (TREC 2024 RAG track): answers scored by the atomic facts they cover. Automated nugget evaluation agrees strongly with human-based variants at run level, but per-topic agreement, which you need to diagnose individual failures, is not yet established [37].&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Read both kinds together:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Retrieval&lt;/th&gt;
&lt;th&gt;Answer&lt;/th&gt;
&lt;th&gt;Likely cause&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Good&lt;/td&gt;
&lt;td&gt;Good&lt;/td&gt;
&lt;td&gt;Working as intended&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Good&lt;/td&gt;
&lt;td&gt;Bad&lt;/td&gt;
&lt;td&gt;Generator, prompt, context ordering or context length&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bad&lt;/td&gt;
&lt;td&gt;Good&lt;/td&gt;
&lt;td&gt;Model answered from parametric knowledge; fragile, and not grounded&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bad&lt;/td&gt;
&lt;td&gt;Bad&lt;/td&gt;
&lt;td&gt;Fix retrieval first&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The "bad retrieval, good answer" cell is why end-to-end metrics alone are not enough: on public-knowledge questions the generator can mask a broken retriever, which stops working the moment questions concern your private documents.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tools and benchmarks in late 2026
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool / benchmark&lt;/th&gt;
&lt;th&gt;What it gives you for retrieval&lt;/th&gt;
&lt;th&gt;Watch out for&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ragas&lt;/td&gt;
&lt;td&gt;LLM-based, non-LLM and ID-based context precision and recall; faithfulness; testset generation [5][6][9]&lt;/td&gt;
&lt;td&gt;Legacy metric API slated for deprecation in 0.4 and removal in 1.0 [5]; pin versions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepEval&lt;/td&gt;
&lt;td&gt;Contextual precision, recall and relevancy as test-case metrics [10][11][12]&lt;/td&gt;
&lt;td&gt;Scores from different evaluation modes are not comparable [10]&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Langfuse&lt;/td&gt;
&lt;td&gt;Datasets, experiments, scores, LLM-as-judge evaluators, annotation queues for human labels; Ragas integration [16][17]&lt;/td&gt;
&lt;td&gt;It is the platform; metric definitions come from what you plug in&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Haystack&lt;/td&gt;
&lt;td&gt;Statistical MRR, MAP, nDCG and recall evaluators plus LLM-based ones [13][14][15]&lt;/td&gt;
&lt;td&gt;Check &lt;code&gt;single_hit&lt;/code&gt; versus &lt;code&gt;multi_hit&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ranx&lt;/td&gt;
&lt;td&gt;Fast IR metrics, paired significance tests, 20+ fusion methods including RRF [42]&lt;/td&gt;
&lt;td&gt;Needs run/qrels formatted data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BEIR&lt;/td&gt;
&lt;td&gt;18 datasets, 9 tasks, zero-shot retrieval [20]&lt;/td&gt;
&lt;td&gt;Lexical bias in judgments&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MMTEB / MTEB&lt;/td&gt;
&lt;td&gt;500+ tasks across 250+ languages; MTEB(eng, v2) designed as zero-shot [21]&lt;/td&gt;
&lt;td&gt;Leaderboard rank is not your corpus&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RTEB&lt;/td&gt;
&lt;td&gt;Retrieval-focused, open plus private datasets to expose overfitting; NDCG@10 default; 20 languages, enterprise domains [22]&lt;/td&gt;
&lt;td&gt;Beta&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BRIGHT&lt;/td&gt;
&lt;td&gt;Reasoning-intensive retrieval [23]&lt;/td&gt;
&lt;td&gt;Very different from FAQ-style RAG&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;FreshStack&lt;/td&gt;
&lt;td&gt;Technical-docs retrieval with nugget-based α-nDCG@10, coverage@20, recall@50 [24]&lt;/td&gt;
&lt;td&gt;Five topics&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LIMIT&lt;/td&gt;
&lt;td&gt;Stress test for single-vector embedding limits [25]&lt;/td&gt;
&lt;td&gt;Deliberately simple queries; a stress test, not a target&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Public benchmarks are for shortlisting embedding models and rerankers. RTEB exists precisely because public scores can overstate real-world performance through train-test overlap [22]. The decision should come from your own labelled set.&lt;/p&gt;

&lt;h2&gt;
  
  
  A practical evaluation recipe
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Define relevance as spans or claims&lt;/strong&gt;, not chunk IDs, and write it down.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Start with real or expert-written questions&lt;/strong&gt;, labelled with spans, a reference answer and stratum tags, including some unanswerable ones. Size it with the interval arithmetic above: around 100 detects only large differences.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Add synthetic questions for coverage&lt;/strong&gt; with filtering (exact-match evidence, deduplication, relevance threshold) in a separate split, and compare its length and lexical overlap with the real split.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Freeze the setup&lt;/strong&gt;: corpus snapshot, question-set version, tool versions, judge model and prompt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Report a small panel per stratum&lt;/strong&gt;: hit@k at the generator's k, MRR@k or nDCG@k, recall@N at the reranker's candidate depth, plus tokens in context and latency.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Report counts and 95% intervals&lt;/strong&gt;; compare configurations with a paired randomization or bootstrap test.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Change one thing at a time&lt;/strong&gt;: chunker, retriever, fusion, reranker. Evaluate first stage and reranker separately.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validate any LLM judge&lt;/strong&gt; against human labels and adversarial passages, using a different model family from your generator.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run end-to-end metrics too&lt;/strong&gt; and read them with the 2×2 table above.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read individual failures.&lt;/strong&gt; On the toy set, the abbreviation pattern is visible only in per-question ranks, not in the averages.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Feed production queries back&lt;/strong&gt; into the real split so the set follows your users, not your generator.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Relevance unit (span, claim, chunk, document) documented&lt;/li&gt;
&lt;li&gt;[ ] Real or expert questions are the decision set; synthetic questions a separate split&lt;/li&gt;
&lt;li&gt;[ ] Questions tagged: lexical / paraphrased / identifier / multi-hop / unanswerable&lt;/li&gt;
&lt;li&gt;[ ] Synthetic split checked for length, lexical overlap and topic coverage&lt;/li&gt;
&lt;li&gt;[ ] One presence metric (hit@k or recall@k) plus one rank metric (MRR or nDCG)&lt;/li&gt;
&lt;li&gt;[ ] k matches what the generator or reranker receives&lt;/li&gt;
&lt;li&gt;[ ] Metric variant and tool version pinned&lt;/li&gt;
&lt;li&gt;[ ] Per-stratum counts with 95% intervals; paired tests for comparisons&lt;/li&gt;
&lt;li&gt;[ ] Context-cost metric for chunking comparisons&lt;/li&gt;
&lt;li&gt;[ ] LLM judge validated on human labels and adversarial passages&lt;/li&gt;
&lt;li&gt;[ ] Individual failures reviewed, not just averages&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What I'm testing next
&lt;/h2&gt;

&lt;p&gt;Next in the lab: hybrid search with reciprocal rank fusion, measured with the same Day 04 harness, and a learned-embedding retriever on the paraphrased set. Because BM25 and TF-IDF cosine ranked identically on this corpus, fusing those two alone is unlikely to change anything; the paraphrased questions, including the two abbreviation misses, are where a learned-embedding retriever should make the difference. With only 12 questions, I will read any change there as a direction, not a result.&lt;/p&gt;

&lt;p&gt;If you evaluate retrieval in production, I would like to know which metric panel you settled on and how many questions it took before the numbers stopped moving.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Anthropic, "Introducing Contextual Retrieval" (2024): &lt;a href="https://www.anthropic.com/engineering/contextual-retrieval" rel="noopener noreferrer"&gt;https://www.anthropic.com/engineering/contextual-retrieval&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Google Research, "Deeper insights into retrieval augmented generation: The role of sufficient context" (blog on Joren et al., ICLR 2025): &lt;a href="https://research.google/blog/deeper-insights-into-retrieval-augmented-generation-the-role-of-sufficient-context/" rel="noopener noreferrer"&gt;https://research.google/blog/deeper-insights-into-retrieval-augmented-generation-the-role-of-sufficient-context/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Voorhees and Tice, "The TREC-8 Question Answering Track", LREC 2000: &lt;a href="http://www.lrec-conf.org/proceedings/lrec2000/pdf/26.pdf" rel="noopener noreferrer"&gt;http://www.lrec-conf.org/proceedings/lrec2000/pdf/26.pdf&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Järvelin and Kekäläinen, "Cumulated gain-based evaluation of IR techniques", ACM TOIS, 2002 (DOI 10.1145/582415.582418): &lt;a href="https://trepo.tuni.fi/handle/10024/65718" rel="noopener noreferrer"&gt;https://trepo.tuni.fi/handle/10024/65718&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Ragas docs, Context Precision: &lt;a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/" rel="noopener noreferrer"&gt;https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Ragas docs, Context Recall: &lt;a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/" rel="noopener noreferrer"&gt;https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Ragas docs, Faithfulness: &lt;a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/" rel="noopener noreferrer"&gt;https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Ragas docs, Answer Correctness: &lt;a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_correctness/" rel="noopener noreferrer"&gt;https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_correctness/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Ragas docs, Testset Generation for RAG: &lt;a href="https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/" rel="noopener noreferrer"&gt;https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;DeepEval docs, Contextual Precision: &lt;a href="https://deepeval.com/docs/metrics-contextual-precision" rel="noopener noreferrer"&gt;https://deepeval.com/docs/metrics-contextual-precision&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;DeepEval docs, Contextual Recall: &lt;a href="https://www.deepeval.com/docs/metrics-contextual-recall" rel="noopener noreferrer"&gt;https://www.deepeval.com/docs/metrics-contextual-recall&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;DeepEval docs, Contextual Relevancy: &lt;a href="https://www.deepeval.com/docs/metrics-contextual-relevancy" rel="noopener noreferrer"&gt;https://www.deepeval.com/docs/metrics-contextual-relevancy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Haystack docs, Evaluators: &lt;a href="https://docs.haystack.deepset.ai/docs/evaluators" rel="noopener noreferrer"&gt;https://docs.haystack.deepset.ai/docs/evaluators&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Haystack docs, DocumentRecallEvaluator: &lt;a href="https://docs.haystack.deepset.ai/docs/documentrecallevaluator" rel="noopener noreferrer"&gt;https://docs.haystack.deepset.ai/docs/documentrecallevaluator&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Haystack docs, DocumentNDCGEvaluator: &lt;a href="https://docs.haystack.deepset.ai/docs/documentndcgevaluator" rel="noopener noreferrer"&gt;https://docs.haystack.deepset.ai/docs/documentndcgevaluator&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Langfuse docs, Evaluation core concepts: &lt;a href="https://langfuse.com/docs/evaluation/core-concepts" rel="noopener noreferrer"&gt;https://langfuse.com/docs/evaluation/core-concepts&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Langfuse docs, Synthetic datasets: &lt;a href="https://langfuse.com/docs/datasets/example-synthetic-datasets" rel="noopener noreferrer"&gt;https://langfuse.com/docs/datasets/example-synthetic-datasets&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Smith and Troynikov, "Evaluating Chunking Strategies for Retrieval", Chroma Technical Report (2024): &lt;a href="https://www.trychroma.com/research/evaluating-chunking" rel="noopener noreferrer"&gt;https://www.trychroma.com/research/evaluating-chunking&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Ru et al., "RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation", NeurIPS 2024: &lt;a href="https://arxiv.org/abs/2408.08067" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2408.08067&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Thakur et al., "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models": &lt;a href="https://arxiv.org/abs/2104.08663" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2104.08663&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Enevoldsen et al., "MMTEB: Massive Multilingual Text Embedding Benchmark", ICLR 2025: &lt;a href="https://arxiv.org/abs/2502.13595" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2502.13595&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Hugging Face, "Introducing RTEB: A New Standard for Retrieval Evaluation" (Oct 2025): &lt;a href="https://huggingface.co/blog/rteb" rel="noopener noreferrer"&gt;https://huggingface.co/blog/rteb&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Su et al., "BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval", ICLR 2025: &lt;a href="https://arxiv.org/abs/2407.12883" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2407.12883&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Thakur et al., "FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents" (2025): &lt;a href="https://arxiv.org/abs/2504.13128" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2504.13128&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Weller, Boratko, Naim and Lee, "On the Theoretical Limitations of Embedding-Based Retrieval" (ICLR 2026): &lt;a href="https://arxiv.org/abs/2508.21038" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2508.21038&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Rahmani et al., "Synthetic Test Collections for Retrieval Evaluation", SIGIR 2024: &lt;a href="https://arxiv.org/abs/2405.07767" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2405.07767&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Rahmani, Ramineni, Craswell, Mitra and Yilmaz, "Towards Understanding Bias in Synthetic Data for Evaluation" (2025): &lt;a href="https://arxiv.org/abs/2506.10301" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2506.10301&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;van Elburg, van der Putten and Marx, "Can we Evaluate RAGs with Synthetic Data?" (2025): &lt;a href="https://arxiv.org/abs/2508.11758" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2508.11758&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Kucia and Gawlik, "Beyond Benchmark Scores: How Synthetic and Authentic Query Distributions Diverge in RAG Evaluation" (Sep 2026): &lt;a href="https://arxiv.org/abs/2609.14579" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2609.14579&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Thomas, Spielman, Craswell and Mitra, "Large language models can accurately predict searcher preferences": &lt;a href="https://arxiv.org/abs/2309.10621" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2309.10621&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Upadhyay et al., "UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor" (2024): &lt;a href="https://arxiv.org/abs/2406.06519" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2406.06519&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Alaofi, Thomas, Scholer and Sanderson, "LLMs can be Fooled into Labelling a Document as Relevant", SIGIR-AP 2024: &lt;a href="https://arxiv.org/abs/2501.17969" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2501.17969&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Clarke and Dietz, "LLM-based relevance assessment still can't replace human relevance assessment": &lt;a href="https://arxiv.org/abs/2412.17156" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2412.17156&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Yu, Li, Zuccon, Mackenzie and Leelanupab, "When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment" (2026): &lt;a href="https://arxiv.org/abs/2602.17170" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2602.17170&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023: &lt;a href="https://arxiv.org/abs/2306.05685" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2306.05685&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Saad-Falcon et al., "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems": &lt;a href="https://arxiv.org/abs/2311.09476" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2311.09476&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Pradeep et al., "The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models" (2025): &lt;a href="https://arxiv.org/abs/2504.15068" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2504.15068&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", TACL 2024: &lt;a href="https://aclanthology.org/2024.tacl-1.9/" rel="noopener noreferrer"&gt;https://aclanthology.org/2024.tacl-1.9/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Smucker, Allan and Carterette, "A Comparison of Statistical Significance Tests for Information Retrieval Evaluation", CIKM 2007: &lt;a href="https://ciir-publications.cs.umass.edu/getpdf.php?id=744" rel="noopener noreferrer"&gt;https://ciir-publications.cs.umass.edu/getpdf.php?id=744&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Miller, "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations" (2024): &lt;a href="https://arxiv.org/abs/2411.00640" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2411.00640&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Cormack, Clarke and Buettcher, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods", SIGIR 2009: &lt;a href="https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf" rel="noopener noreferrer"&gt;https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;ranx on PyPI: &lt;a href="https://pypi.org/project/ranx/" rel="noopener noreferrer"&gt;https://pypi.org/project/ranx/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Muhammad Hussnain Jamil, applied-llm-lab (Days 01 to 04): &lt;a href="https://github.com/Hussnainoffical/applied-llm-lab" rel="noopener noreferrer"&gt;https://github.com/Hussnainoffical/applied-llm-lab&lt;/a&gt;
&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>rag</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>python</category>
    </item>
    <item>
      <title>Why Your RAG System Misses the Answer: A 2026 Evidence-Based Guide to Chunking, Hybrid Search and Reranking</title>
      <dc:creator>Muhammad Hussnain Jamil</dc:creator>
      <pubDate>Fri, 30 Oct 2026 05:21:41 +0000</pubDate>
      <link>https://dev.to/hussnainoffical/why-your-rag-system-misses-the-answer-a-2026-evidence-based-guide-to-chunking-hybrid-search-and-29kf</link>
      <guid>https://dev.to/hussnainoffical/why-your-rag-system-misses-the-answer-a-2026-evidence-based-guide-to-chunking-hybrid-search-and-29kf</guid>
      <description>&lt;p&gt;When a RAG system gives a wrong answer, the model is usually blamed. In practice the failure often happens earlier: the right passage never reached the model, because it was split in the wrong place, scored badly or ranked too low.&lt;/p&gt;

&lt;p&gt;In 2026 there is no shortage of advice about how to fix this. Semantic chunking, late chunking, contextual retrieval, proposition chunking, agentic search, million-token context windows. Most of it comes from vendors or tutorials. This year, though, several large independent studies tested these ideas head to head, at scale, with proper statistics.&lt;/p&gt;

&lt;p&gt;This article pulls that evidence together. It covers:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Why retrieval still matters when context windows are huge&lt;/li&gt;
&lt;li&gt;Every major chunking method, how it works, what it costs and what the 2024–2026 studies found&lt;/li&gt;
&lt;li&gt;Why the retriever (hybrid search, rerankers, embeddings) usually matters more than the chunker&lt;/li&gt;
&lt;li&gt;How to evaluate your own pipeline&lt;/li&gt;
&lt;li&gt;A practical recipe, plus three small experiments of my own that show the mechanisms on a toy scale&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Every number below links to its source. Where a result comes from a vendor, I say so.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Long context did not make retrieval obsolete
&lt;/h2&gt;

&lt;p&gt;The obvious question first: if a model can read 1 million tokens, why chunk anything?&lt;/p&gt;

&lt;p&gt;Three lines of evidence say retrieval is still worth it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quality drops as input grows.&lt;/strong&gt; Chroma's &lt;a href="https://trychroma.com/research/context-rot" rel="noopener noreferrer"&gt;Context Rot study&lt;/a&gt; (July 2025) tested 18 models, including GPT-4.1, Claude 4 and Gemini 2.5. Performance degraded as input length increased, even on simple tasks, and distracting passages made it worse. On LongMemEval, focused prompts of about 300 tokens clearly beat full prompts of about 113K tokens for every model.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Effective context is much shorter than the advertised window.&lt;/strong&gt; In &lt;a href="https://huggingface.co/papers/2502.05167" rel="noopener noreferrer"&gt;NoLiMa&lt;/a&gt; (ICML 2025), 10 of 12 models that support 128K+ tokens fell below half of their short-context score at just 32K tokens when the question and the answer did not share keywords. GPT-4o dropped from 99.3% to 69.7%.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Long context costs far more.&lt;/strong&gt; A June 2026 study, &lt;a href="https://arxiv.org/html/2606.20898v1" rel="noopener noreferrer"&gt;"The Token Tax"&lt;/a&gt;, compared long-context prompting with RAG on 324 matched questions. Long context was more accurate (73.1% vs 65.4%), but it cost about 26 times more per query ($0.118 vs $0.0045) and used about 29 times more input tokens. Their recommendation is a sensible default: use RAG first and escalate to long context only for low-confidence questions.&lt;/p&gt;

&lt;p&gt;So the realistic 2026 position is not "RAG or long context". It is "retrieve well, then decide how much context the model really needs". Retrieving well starts with the chunk.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. What a chunk is for
&lt;/h2&gt;

&lt;p&gt;A chunk has two jobs at the same time:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Unit of retrieval:&lt;/strong&gt; it is what gets scored against the question. Smaller chunks give sharper matches.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unit of context:&lt;/strong&gt; it is what the model reads. Larger chunks keep the surrounding meaning.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every chunking method is a different trade-off between these two jobs. A useful way to see the problem is Chroma's &lt;a href="https://www.trychroma.com/research/evaluating-chunking" rel="noopener noreferrer"&gt;chunking evaluation&lt;/a&gt; (2024), which measured not only whether the relevant tokens were retrieved (recall) but also how many irrelevant tokens came with them (precision and IoU). With similar recall, strategies differed by about five times in wasted tokens. Recall alone hides that cost.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. The chunking menu in 2026
&lt;/h2&gt;

&lt;h3&gt;
  
  
  3.1 Fixed-size and recursive splitting
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;How it works:&lt;/strong&gt; cut every N tokens or characters, optionally with an overlap. Recursive splitters try paragraph breaks first, then sentences, then words, so they only cut mid-sentence as a last resort.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Evidence:&lt;/strong&gt; the simplest method keeps winning.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The largest study so far, &lt;a href="https://arxiv.org/pdf/2608.16586" rel="noopener noreferrer"&gt;Caspari et al., accepted at CIKM'26&lt;/a&gt;, compared 8 methods on corpora of up to 10 million documents with three modern embedders (Qwen3-Embedding-0.6B, EmbeddingGemma-300M, Snowflake Arctic). Their conclusion: expensive methods "rarely" beat simple ones consistently, and &lt;strong&gt;token-based chunking is a strong default at scale&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/html/2606.00881v1" rel="noopener noreferrer"&gt;Śmigielski et al.&lt;/a&gt; (May 2026) reported Accuracy@5 of 89.36 for recursive semantic chunking and 87.71 for plain fixed-size chunking. Fixed-size chunking took under a second; the proposition-based method took 15 hours.&lt;/li&gt;
&lt;li&gt;One warning from &lt;a href="https://arxiv.org/html/2603.06976" rel="noopener noreferrer"&gt;Shaukat et al.&lt;/a&gt; (March 2026): fixed &lt;em&gt;character&lt;/em&gt; chunking was the worst of 36 strategies (nDCG@5 of 0.244). Token-aware and structure-aware splitting is fine; blindly cutting characters is not.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Use it when:&lt;/strong&gt; you need a baseline, a large corpus or predictable cost. That is most projects.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.2 Sentence and paragraph grouping
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;How it works:&lt;/strong&gt; split into sentences or paragraphs, then pack whole units until a size limit is reached.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Evidence:&lt;/strong&gt; in Shaukat et al.'s 1,080-configuration study (36 strategies × 6 domains × 5 embedders), &lt;strong&gt;paragraph group chunking ranked first&lt;/strong&gt; (nDCG@5 ≈ 0.459, Hit@5 ≈ 59%). They also found that a stronger embedding model does not make up for bad segmentation. The gains from good chunking and from a good embedder add up.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Use it when:&lt;/strong&gt; your documents have real paragraphs, for example articles, reports and documentation.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.3 Semantic chunking
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;How it works:&lt;/strong&gt; embed each sentence and start a new chunk where the similarity between neighbouring sentences drops.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Evidence:&lt;/strong&gt; weaker than its popularity suggests.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/html/2410.13070v1" rel="noopener noreferrer"&gt;Qu, Bao and Tu (Vectara), NAACL 2025 Findings&lt;/a&gt;: "the computational costs associated with semantic chunking are not justified by consistent performance gains." On normal documents, fixed-size chunks won (HotpotQA F1@5: 90.59 vs 87.37 for breakpoint-based semantic chunking). Semantic chunking only won on artificially "stitched" documents, where unrelated texts had been glued together.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2607.01852" rel="noopener noreferrer"&gt;Kreileder et al.&lt;/a&gt; (July 2026) found the same on academic theses: cluster-based semantic chunking did not beat fixed or recursive chunking.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Use it when:&lt;/strong&gt; your documents really do mix unrelated topics without clear structure. Otherwise, test it against a recursive splitter before paying for it.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.4 Late chunking
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;How it works:&lt;/strong&gt; &lt;a href="https://jina.ai/news/late-chunking-in-long-context-embedding-models/" rel="noopener noreferrer"&gt;Jina's late chunking&lt;/a&gt; runs the whole document through a long-context embedding model first, then averages the token embeddings inside each chunk's boundaries. Each chunk vector therefore "knows" about the rest of the document.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Evidence:&lt;/strong&gt; helpful on long documents in Jina's own tests (for example NFCorpus nDCG@10 from 23.46 to 29.98), but at scale it was weaker. In the CIKM'26 study, late chunking on a 1M-document corpus scored Recall@100 of 50.54 against 57.09 for plain token chunking, and it needs a lot of memory.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Use it when:&lt;/strong&gt; you have long, coherent documents and a long-context embedder, and you have measured a gain on your own data.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.5 Contextual retrieval and contextual embeddings
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;How it works:&lt;/strong&gt; &lt;a href="https://www.anthropic.com/news/contextual-retrieval" rel="noopener noreferrer"&gt;Anthropic's contextual retrieval&lt;/a&gt; (2024) asks an LLM to write 50–100 tokens explaining where each chunk sits in its document, and prepends that to the chunk before embedding and BM25 indexing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Evidence:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;In Anthropic's tests, the top-20 retrieval failure rate fell by 35% with contextual embeddings, 49% when contextual BM25 was added, and 67% when a reranker was added as well (5.7% → 1.9%). With prompt caching, it costs about $1.02 per million document tokens.&lt;/li&gt;
&lt;li&gt;The 2026 independent evidence is more cautious. The CIKM'26 study found the cost was "rarely matched by retrieval gains", and contextual chunking processed only 0.26 documents per second at 100 generated tokens per second. On the other hand, in a financial benchmark (below), "contextual hybrid" was the best method &lt;em&gt;without&lt;/em&gt; a reranker.&lt;/li&gt;
&lt;li&gt;A newer alternative skips the LLM call: &lt;a href="https://www.mongodb.com/company/blog/product-release-announcements/voyage-context-3-focused-chunk-level-details-global-document-context" rel="noopener noreferrer"&gt;voyage-context-3&lt;/a&gt; (July 2025) embeds each chunk together with its document context in one model. Voyage reports large gains over Anthropic's method and over late chunking; these are vendor numbers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cheap version that works:&lt;/strong&gt; the CIKM'26 authors found that simply &lt;strong&gt;prepending the document title&lt;/strong&gt; ("enriched title") was the most reliable low-cost improvement. Docling's &lt;a href="https://docling-project.github.io/docling/_generated/examples/hybrid_chunking/" rel="noopener noreferrer"&gt;HybridChunker&lt;/a&gt; does the same with &lt;code&gt;contextualize()&lt;/code&gt;, which prepends the title and section headings.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.6 Proposition chunking
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;How it works:&lt;/strong&gt; an LLM rewrites the text into short, self-contained factual statements, and each one becomes a chunk.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Evidence:&lt;/strong&gt; the original &lt;a href="https://arxiv.org/abs/2312.06648" rel="noopener noreferrer"&gt;Dense X Retrieval&lt;/a&gt; paper reported strong gains, but in the 2026 head-to-head by Śmigielski et al. it was the weakest and slowest method (Accuracy@5 69.10, about 15 hours of processing).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Use it when:&lt;/strong&gt; you have a small, high-value knowledge base of short facts and can afford the LLM pass.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.7 Hierarchical chunks (small-to-big, parent–child)
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;How it works:&lt;/strong&gt; retrieve small chunks for precise matching, then hand the model their larger parent section. &lt;a href="https://arxiv.org/html/2509.11552v2" rel="noopener noreferrer"&gt;HiChunk&lt;/a&gt; (ACL 2026) builds a multi-level hierarchy and merges child chunks into parents within a token budget.&lt;/p&gt;

&lt;p&gt;HiChunk's most useful point is about evaluation: most benchmarks have only about 1.7 evidence sentences per question, so they cannot tell chunkers apart. If your evaluation questions each need one sentence, every chunker looks the same.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.8 Layout-aware chunking for PDFs
&lt;/h3&gt;

&lt;p&gt;Real documents are not plain text. Tables, headings and page structure matter.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;In Unstructured's &lt;a href="https://arxiv.org/html/2402.05131v3" rel="noopener noreferrer"&gt;FinanceBench study&lt;/a&gt;, chunks based on document elements (titles, tables, lists) improved question-answering accuracy from 48.23% to 53.19% compared with 512-token chunks.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses/" rel="noopener noreferrer"&gt;NVIDIA's 2025 chunking study&lt;/a&gt; found that &lt;strong&gt;page-level chunks&lt;/strong&gt; had the best average accuracy (0.648) and the lowest variance across five datasets. Factoid questions did best with 256–512 token chunks; analytical questions did best with 1,024 tokens or whole pages.&lt;/li&gt;
&lt;li&gt;Docling's HybridChunker can repeat table headers in each chunk and should use the same tokenizer as your embedding model.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3.9 Visual retrieval: skipping text extraction
&lt;/h3&gt;

&lt;p&gt;For slide decks, scanned reports and pages full of charts, you can embed page images directly with ColPali-style models. On &lt;a href="https://arxiv.org/html/2601.08620v1" rel="noopener noreferrer"&gt;ViDoRe V3&lt;/a&gt; (January 2026: about 26K pages, 3,099 human-verified queries in 6 languages including German), the best visual retriever scored 59.8 nDCG@10 against 51.0 for the best text-based retriever. A text reranker still added 13.2 points to the text pipeline, so reranking matters there too.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. What the large studies agree on
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Study&lt;/th&gt;
&lt;th&gt;Scale&lt;/th&gt;
&lt;th&gt;Main finding&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.trychroma.com/research/evaluating-chunking" rel="noopener noreferrer"&gt;Chroma 2024&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;472 queries, 5 corpora&lt;/td&gt;
&lt;td&gt;Strategies differ by up to 9% recall and ~5× in wasted tokens; 200-token recursive splitting was a strong choice&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://arxiv.org/html/2410.13070v1" rel="noopener noreferrer"&gt;Vectara, NAACL 2025&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Several QA datasets&lt;/td&gt;
&lt;td&gt;Semantic chunking's cost not justified on normal documents&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses/" rel="noopener noreferrer"&gt;NVIDIA 2025&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;5 datasets&lt;/td&gt;
&lt;td&gt;Page-level chunks most accurate and most stable; chunk size should follow query type&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://arxiv.org/html/2603.06976" rel="noopener noreferrer"&gt;Shaukat et al. 2026&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;1,080 configurations&lt;/td&gt;
&lt;td&gt;Paragraph grouping best; character cutting worst; chunker and embedder gains add up&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://arxiv.org/html/2606.00881v1" rel="noopener noreferrer"&gt;Śmigielski et al. 2026&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Cost vs quality&lt;/td&gt;
&lt;td&gt;Recursive and fixed-size give the best balance; LLM-based methods slow and fragile&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://arxiv.org/pdf/2608.16586" rel="noopener noreferrer"&gt;Caspari et al., CIKM'26&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Up to 10M documents&lt;/td&gt;
&lt;td&gt;Complex chunking rarely wins; token chunking + title is the default&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The pattern is consistent: &lt;strong&gt;structure-aware, size-limited chunks plus a little cheap context (title, headings) get you most of the way.&lt;/strong&gt; Expensive chunkers can win on specific datasets, but the gains shrink as the corpus grows and they cost a lot of compute.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. A small experiment: watching chunking fail
&lt;/h2&gt;

&lt;p&gt;In my lab repository, &lt;a href="https://github.com/Hussnainoffical/applied-llm-lab" rel="noopener noreferrer"&gt;applied-llm-lab&lt;/a&gt;, I write each retrieval component from scratch with tests. Three experiments show the mechanisms behind the studies above on a toy scale. This is a tiny, five-question set, so it illustrates the mechanism; it is not a benchmark.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Experiment 1: three chunkers on the same text.&lt;/strong&gt; At almost the same chunk size, fixed-size chunking cut 3 of 4 chunks mid-sentence. Sentence packing cut none.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;by_sentence&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_chars&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Pack whole sentences into chunks up to max_chars.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;sentences&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;(?&amp;lt;=[.!?])\s+&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;
    &lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;sentences&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;max_chars&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Experiment 2: BM25 from scratch.&lt;/strong&gt; I implemented Okapi BM25 (k1 = 1.5, b = 0.75) and asked five questions, each with the exact fact the top chunk must contain in full.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;norm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;k1&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;avgdl&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;idf&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;k1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
               &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;tokenize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tf&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Experiment 3: TF-IDF cosine search with NumPy&lt;/strong&gt;, plus three paraphrased questions that ask for the same facts in different words.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Chunking&lt;/th&gt;
&lt;th&gt;BM25, original questions&lt;/th&gt;
&lt;th&gt;Cosine, original questions&lt;/th&gt;
&lt;th&gt;BM25, paraphrased&lt;/th&gt;
&lt;th&gt;Cosine, paraphrased&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fixed size (120 chars)&lt;/td&gt;
&lt;td&gt;2 / 5&lt;/td&gt;
&lt;td&gt;2 / 5&lt;/td&gt;
&lt;td&gt;0 / 3&lt;/td&gt;
&lt;td&gt;1 / 3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fixed size + 30 overlap&lt;/td&gt;
&lt;td&gt;4 / 5&lt;/td&gt;
&lt;td&gt;4 / 5&lt;/td&gt;
&lt;td&gt;0 / 3&lt;/td&gt;
&lt;td&gt;0 / 3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sentence packing (160)&lt;/td&gt;
&lt;td&gt;5 / 5&lt;/td&gt;
&lt;td&gt;5 / 5&lt;/td&gt;
&lt;td&gt;0 / 3&lt;/td&gt;
&lt;td&gt;0 / 3&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three lessons, each matching the large studies:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Chunking decided more than the scoring formula.&lt;/strong&gt; BM25 and cosine picked the same top chunk in 14 of 15 cases. Changing the chunker moved the score from 2/5 to 5/5. This is Shaukat et al.'s finding in miniature.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Overlap is a patch.&lt;/strong&gt; It recovered most of the gap but added a chunk to index. At scale, that means more storage and more embedding cost.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lexical methods cannot handle paraphrases.&lt;/strong&gt; Both failed on every reworded question (the single "hit" was an accident: the chunker cut "con|text" and the leftover "text" matched the question). That gap is why the next section matters.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  6. The retriever matters more than the chunker
&lt;/h2&gt;

&lt;h3&gt;
  
  
  6.1 Hybrid search plus a reranker
&lt;/h3&gt;

&lt;p&gt;The clearest 2026 result comes from &lt;a href="https://arxiv.org/html/2604.01733v1" rel="noopener noreferrer"&gt;Akarsu et al.&lt;/a&gt; (April 2026), who benchmarked retrieval strategies on T2-RAGBench: 23,088 questions over 7,318 financial documents mixing text and tables.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Method&lt;/th&gt;
&lt;th&gt;Recall@5&lt;/th&gt;
&lt;th&gt;nDCG@10&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;BM25&lt;/td&gt;
&lt;td&gt;0.644&lt;/td&gt;
&lt;td&gt;0.515&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dense (text-embedding-3-large)&lt;/td&gt;
&lt;td&gt;0.587&lt;/td&gt;
&lt;td&gt;0.466&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HyDE&lt;/td&gt;
&lt;td&gt;0.544&lt;/td&gt;
&lt;td&gt;0.433&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Multi-query + RRF&lt;/td&gt;
&lt;td&gt;0.640&lt;/td&gt;
&lt;td&gt;0.506&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hybrid (BM25 + dense, RRF)&lt;/td&gt;
&lt;td&gt;0.695&lt;/td&gt;
&lt;td&gt;0.551&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Contextual hybrid&lt;/td&gt;
&lt;td&gt;0.717&lt;/td&gt;
&lt;td&gt;0.571&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hybrid + reranker&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.816&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.683&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three things stand out:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;BM25 beat a strong dense model.&lt;/strong&gt; On financial text full of exact terms and numbers, keyword matching still wins.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HyDE hurt.&lt;/strong&gt; Generating a hypothetical answer and searching with it made retrieval worse.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hybrid plus reranking was best by a wide margin:&lt;/strong&gt; Recall@5 rose by about 23 points over dense retrieval alone and about 17 points over BM25.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;My third experiment explains why hybrid helps: lexical search catches exact terms, dense embeddings catch paraphrases, and each covers the other's blind spot. Reciprocal rank fusion (RRF) merges the two ranked lists without needing to calibrate their scores.&lt;/p&gt;

&lt;h3&gt;
  
  
  6.2 Why one vector per chunk has a ceiling
&lt;/h3&gt;

&lt;p&gt;Google DeepMind's &lt;a href="https://arxiv.org/html/2508.21038v1" rel="noopener noreferrer"&gt;LIMIT paper&lt;/a&gt; (ICLR 2026) showed a theoretical limit: the embedding dimension bounds which sets of top-k documents a single-vector retriever can return for any query. Even state-of-the-art MTEB models scored under 20 Recall@100 on a deliberately simple dataset built around that limit. The practical conclusion is to add a stage that is not a single vector: sparse BM25, multi-vector late interaction, or a cross-encoder reranker.&lt;/p&gt;

&lt;h3&gt;
  
  
  6.3 Late interaction is back
&lt;/h3&gt;

&lt;p&gt;ColBERT-style models keep one vector per token and compare query and document tokens individually. In March 2026, LightOn reported that its 149M-parameter &lt;a href="https://lighton.ai/lighton-blogs/the-bloated-retriever-era-is-over" rel="noopener noreferrer"&gt;Reason-ModernColBERT&lt;/a&gt; reached 87.59% accuracy on BrowseComp-Plus with GPT-5, against 80.00% for the previous best, while needing fewer search calls. It also beat an 8B dense embedder about 54 times its size. These are vendor numbers, but they fit the LIMIT theory.&lt;/p&gt;

&lt;h3&gt;
  
  
  6.4 Rerankers in 2026
&lt;/h3&gt;

&lt;p&gt;A reranker reads the question and each candidate chunk together and scores how well they match. It is slower than vector search, so you use it on the top 50–150 candidates only. Anthropic recommends retrieving 150 and reranking down to 20.&lt;/p&gt;

&lt;p&gt;The community &lt;a href="https://github.com/agentset-ai/reranker-eval" rel="noopener noreferrer"&gt;reranker-eval&lt;/a&gt; ranking currently puts zerank-2, Cohere Rerank 4 Pro, zerank-1 and Voyage rerank-2.5 at the top, with Qwen3-Reranker-8B and BGE-reranker-v2-m3 as strong open-weight options. Cohere's Rerank 4 (December 2025) raised its context to 32K tokens, so it can score long chunks.&lt;/p&gt;

&lt;h3&gt;
  
  
  6.5 Embedding models, and a warning about leaderboards
&lt;/h3&gt;

&lt;p&gt;Strong options in 2026 include the &lt;a href="https://huggingface.co/Qwen/Qwen3-Embedding-4B" rel="noopener noreferrer"&gt;Qwen3-Embedding&lt;/a&gt; family (0.6B to 8B, 32K context), &lt;a href="https://blog.voyageai.com/2026/01/15/new-models-and-expanded-availability/" rel="noopener noreferrer"&gt;Voyage 4&lt;/a&gt; (whose models share one embedding space, so you can switch models without re-indexing), &lt;a href="https://www.elastic.co/search-labs/blog/jina-embeddings-v5-text" rel="noopener noreferrer"&gt;jina-embeddings-v5&lt;/a&gt; and the multimodal &lt;a href="https://siliconangle.com/2026/10/06/google-expands-embeddinggemma-beyond-text-to-images-audio-and-video/" rel="noopener noreferrer"&gt;EmbeddingGemma 2&lt;/a&gt; (October 2026).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;For German documents&lt;/strong&gt;, the community model &lt;a href="https://huggingface.co/blog/malteos/most-embed-de-german-embedding-model" rel="noopener noreferrer"&gt;most-embed-de&lt;/a&gt; (September 2026) reports first place on the German MTEB retrieval tasks; these are the author's own numbers. ViDoRe V3 found a 2–3 point penalty when queries and documents are in different languages, so test with real German queries if you serve German users.&lt;/p&gt;

&lt;p&gt;The warning: &lt;a href="https://trychroma.com/research/generative-benchmarking" rel="noopener noreferrer"&gt;Chroma found&lt;/a&gt; that MTEB rankings did not transfer to a real-world dataset. A model that ranked higher on MTEB scored lower on their production data (Recall@10 of 0.532 vs 0.602). Always test the top two or three candidates on your own questions.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Agentic retrieval: where it fits
&lt;/h2&gt;

&lt;p&gt;Agents now search iteratively: they run a query, read results, refine the query and search again. Anthropic's &lt;a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents" rel="noopener noreferrer"&gt;context engineering guidance&lt;/a&gt; describes "just-in-time" retrieval through tools, combined with some context loaded up front.&lt;/p&gt;

&lt;p&gt;Two findings keep this grounded:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The retriever still sets the ceiling.&lt;/strong&gt; On &lt;a href="https://arxiv.org/abs/2508.06600" rel="noopener noreferrer"&gt;BrowseComp-Plus&lt;/a&gt;, the same agent (GPT-5) scored 55.9% with BM25 and 70.1% with Qwen3-Embedding-8B, and needed fewer searches with the better retriever.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Query rewriting is not free accuracy.&lt;/strong&gt; In the T2-RAGBench study, HyDE reduced recall and multi-query fusion only matched plain BM25.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An agent multiplies the quality of its search tool. It does not replace it.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. How to evaluate your own pipeline
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Metrics:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Hit rate@k / Recall@k:&lt;/strong&gt; did the needed chunk appear in the top k?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MRR:&lt;/strong&gt; how high was the first correct chunk?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;nDCG@k:&lt;/strong&gt; rewards correct chunks near the top, with graded relevance.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Token precision / IoU:&lt;/strong&gt; how much irrelevant text came along (from the Chroma method). Useful when context cost matters.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Build a test set from your own documents.&lt;/strong&gt; Chroma's &lt;a href="https://trychroma.com/research/generative-benchmarking" rel="noopener noreferrer"&gt;generative benchmarking&lt;/a&gt; generates realistic questions from your corpus and filters them; after five iterations their LLM judge agreed with humans 75.2% of the time, up from 46%. Even 50–100 good questions beat any public leaderboard for your use case.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Make questions that need real evidence.&lt;/strong&gt; Following HiChunk's point, include questions whose answers span several sentences or a table, or every chunker will look the same.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Be careful with LLM-judged scores.&lt;/strong&gt; &lt;a href="https://docs.ragas.io/en/latest/howtos/migrations/migrate_from_v03_to_v04/" rel="noopener noreferrer"&gt;RAGAS v0.4&lt;/a&gt; (December 2025) changed its API (&lt;code&gt;@experiment()&lt;/code&gt; replaces &lt;code&gt;evaluate()&lt;/code&gt;), and the 2026 thesis study found its faithfulness score unreliable. Use retrieval metrics with a fixed answer key wherever you can.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. A practical recipe for late 2026
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Start simple.&lt;/strong&gt; Recursive or paragraph-aware chunks of about 256–512 tokens, small overlap, same tokenizer as the embedder.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Add cheap context.&lt;/strong&gt; Prepend the document title and section headings to each chunk.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Respect structure.&lt;/strong&gt; For PDFs, use a layout-aware parser (for example Docling) and keep tables whole with their headers. Consider page-level chunks for reports.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use hybrid retrieval.&lt;/strong&gt; BM25 + dense embeddings, merged with RRF.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rerank.&lt;/strong&gt; Retrieve 50–150 candidates and rerank to the top 5–20.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Evaluate on your own questions&lt;/strong&gt; with Recall@k, MRR and nDCG before changing anything expensive.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Only then try advanced chunking&lt;/strong&gt; (contextual, late, semantic, propositions), and keep it only if it beats step 6 on your data at a cost you accept.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use long context selectively&lt;/strong&gt;, for questions where retrieval confidence is low.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  10. What I'm testing next
&lt;/h2&gt;

&lt;p&gt;Next in the lab: hit rate and MRR on a larger question set, learned embeddings against TF-IDF on the paraphrased questions, hybrid search with RRF, and a cross-encoder reranker on top. I'll publish the numbers as they come in, including the ones that don't go my way.&lt;/p&gt;

&lt;p&gt;Code and tests: &lt;a href="https://github.com/Hussnainoffical/applied-llm-lab" rel="noopener noreferrer"&gt;github.com/Hussnainoffical/applied-llm-lab&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I'm Muhammad Hussnain Jamil, an applied LLM and AI automation engineer and co-author of published research on multi-agent LLM systems. I'm learning German (A2) and open to AI roles in Germany. &lt;a href="https://muhammad-hussnain-jamil-portfolio.vercel.app" rel="noopener noreferrer"&gt;Portfolio&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Caspari et al., "When Is Complex Chunking Worth It?", CIKM'26: &lt;a href="https://arxiv.org/pdf/2608.16586" rel="noopener noreferrer"&gt;https://arxiv.org/pdf/2608.16586&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Śmigielski et al., 2026: &lt;a href="https://arxiv.org/html/2606.00881v1" rel="noopener noreferrer"&gt;https://arxiv.org/html/2606.00881v1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Shaukat et al., 2026: &lt;a href="https://arxiv.org/html/2603.06976" rel="noopener noreferrer"&gt;https://arxiv.org/html/2603.06976&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Qu, Bao and Tu (Vectara), NAACL 2025 Findings: &lt;a href="https://arxiv.org/html/2410.13070v1" rel="noopener noreferrer"&gt;https://arxiv.org/html/2410.13070v1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Kreileder et al., 2026: &lt;a href="https://arxiv.org/abs/2607.01852" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2607.01852&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Chroma, Evaluating Chunking: &lt;a href="https://www.trychroma.com/research/evaluating-chunking" rel="noopener noreferrer"&gt;https://www.trychroma.com/research/evaluating-chunking&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;NVIDIA, Finding the Best Chunking Strategy: &lt;a href="https://developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses/" rel="noopener noreferrer"&gt;https://developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic, Contextual Retrieval: &lt;a href="https://www.anthropic.com/news/contextual-retrieval" rel="noopener noreferrer"&gt;https://www.anthropic.com/news/contextual-retrieval&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Jina, Late Chunking: &lt;a href="https://jina.ai/news/late-chunking-in-long-context-embedding-models/" rel="noopener noreferrer"&gt;https://jina.ai/news/late-chunking-in-long-context-embedding-models/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Voyage/MongoDB, voyage-context-3: &lt;a href="https://www.mongodb.com/company/blog/product-release-announcements/voyage-context-3-focused-chunk-level-details-global-document-context" rel="noopener noreferrer"&gt;https://www.mongodb.com/company/blog/product-release-announcements/voyage-context-3-focused-chunk-level-details-global-document-context&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Chen et al., Dense X Retrieval: &lt;a href="https://arxiv.org/abs/2312.06648" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2312.06648&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;HiChunk, ACL 2026: &lt;a href="https://arxiv.org/html/2509.11552v2" rel="noopener noreferrer"&gt;https://arxiv.org/html/2509.11552v2&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Jimeno Yepes et al., element-based chunking (FinanceBench): &lt;a href="https://arxiv.org/html/2402.05131v3" rel="noopener noreferrer"&gt;https://arxiv.org/html/2402.05131v3&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Docling HybridChunker: &lt;a href="https://docling-project.github.io/docling/_generated/examples/hybrid_chunking/" rel="noopener noreferrer"&gt;https://docling-project.github.io/docling/_generated/examples/hybrid_chunking/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;ViDoRe V3: &lt;a href="https://arxiv.org/html/2601.08620v1" rel="noopener noreferrer"&gt;https://arxiv.org/html/2601.08620v1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Akarsu et al., From BM25 to Corrective RAG (T2-RAGBench): &lt;a href="https://arxiv.org/html/2604.01733v1" rel="noopener noreferrer"&gt;https://arxiv.org/html/2604.01733v1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LIMIT, Google DeepMind: &lt;a href="https://arxiv.org/html/2508.21038v1" rel="noopener noreferrer"&gt;https://arxiv.org/html/2508.21038v1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LightOn, Reason-ModernColBERT: &lt;a href="https://lighton.ai/lighton-blogs/the-bloated-retriever-era-is-over" rel="noopener noreferrer"&gt;https://lighton.ai/lighton-blogs/the-bloated-retriever-era-is-over&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Agentset reranker-eval: &lt;a href="https://github.com/agentset-ai/reranker-eval" rel="noopener noreferrer"&gt;https://github.com/agentset-ai/reranker-eval&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Qwen3-Embedding: &lt;a href="https://huggingface.co/Qwen/Qwen3-Embedding-4B" rel="noopener noreferrer"&gt;https://huggingface.co/Qwen/Qwen3-Embedding-4B&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Voyage 4: &lt;a href="https://blog.voyageai.com/2026/01/15/new-models-and-expanded-availability/" rel="noopener noreferrer"&gt;https://blog.voyageai.com/2026/01/15/new-models-and-expanded-availability/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;most-embed-de: &lt;a href="https://huggingface.co/blog/malteos/most-embed-de-german-embedding-model" rel="noopener noreferrer"&gt;https://huggingface.co/blog/malteos/most-embed-de-german-embedding-model&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Chroma, Context Rot: &lt;a href="https://trychroma.com/research/context-rot" rel="noopener noreferrer"&gt;https://trychroma.com/research/context-rot&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;NoLiMa: &lt;a href="https://huggingface.co/papers/2502.05167" rel="noopener noreferrer"&gt;https://huggingface.co/papers/2502.05167&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The Token Tax: &lt;a href="https://arxiv.org/html/2606.20898v1" rel="noopener noreferrer"&gt;https://arxiv.org/html/2606.20898v1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic, Effective Context Engineering: &lt;a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents" rel="noopener noreferrer"&gt;https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;BrowseComp-Plus: &lt;a href="https://arxiv.org/abs/2508.06600" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2508.06600&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Chroma, Generative Benchmarking: &lt;a href="https://trychroma.com/research/generative-benchmarking" rel="noopener noreferrer"&gt;https://trychroma.com/research/generative-benchmarking&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;RAGAS v0.4 migration guide: &lt;a href="https://docs.ragas.io/en/latest/howtos/migrations/migrate_from_v03_to_v04/" rel="noopener noreferrer"&gt;https://docs.ragas.io/en/latest/howtos/migrations/migrate_from_v03_to_v04/&lt;/a&gt;
&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>rag</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>python</category>
    </item>
  </channel>
</rss>
