<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: laolei</title>
    <description>The latest articles on DEV Community by laolei (@laolei).</description>
    <link>https://dev.to/laolei</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4060296%2Ffcf1c23e-7007-4f56-952b-d3c612b20b62.png</url>
      <title>DEV Community: laolei</title>
      <link>https://dev.to/laolei</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/laolei"/>
    <language>en</language>
    <item>
      <title>AI won't replace mathematicians, but mathematicians who use AI well will replace those who don't — just like programmers.</title>
      <dc:creator>laolei</dc:creator>
      <pubDate>Tue, 17 Nov 2026 03:28:30 +0000</pubDate>
      <link>https://dev.to/laolei/ai-wont-replace-mathematicians-but-mathematicians-who-use-ai-well-will-replace-those-who-dont--301n</link>
      <guid>https://dev.to/laolei/ai-wont-replace-mathematicians-but-mathematicians-who-use-ai-well-will-replace-those-who-dont--301n</guid>
      <description>&lt;p&gt;Every knowledge-based profession faces the same situation.&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Is Your Agent's LLM Judge Unreliable? Use Logprobs to Make Scores Comparable</title>
      <dc:creator>laolei</dc:creator>
      <pubDate>Mon, 16 Nov 2026 03:44:45 +0000</pubDate>
      <link>https://dev.to/laolei/is-your-agents-llm-judge-unreliable-use-logprobs-to-make-scores-comparable-1j6m</link>
      <guid>https://dev.to/laolei/is-your-agents-llm-judge-unreliable-use-logprobs-to-make-scores-comparable-1j6m</guid>
      <description>&lt;h1&gt;
  
  
  Is Your Agent's LLM Judge Unreliable? Use Logprobs to Make Scores Comparable
&lt;/h1&gt;

&lt;blockquote&gt;
&lt;p&gt;One sentence: &lt;strong&gt;the model gives you an integer; a decision needs a measurement.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This article gives the fix first (three steps), then explains why it works,&lt;br&gt;
and ends with the engineering constraints you cannot skip.&lt;/p&gt;

&lt;p&gt;The code below comes from the real implementation in langchainrust 0.30.0:&lt;br&gt;
&lt;code&gt;lc-core/src/language_models/chat.rs&lt;/code&gt;,&lt;br&gt;
&lt;code&gt;lc-evaluation/src/verifier.rs&lt;/code&gt;,&lt;br&gt;
&lt;code&gt;lc-evaluation/src/judge_reliability.rs&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Project: &lt;a href="https://github.com/atliliw/langchainrust" rel="noopener noreferrer"&gt;https://github.com/atliliw/langchainrust&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  1. The Problem: Two 8s in an Agent — Which One?
&lt;/h2&gt;

&lt;p&gt;This is a common agent scenario:&lt;br&gt;
&lt;strong&gt;the agent produces several candidates at one step (answers, plans, or tool calls),&lt;br&gt;
then calls another LLM as a judge and picks one to continue.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The agent generates 3 candidates for the same task;&lt;/li&gt;
&lt;li&gt;the judge model scores them on a 1–10 scale;&lt;/li&gt;
&lt;li&gt;the agent picks the highest-scoring one to execute or return.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In agent systems this is called &lt;strong&gt;best-of-N / verifier-guided selection&lt;/strong&gt;.&lt;br&gt;
It shows up in:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;self-checking before returning the final answer;&lt;/li&gt;
&lt;li&gt;choosing among candidate tool calls;&lt;/li&gt;
&lt;li&gt;pruning branches in multi-step tasks;&lt;/li&gt;
&lt;li&gt;model or human review before high-risk actions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The common factor: &lt;strong&gt;the judge is itself an LLM.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;So the judge returns:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Candidate A: 8
Candidate B: 8
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Which one do you pick?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You can't. The integer 8 only tells you "it chose 8".&lt;br&gt;
It does not tell you "how confident it was".&lt;/p&gt;

&lt;p&gt;The distribution behind that same call looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Candidate A: 8 @ 0.90, 7 @ 0.08, 9 @ 0.02
Candidate B: 8 @ 0.50, 7 @ 0.40, 9 @ 0.10
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A is very sure about 8. B only picked 8 by a 0.50-to-0.40 margin over 7 —&lt;br&gt;
it had a 40% chance of actually wanting to say 7.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Two 8s are not the same 8.&lt;/strong&gt;&lt;/p&gt;


&lt;h2&gt;
  
  
  2. The Fix: Three Steps from an Integer to a Measurement
&lt;/h2&gt;

&lt;p&gt;You do not need a different model, and you do not need to retrain anything.&lt;br&gt;
You only need to take back the information the model already computed&lt;br&gt;
but does not return by default:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Constrain the output&lt;/strong&gt;: make the judge emit exactly one digit token (0–9).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Request the distribution&lt;/strong&gt;: turn on &lt;code&gt;logprobs&lt;/code&gt; and &lt;code&gt;top_logprobs&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compute two numbers&lt;/strong&gt;: the expected score (how good on average)
and the confidence (how concentrated the judgment is).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Same two candidates, two selection methods:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Candidate&lt;/th&gt;
&lt;th&gt;Judge output&lt;/th&gt;
&lt;th&gt;Integer only&lt;/th&gt;
&lt;th&gt;Verifier (expected / confidence)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;7.94 / 0.90&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;7.70 / 0.50&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Integer only&lt;/strong&gt;: a tie — you cannot decide.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verifier&lt;/strong&gt;: A is both better on average and more certain.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is the whole fix. In code it is one extra line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;OpenAIConfig&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;from_env_result&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;with_top_logprobs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The rest of the article unpacks this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Section 3: why a "direct scoring" model has a distribution at all;&lt;/li&gt;
&lt;li&gt;Section 4: what the &lt;code&gt;logprobs&lt;/code&gt; / &lt;code&gt;top_logprobs&lt;/code&gt; fields actually are;&lt;/li&gt;
&lt;li&gt;Section 5: the full derivation and what the integer throws away;&lt;/li&gt;
&lt;li&gt;Section 6: why a distribution alone is not enough (reliability constraints);&lt;/li&gt;
&lt;li&gt;Section 7: the verifier implementation and its three traps.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  3. First, a Misconception: This Is Not "Direct Scoring"
&lt;/h2&gt;

&lt;p&gt;The first reaction is usually: isn't the judge "just directly giving a score"?&lt;br&gt;
Why would there be a probability distribution?&lt;/p&gt;

&lt;p&gt;Because the model has no "scoring module" and no regression head that emits&lt;br&gt;
a continuous value. It does exactly what it does when writing a poem or code:&lt;br&gt;
&lt;strong&gt;predict the next token.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The prompt looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Score the answer below on a 0-9 scale.
Output exactly one digit.
Answer: ...
Score:
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model's job is to complete the token after &lt;code&gt;Score:&lt;/code&gt;.&lt;br&gt;
It computes one softmax over the whole vocabulary; &lt;code&gt;8&lt;/code&gt; has the highest&lt;br&gt;
probability, so it emits &lt;code&gt;8&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;So:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;What you see:      8
What the model did: computed a probability over the vocabulary,
                   8 ranked first, emitted 8
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;"Direct scoring" is an analogy to human behavior, not the model's operation.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Because of that, there is always a distribution behind the chosen token.&lt;br&gt;
The default API response simply does not return it.&lt;/p&gt;
&lt;h3&gt;
  
  
  Three layers to keep separate
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Content&lt;/th&gt;
&lt;th&gt;Who provides it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Choice&lt;/td&gt;
&lt;td&gt;It emitted 8&lt;/td&gt;
&lt;td&gt;The model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Confidence&lt;/td&gt;
&lt;td&gt;Whether 8 had probability 0.9 or 0.5&lt;/td&gt;
&lt;td&gt;The model (&lt;code&gt;logprobs&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;True quality&lt;/td&gt;
&lt;td&gt;How good an 8 actually is&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;The model does not measure this&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The third layer is the easiest to misread:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;code&gt;0.90&lt;/code&gt; does not mean "this answer has a 90% chance of being an 8-quality answer".&lt;br&gt;
It means "the model is very sure the next token is 8".&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The model gives you an &lt;strong&gt;opinion and the strength of that opinion&lt;/strong&gt;,&lt;br&gt;
not a calibrated quality measurement.&lt;/p&gt;
&lt;h3&gt;
  
  
  An analogy
&lt;/h3&gt;

&lt;p&gt;This is not a thermometer. It is a multiple-choice answer.&lt;/p&gt;

&lt;p&gt;You ask a person, "Rate this answer from 1 to 10." They say "8".&lt;br&gt;
You see the answer; you do not see whether they said it instantly&lt;br&gt;
or hesitated between 7 and 8.&lt;/p&gt;

&lt;p&gt;The model is different: its hesitation is digitized and readable.&lt;br&gt;
&lt;code&gt;0.90&lt;/code&gt; is an instant answer; &lt;code&gt;0.50&lt;/code&gt; is a hesitant one.&lt;/p&gt;


&lt;h2&gt;
  
  
  4. What &lt;code&gt;logprobs&lt;/code&gt; and &lt;code&gt;top_logprobs&lt;/code&gt; Actually Are
&lt;/h2&gt;

&lt;p&gt;Step 2 says "take the distribution back". This section explains exactly&lt;br&gt;
what you are taking, what the fields are called, and what the response looks like.&lt;/p&gt;
&lt;h3&gt;
  
  
  4.1 Start with tokens
&lt;/h3&gt;

&lt;p&gt;The model does not generate characters or words. It generates &lt;strong&gt;tokens&lt;/strong&gt;.&lt;br&gt;
A token can be:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8
10
points
(a space)
the
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At each step the model does one thing: &lt;strong&gt;given the text so far,&lt;br&gt;
predict the next token.&lt;/strong&gt; It computes a probability over the entire&lt;br&gt;
vocabulary (often ~100k tokens):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;next token "8"      probability 0.90
next token "7"      probability 0.08
next token "9"      probability 0.02
next token "points" probability 0.001
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  4.2 What &lt;code&gt;logprob&lt;/code&gt; is
&lt;/h3&gt;

&lt;p&gt;If the probability is &lt;code&gt;p&lt;/code&gt;, the &lt;code&gt;logprob&lt;/code&gt; is &lt;code&gt;ln(p)&lt;/code&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Probability p&lt;/th&gt;
&lt;th&gt;&lt;code&gt;logprob = ln(p)&lt;/code&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0.90&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-0.105&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.50&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-0.693&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.08&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-2.526&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.02&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-3.912&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.001&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-6.908&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Why logarithms instead of probabilities?&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Numerical stability&lt;/strong&gt;: multiplying a long chain of probabilities
underflows; adding their logs does not.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Training/inference parity&lt;/strong&gt;: models are trained on log probabilities
(cross-entropy).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Easy comparison&lt;/strong&gt;: closer to 0 means more likely; more negative means less.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Converting back is one step:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;probability = exp(logprob)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;exp(-0.105) ≈ 0.90&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.3 &lt;code&gt;logprobs&lt;/code&gt;: return the generated token's log probability
&lt;/h3&gt;

&lt;p&gt;In the request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"logprobs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This means:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Return the log probability of &lt;strong&gt;the token you actually generated&lt;/strong&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If the model generated &lt;code&gt;8&lt;/code&gt;, the response includes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"token"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"8"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"logprob"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;-0.105&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Without it you only see &lt;code&gt;8&lt;/code&gt; and have no idea how sure the model was.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.4 &lt;code&gt;top_logprobs&lt;/code&gt;: also return the N most likely candidates
&lt;/h3&gt;

&lt;p&gt;In the request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"logprobs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"top_logprobs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This means:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;At each generated position, in addition to the generated token,&lt;br&gt;
return the &lt;strong&gt;5 most likely candidate tokens&lt;/strong&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The response looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"token"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"8"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"logprob"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;-0.105&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"top_logprobs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"token"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"8"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"logprob"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;-0.105&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"token"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"7"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"logprob"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;-2.526&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"token"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"9"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"logprob"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;-3.912&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"token"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"6"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"logprob"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;-4.605&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"token"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"logprob"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;-5.298&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;The first entry is usually the generated token.&lt;/li&gt;
&lt;li&gt;The rest are the alternatives the model almost chose.&lt;/li&gt;
&lt;li&gt;This is exactly where data like &lt;code&gt;8@0.90, 7@0.08, 9@0.02&lt;/code&gt; comes from.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One naming quirk:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the request parameter &lt;code&gt;top_logprobs&lt;/code&gt; is a &lt;strong&gt;number&lt;/strong&gt; (how many candidates);&lt;/li&gt;
&lt;li&gt;the response field &lt;code&gt;top_logprobs&lt;/code&gt; is an &lt;strong&gt;array&lt;/strong&gt; (the candidate list).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In the code, &lt;code&gt;entry.top_logprobs&lt;/code&gt; refers to the response array.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.5 Three common misreadings
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;logprob&lt;/code&gt; is always negative&lt;/strong&gt; (probability ≤ 1, so &lt;code&gt;ln(p) ≤ 0&lt;/code&gt;).
Closer to 0 means more likely.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The top-k probabilities do not sum to 1.&lt;/strong&gt; Only the top N are returned;
the remaining tens of thousands of tokens are not in the response.
You must normalize when scoring.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is not "the probability that the answer is correct."&lt;/strong&gt;
It is only "how sure the model is that the next token is 8".
It does not make 8 a good answer.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  5. Full Derivation: What the Integer Throws Away
&lt;/h2&gt;

&lt;p&gt;Section 2 gave the result. This section fills in the derivation so you can&lt;br&gt;
check it yourself.&lt;/p&gt;
&lt;h3&gt;
  
  
  5.1 Two things are lost
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The distribution&lt;/strong&gt;: how much probability mass went to 7 and 9;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The confidence&lt;/strong&gt;: whether the 8 was a "90% sure" 8 or a "50% sure" 8.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Taking the argmax throws away both at the same instant.&lt;/p&gt;
&lt;h3&gt;
  
  
  5.2 Three candidates, two formulas
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Candidate&lt;/th&gt;
&lt;th&gt;Distribution (score @ probability)&lt;/th&gt;
&lt;th&gt;Expected&lt;/th&gt;
&lt;th&gt;Confidence&lt;/th&gt;
&lt;th&gt;argmax&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;
&lt;a href="mailto:8@0.90"&gt;8@0.90&lt;/a&gt;, &lt;a href="mailto:7@0.08"&gt;7@0.08&lt;/a&gt;, &lt;a href="mailto:9@0.02"&gt;9@0.02&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;7.94&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.90&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B&lt;/td&gt;
&lt;td&gt;
&lt;a href="mailto:8@0.50"&gt;8@0.50&lt;/a&gt;, &lt;a href="mailto:7@0.40"&gt;7@0.40&lt;/a&gt;, &lt;a href="mailto:9@0.10"&gt;9@0.10&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;7.70&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.50&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C&lt;/td&gt;
&lt;td&gt;
&lt;a href="mailto:9@0.50"&gt;9@0.50&lt;/a&gt;, &lt;a href="mailto:6@0.40"&gt;6@0.40&lt;/a&gt;, &lt;a href="mailto:7@0.10"&gt;7@0.10&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;7.60&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.50&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There are only two formulas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;expected score = Σ (probability_i × score_i)
confidence     = max probability / total probability
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(In the example the probabilities already sum to 1, so the denominator is 1.&lt;br&gt;
In real calls the top-k probabilities usually sum to less than 1,&lt;br&gt;
so you must normalize — Section 7 covers this.)&lt;/p&gt;
&lt;h3&gt;
  
  
  5.3 Three ways to select
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Method A: integer only (argmax)&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A = 8, B = 8, C = 9   → A and B are indistinguishable; C ranks first
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Method B: expected score&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A 7.94 &amp;gt; B 7.70 &amp;gt; C 7.60   → select A
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Method C: expected score + confidence&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A (7.94, 0.90)   → better on average and more certain
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The three methods disagree (A selects C; B and C select A).&lt;br&gt;
This is not a contrived coincidence:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;C emits 9, but only 50% of its mass is on 9 and 40% is on 6 —
&lt;strong&gt;argmax reads a wavering distribution as a high score&lt;/strong&gt;;&lt;/li&gt;
&lt;li&gt;A and B both emit 8, one with 0.90 and one with 0.50 —
&lt;strong&gt;argmax collapses two very different states into the same number&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The problem is not "whether the judge is accurate".&lt;br&gt;
It is that &lt;strong&gt;the information argmax discards is exactly the information&lt;br&gt;
you need to judge accuracy.&lt;/strong&gt;&lt;/p&gt;


&lt;h2&gt;
  
  
  6. Why a Distribution Alone Is Not Enough: Reliability Protocol
&lt;/h2&gt;

&lt;p&gt;The verifier fixes "scores are not comparable".&lt;br&gt;
It does not fix "the judge itself is biased".&lt;br&gt;
Even with a distribution, you cannot rely on a single judgment.&lt;/p&gt;
&lt;h3&gt;
  
  
  6.1 Four systematic biases
&lt;/h3&gt;

&lt;p&gt;These are not occasional mistakes. They are repeatable:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Position bias&lt;/strong&gt;: swap A and B and the verdict can flip.
A single judgment cannot support "A is better".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Same-family self-preference&lt;/strong&gt;: a model judging its own family's output
applies a different standard. Same-family scores cannot be compared across families.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Length and format bias&lt;/strong&gt;: longer answers, more headings, and more
"standard-answer" formatting raise scores regardless of content.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Score saturation&lt;/strong&gt;: most samples pile up at 8, 9, and 10.
The scale has almost no discriminative power.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There is also a statistical trap: &lt;strong&gt;raw agreement is inflated&lt;/strong&gt;.&lt;br&gt;
It includes "random chance agreement"; subtract that and the real&lt;br&gt;
consistency drops noticeably. Do not treat "85% agreement" as a conclusion —&lt;br&gt;
look at kappa first.&lt;/p&gt;
&lt;h3&gt;
  
  
  6.2 Four tools
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;lc-evaluation/src/judge_reliability.rs&lt;/code&gt; provides:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Order swap&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;fn&lt;/span&gt; &lt;span class="nf"&gt;order_swap_consistent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;original&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;swapped&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;match&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;original&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;swapped&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;AWins&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;BWins&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;AWins&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;BWins&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;AWins&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;BWins&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Tie&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the swapped round, "B wins" means the original A won.&lt;br&gt;
&lt;strong&gt;A candidate wins only if both rounds point to the same original answer.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cohen's kappa&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;kappa&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;cohen_kappa&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;judge_a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;judge_b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;How much agreement remains after subtracting chance agreement.&lt;br&gt;
Degenerate samples where everything is positive or everything is negative&lt;br&gt;
return &lt;code&gt;None&lt;/code&gt; instead of pretending to have a conclusion.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Same-family detection&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="nf"&gt;assert!&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;same_family&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"gpt-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"gpt-4o"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nf"&gt;assert!&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nf"&gt;same_family&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"gpt-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"claude-3"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Either reject same-family judging or mark it explicitly in the report.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Summary report&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="nc"&gt;ReliabilityReport&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;agreement&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;f64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;kappa&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;Option&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;f64&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;ties&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You need to know how many pairs became ties because the order disagreed.&lt;br&gt;
&lt;strong&gt;A high tie rate means the judge has no discriminative power;&lt;br&gt;
no matter how good the scores look, do not use them.&lt;/strong&gt;&lt;/p&gt;


&lt;h2&gt;
  
  
  7. Implementing the Verifier: Three Steps and One Function
&lt;/h2&gt;
&lt;h3&gt;
  
  
  Step 1: constrain the output to one token
&lt;/h3&gt;

&lt;p&gt;Do not let the model freely emit &lt;code&gt;"8/10"&lt;/code&gt;, &lt;code&gt;"8 points"&lt;/code&gt;, or &lt;code&gt;"eight"&lt;/code&gt;.&lt;br&gt;
Make it emit exactly one character from 0 to 9.&lt;/p&gt;

&lt;p&gt;Why this matters:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;multi-character tokens depend on the tokenizer; parsing becomes unstable;&lt;/li&gt;
&lt;li&gt;multi-digit scores (10) may be split into &lt;code&gt;1&lt;/code&gt; and &lt;code&gt;0&lt;/code&gt; by some providers;&lt;/li&gt;
&lt;li&gt;once the output format drifts, the whole scoring pipeline breaks.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;
  
  
  Step 2: request the top-k candidates and their log probabilities
&lt;/h3&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;OpenAIConfig&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;from_env_result&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;with_top_logprobs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;llm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;OpenAIChat&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The request body gains two fields:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"logprobs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"top_logprobs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(The field semantics and response shape are covered in Section 4;&lt;br&gt;
here we only care about how to turn them on.)&lt;/p&gt;

&lt;p&gt;They are off by default so that callers who do not need this feature&lt;br&gt;
keep a byte-identical request body.&lt;/p&gt;
&lt;h3&gt;
  
  
  Step 3: compress the distribution into a score
&lt;/h3&gt;

&lt;p&gt;The core function lives in &lt;code&gt;lc-evaluation/src/verifier.rs&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;fn&lt;/span&gt; &lt;span class="nf"&gt;score_from_logprobs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;token_logprobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nc"&gt;TokenLogprob&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;max_score&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;u32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;Option&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;VerificationScore&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;token_logprobs&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="kd"&gt;mut&lt;/span&gt; &lt;span class="n"&gt;mass&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;vec!&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mf"&gt;0.0f&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_score&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

        &lt;span class="c1"&gt;// The generated token is often included in the candidate list;&lt;/span&gt;
        &lt;span class="c1"&gt;// adding it separately would double-count its probability.&lt;/span&gt;
        &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="kd"&gt;mut&lt;/span&gt; &lt;span class="n"&gt;add&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;logprob&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;f64&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="kc"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;parse_score_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_score&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;mass&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;logprob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;chosen_listed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;top_logprobs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;alt&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;alt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;chosen_listed&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;logprob&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;alt&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;top_logprobs&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;alt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;alt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;logprob&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="c1"&gt;// The top-k probabilities do not sum to 1; normalize.&lt;/span&gt;
        &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;f64&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mass&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_finite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;f64&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mass&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="kt"&gt;f64&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;confidence&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mass&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cloned&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fold&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.0f&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;f64&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;VerificationScore&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;max_score&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;f64&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_score&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;confidence&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;confidence&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;clamp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ScoreSource&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Logprob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="kc"&gt;None&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There are three traps in this snippet. Get any one wrong and the score&lt;br&gt;
will lie to you at some boundary:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Double-counting the generated token&lt;/strong&gt;: the candidate list usually
contains the generated token. Adding it once directly and once while
iterating the list doubles its probability.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Non-digit tokens leaking in&lt;/strong&gt;: &lt;code&gt;The&lt;/code&gt;, &lt;code&gt;score&lt;/code&gt;, and &lt;code&gt;42&lt;/code&gt; must not
contribute score mass. &lt;code&gt;parse_score_token&lt;/code&gt; trims and rejects out-of-range values.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No normalization&lt;/strong&gt;: the top-k probabilities sum to less than 1.
Without dividing by the total, the expected score is systematically low.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Keep two outputs separate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;score&lt;/code&gt; answers "how good on average";&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;confidence&lt;/code&gt; answers "how concentrated this judgment is".&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;A score without a confidence is just the distribution flattened again.&lt;/strong&gt;&lt;/p&gt;


&lt;h2&gt;
  
  
  8. When to Use It, When Not To
&lt;/h2&gt;
&lt;h3&gt;
  
  
  8.1 Where to hook it into an agent
&lt;/h3&gt;

&lt;p&gt;Three common integration points, ordered by cost:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Hook point&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;th&gt;Recommendation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Before returning the final answer&lt;/td&gt;
&lt;td&gt;N candidates, return one&lt;/td&gt;
&lt;td&gt;On by default&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Before a tool call&lt;/td&gt;
&lt;td&gt;Candidate arguments/actions, execute one&lt;/td&gt;
&lt;td&gt;Only for high-risk tools&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mid multi-step task&lt;/td&gt;
&lt;td&gt;Prune branches, keep the best k&lt;/td&gt;
&lt;td&gt;Only at key branch points&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The corresponding framework call (&lt;code&gt;lc-evaluation&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;verifier&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LogprobVerifier&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;with_max_score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;with_top_logprobs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;best_of_n&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;verifier&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;candidates&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;println!&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="s2"&gt;"{i}: score={:.2} confidence={:.2} source={:?}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;confidence&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do &lt;strong&gt;not&lt;/strong&gt; run best-of-N at every step.&lt;br&gt;
An agent with M steps and N candidates per step costs&lt;br&gt;
M × N generations plus M × N judgments.&lt;br&gt;
Turn it on only where "choosing wrong is expensive".&lt;/p&gt;

&lt;p&gt;Cost-control rules:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;N defaults to 3, at most 5;&lt;/li&gt;
&lt;li&gt;skip scoring when candidates are nearly identical;&lt;/li&gt;
&lt;li&gt;keep the task prefix fixed to exploit prompt caching;&lt;/li&gt;
&lt;li&gt;enable the verifier only for samples that need selection, not for everything.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;
  
  
  8.2 Which scoring method to use
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;Preferred&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Tests / compilation / static checks available&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Execution verification&lt;/strong&gt; (hardest signal)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Logprobs available and you need selection&lt;/td&gt;
&lt;td&gt;Logprob verifier + best-of-N&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No logprobs&lt;/td&gt;
&lt;td&gt;Integer judge + order swap + kappa (fallback)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Very low budget&lt;/td&gt;
&lt;td&gt;Single judge + manual spot checks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;High-risk decision&lt;/td&gt;
&lt;td&gt;Execution verification + verifier + human review&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;h3&gt;
  
  
  8.3 Ties must be deterministic
&lt;/h3&gt;

&lt;p&gt;best-of-N has a subtle trap: &lt;code&gt;Iterator::max_by&lt;/code&gt; returns the &lt;strong&gt;last&lt;/strong&gt;&lt;br&gt;
element on a tie.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Wrong: returns the last one on ties&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;best_index&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;scores&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;max_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;partial_cmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unwrap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unwrap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same input can select different candidates on different runs.&lt;br&gt;
In regression tests this appears as an intermittent failure that is very&lt;br&gt;
hard to trace.&lt;/p&gt;

&lt;p&gt;The correct version uses a strict greater-than and keeps the earliest:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="kd"&gt;mut&lt;/span&gt; &lt;span class="n"&gt;best_index&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0usize&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;best_index&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;best_index&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Failure mode&lt;/strong&gt;: unstable best-of-N results; evaluation reports that&lt;br&gt;
cannot be reproduced for the same candidate set.&lt;/p&gt;
&lt;h3&gt;
  
  
  8.4 Fallback when logprobs are unavailable
&lt;/h3&gt;

&lt;p&gt;Not every provider exposes token log probabilities&lt;br&gt;
(for example, Anthropic's public API).&lt;br&gt;
The verifier needs two implementations:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="nc"&gt;LogprobVerifier&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="no"&gt;M&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;   &lt;span class="c1"&gt;// distribution available; use expectation&lt;/span&gt;
&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="nc"&gt;JudgeVerifier&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="no"&gt;M&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;     &lt;span class="c1"&gt;// no distribution; parse a single integer&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;JudgeVerifier&lt;/code&gt; fixes the confidence at 0.25:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="kc"&gt;Ok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;VerificationScore&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;f64&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;max_score&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;f64&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max_score&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;confidence&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.25&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ScoreSource&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Judge&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is not pretending it is as good as the logprob verifier.&lt;br&gt;
It &lt;strong&gt;explicitly marks the result as low-confidence&lt;/strong&gt; so downstream&lt;br&gt;
consumers can decide whether to trust it.&lt;/p&gt;

&lt;h3&gt;
  
  
  8.5 Ordering principle
&lt;/h3&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;If execution results exist, do not use model scores.&lt;/strong&gt;&lt;br&gt;
The verifier belongs where there is no objective signal:&lt;br&gt;
writing, analysis, option selection, open-ended QA.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  9. Failure Modes Quick Reference
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Root cause&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Scores are systematically high&lt;/td&gt;
&lt;td&gt;Generated token double-counted&lt;/td&gt;
&lt;td&gt;Deduplicate first&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;best-of-N changes between runs&lt;/td&gt;
&lt;td&gt;Tie resolved to the last element&lt;/td&gt;
&lt;td&gt;Strict greater-than&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Score parsing fails&lt;/td&gt;
&lt;td&gt;Multi-digit / free text&lt;/td&gt;
&lt;td&gt;Allow only one token&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scores exceed the scale&lt;/td&gt;
&lt;td&gt;No normalization&lt;/td&gt;
&lt;td&gt;Divide by total probability&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Same-family scores inflated&lt;/td&gt;
&lt;td&gt;Judge and candidate share a family&lt;/td&gt;
&lt;td&gt;Mark or reject&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A/B verdict flips&lt;/td&gt;
&lt;td&gt;No order swap&lt;/td&gt;
&lt;td&gt;Merge both orders&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kappa is &lt;code&gt;None&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;All-positive or all-negative sample&lt;/td&gt;
&lt;td&gt;Change the sample; do not fake a conclusion&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  10. Production Checklist
&lt;/h2&gt;

&lt;p&gt;If you are plugging model scores into an agent's automated decisions,&lt;br&gt;
do these eight things in order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Let the judge emit exactly one legal digit token (0–9).&lt;/li&gt;
&lt;li&gt;Turn on &lt;code&gt;logprobs&lt;/code&gt; and &lt;code&gt;top_logprobs&lt;/code&gt;; do not take only the argmax.&lt;/li&gt;
&lt;li&gt;Deduplicate the generated token before accumulating probability mass.&lt;/li&gt;
&lt;li&gt;Normalize by the total probability.&lt;/li&gt;
&lt;li&gt;Record both &lt;code&gt;score&lt;/code&gt; and &lt;code&gt;confidence&lt;/code&gt;, not just the score.&lt;/li&gt;
&lt;li&gt;Make best-of-N ties deterministic: strict greater-than, keep the earliest.&lt;/li&gt;
&lt;li&gt;When logprobs are unavailable, use &lt;code&gt;JudgeVerifier&lt;/code&gt; with confidence 0.25
and mark it explicitly.&lt;/li&gt;
&lt;li&gt;Run order swap, compute kappa, mark same-family judging, and cap N (default 3).&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  11. Conclusion
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;A judge model gives you an opinion. A verifier gives you a comparable measurement.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The difference is one sentence:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;an opinion is "it chose 8";&lt;/li&gt;
&lt;li&gt;a measurement is "with how much probability it chose 8, and how sure it was".&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you do only one thing, do this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Stop taking just the integer. Take the distribution, and compute&lt;br&gt;
the expected score and its confidence.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Project
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;GitHub: &lt;a href="https://github.com/atliliw/langchainrust" rel="noopener noreferrer"&gt;https://github.com/atliliw/langchainrust&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Docs: &lt;a href="https://atliliw.github.io/langchainrust/" rel="noopener noreferrer"&gt;https://atliliw.github.io/langchainrust/&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Next: "From the Most Likely Token to the Expected Value: Building a Verifier"&lt;/em&gt;&lt;br&gt;
&lt;em&gt;— the implementation retrospective: the two small bugs that taught us the hard lessons.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>mcp</category>
      <category>agents</category>
      <category>rag</category>
    </item>
    <item>
      <title>langchain-rust: Build LLM apps with Ollama + local models in pure Rust — no Python needed</title>
      <dc:creator>laolei</dc:creator>
      <pubDate>Mon, 03 Aug 2026 09:24:01 +0000</pubDate>
      <link>https://dev.to/laolei/langchain-rust-build-llm-apps-with-ollama-local-models-in-pure-rust-no-python-needed-52jl</link>
      <guid>https://dev.to/laolei/langchain-rust-build-llm-apps-with-ollama-local-models-in-pure-rust-no-python-needed-52jl</guid>
      <description>&lt;p&gt;If you're running local models through Ollama and tired of Python's overhead, check out &lt;a href="https://github.com/atliliw/langchainrust" rel="noopener noreferrer"&gt;langchain-rust&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;It's a full LLM framework in pure Rust that works great with local models:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ollama support&lt;/strong&gt; — first-class integration with tool calling, vision, and streaming&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;9 vector store backends&lt;/strong&gt; — InMemory, SQLite, Qdrant, ChromaDB, Redis, PGVector, MongoDB, Pinecone, FileVectorStore&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BM25 keyword search&lt;/strong&gt; — with Chinese/English tokenization, no external dependency&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hybrid retrieval&lt;/strong&gt; — BM25 + Vector with RRF fusion for better recall&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GraphRAG&lt;/strong&gt; — Knowledge graph construction + community detection, all local&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CorrectiveRAG&lt;/strong&gt; — Self-correcting retrieval with hallucination detection&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code Interpreter&lt;/strong&gt; — LocalSandbox (subprocess), E2B cloud, or WASM sandbox&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LocalEmbeddings&lt;/strong&gt; — Run embeddings without calling an API&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Plus: LangGraph workflows, MCP client/server, 7 memory types, guardrails, and 12+ built-in tools.&lt;/p&gt;

&lt;p&gt;Single binary, no virtualenv, no pip conflicts. Just &lt;code&gt;cargo add langchainrust&lt;/code&gt; and go.&lt;/p&gt;

&lt;p&gt;GitHub: &lt;a href="https://github.com/atliliw/langchainrust" rel="noopener noreferrer"&gt;https://github.com/atliliw/langchainrust&lt;/a&gt;&lt;br&gt;
Docs: &lt;a href="https://docs.rs/langchainrust" rel="noopener noreferrer"&gt;https://docs.rs/langchainrust&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>rag</category>
      <category>rust</category>
    </item>
  </channel>
</rss>
