DEV Community

laolei
laolei

Posted on

Is Your Agent's LLM Judge Unreliable? Use Logprobs to Make Scores Comparable

Is Your Agent's LLM Judge Unreliable? Use Logprobs to Make Scores Comparable

One sentence: the model gives you an integer; a decision needs a measurement.

This article gives the fix first (three steps), then explains why it works,
and ends with the engineering constraints you cannot skip.

The code below comes from the real implementation in langchainrust 0.30.0:
lc-core/src/language_models/chat.rs,
lc-evaluation/src/verifier.rs,
lc-evaluation/src/judge_reliability.rs.

Project: https://github.com/atliliw/langchainrust


1. The Problem: Two 8s in an Agent — Which One?

This is a common agent scenario:
the agent produces several candidates at one step (answers, plans, or tool calls),
then calls another LLM as a judge and picks one to continue.

  • The agent generates 3 candidates for the same task;
  • the judge model scores them on a 1–10 scale;
  • the agent picks the highest-scoring one to execute or return.

In agent systems this is called best-of-N / verifier-guided selection.
It shows up in:

  • self-checking before returning the final answer;
  • choosing among candidate tool calls;
  • pruning branches in multi-step tasks;
  • model or human review before high-risk actions.

The common factor: the judge is itself an LLM.

So the judge returns:

Candidate A: 8
Candidate B: 8
Enter fullscreen mode Exit fullscreen mode

Which one do you pick?

You can't. The integer 8 only tells you "it chose 8".
It does not tell you "how confident it was".

The distribution behind that same call looks like this:

Candidate A: 8 @ 0.90, 7 @ 0.08, 9 @ 0.02
Candidate B: 8 @ 0.50, 7 @ 0.40, 9 @ 0.10
Enter fullscreen mode Exit fullscreen mode

A is very sure about 8. B only picked 8 by a 0.50-to-0.40 margin over 7 —
it had a 40% chance of actually wanting to say 7.

Two 8s are not the same 8.


2. The Fix: Three Steps from an Integer to a Measurement

You do not need a different model, and you do not need to retrain anything.
You only need to take back the information the model already computed
but does not return by default:

  1. Constrain the output: make the judge emit exactly one digit token (0–9).
  2. Request the distribution: turn on logprobs and top_logprobs.
  3. Compute two numbers: the expected score (how good on average) and the confidence (how concentrated the judgment is).

Same two candidates, two selection methods:

Candidate Judge output Integer only Verifier (expected / confidence)
A 8 8 7.94 / 0.90
B 8 8 7.70 / 0.50
  • Integer only: a tie — you cannot decide.
  • Verifier: A is both better on average and more certain.

That is the whole fix. In code it is one extra line:

let config = OpenAIConfig::from_env_result()?.with_top_logprobs(5);
Enter fullscreen mode Exit fullscreen mode

The rest of the article unpacks this:

  • Section 3: why a "direct scoring" model has a distribution at all;
  • Section 4: what the logprobs / top_logprobs fields actually are;
  • Section 5: the full derivation and what the integer throws away;
  • Section 6: why a distribution alone is not enough (reliability constraints);
  • Section 7: the verifier implementation and its three traps.

3. First, a Misconception: This Is Not "Direct Scoring"

The first reaction is usually: isn't the judge "just directly giving a score"?
Why would there be a probability distribution?

Because the model has no "scoring module" and no regression head that emits
a continuous value. It does exactly what it does when writing a poem or code:
predict the next token.

The prompt looks like this:

Score the answer below on a 0-9 scale.
Output exactly one digit.
Answer: ...
Score:
Enter fullscreen mode Exit fullscreen mode

The model's job is to complete the token after Score:.
It computes one softmax over the whole vocabulary; 8 has the highest
probability, so it emits 8.

So:

What you see:      8
What the model did: computed a probability over the vocabulary,
                   8 ranked first, emitted 8
Enter fullscreen mode Exit fullscreen mode

"Direct scoring" is an analogy to human behavior, not the model's operation.

Because of that, there is always a distribution behind the chosen token.
The default API response simply does not return it.

Three layers to keep separate

Layer Content Who provides it
Choice It emitted 8 The model
Confidence Whether 8 had probability 0.9 or 0.5 The model (logprobs)
True quality How good an 8 actually is The model does not measure this

The third layer is the easiest to misread:

0.90 does not mean "this answer has a 90% chance of being an 8-quality answer".
It means "the model is very sure the next token is 8".

The model gives you an opinion and the strength of that opinion,
not a calibrated quality measurement.

An analogy

This is not a thermometer. It is a multiple-choice answer.

You ask a person, "Rate this answer from 1 to 10." They say "8".
You see the answer; you do not see whether they said it instantly
or hesitated between 7 and 8.

The model is different: its hesitation is digitized and readable.
0.90 is an instant answer; 0.50 is a hesitant one.


4. What logprobs and top_logprobs Actually Are

Step 2 says "take the distribution back". This section explains exactly
what you are taking, what the fields are called, and what the response looks like.

4.1 Start with tokens

The model does not generate characters or words. It generates tokens.
A token can be:

8
10
points
(a space)
the
Enter fullscreen mode Exit fullscreen mode

At each step the model does one thing: given the text so far,
predict the next token.
It computes a probability over the entire
vocabulary (often ~100k tokens):

next token "8"      probability 0.90
next token "7"      probability 0.08
next token "9"      probability 0.02
next token "points" probability 0.001
...
Enter fullscreen mode Exit fullscreen mode

4.2 What logprob is

If the probability is p, the logprob is ln(p).

Probability p logprob = ln(p)
0.90 -0.105
0.50 -0.693
0.08 -2.526
0.02 -3.912
0.001 -6.908

Why logarithms instead of probabilities?

  1. Numerical stability: multiplying a long chain of probabilities underflows; adding their logs does not.
  2. Training/inference parity: models are trained on log probabilities (cross-entropy).
  3. Easy comparison: closer to 0 means more likely; more negative means less.

Converting back is one step:

probability = exp(logprob)
Enter fullscreen mode Exit fullscreen mode

exp(-0.105) ≈ 0.90.

4.3 logprobs: return the generated token's log probability

In the request:

{ "logprobs": true }
Enter fullscreen mode Exit fullscreen mode

This means:

Return the log probability of the token you actually generated.

If the model generated 8, the response includes:

{ "token": "8", "logprob": -0.105 }
Enter fullscreen mode Exit fullscreen mode

Without it you only see 8 and have no idea how sure the model was.

4.4 top_logprobs: also return the N most likely candidates

In the request:

{ "logprobs": true, "top_logprobs": 5 }
Enter fullscreen mode Exit fullscreen mode

This means:

At each generated position, in addition to the generated token,
return the 5 most likely candidate tokens.

The response looks like this:

{
  "token": "8",
  "logprob": -0.105,
  "top_logprobs": [
    { "token": "8", "logprob": -0.105 },
    { "token": "7", "logprob": -2.526 },
    { "token": "9", "logprob": -3.912 },
    { "token": "6", "logprob": -4.605 },
    { "token": "1", "logprob": -5.298 }
  ]
}
Enter fullscreen mode Exit fullscreen mode
  • The first entry is usually the generated token.
  • The rest are the alternatives the model almost chose.
  • This is exactly where data like 8@0.90, 7@0.08, 9@0.02 comes from.

One naming quirk:

  • the request parameter top_logprobs is a number (how many candidates);
  • the response field top_logprobs is an array (the candidate list).

In the code, entry.top_logprobs refers to the response array.

4.5 Three common misreadings

  1. logprob is always negative (probability ≤ 1, so ln(p) ≤ 0). Closer to 0 means more likely.
  2. The top-k probabilities do not sum to 1. Only the top N are returned; the remaining tens of thousands of tokens are not in the response. You must normalize when scoring.
  3. It is not "the probability that the answer is correct." It is only "how sure the model is that the next token is 8". It does not make 8 a good answer.

5. Full Derivation: What the Integer Throws Away

Section 2 gave the result. This section fills in the derivation so you can
check it yourself.

5.1 Two things are lost

  1. The distribution: how much probability mass went to 7 and 9;
  2. The confidence: whether the 8 was a "90% sure" 8 or a "50% sure" 8.

Taking the argmax throws away both at the same instant.

5.2 Three candidates, two formulas

Candidate Distribution (score @ probability) Expected Confidence argmax
A 8@0.90, 7@0.08, 9@0.02 7.94 0.90 8
B 8@0.50, 7@0.40, 9@0.10 7.70 0.50 8
C 9@0.50, 6@0.40, 7@0.10 7.60 0.50 9

There are only two formulas:

expected score = Σ (probability_i × score_i)
confidence     = max probability / total probability
Enter fullscreen mode Exit fullscreen mode

(In the example the probabilities already sum to 1, so the denominator is 1.
In real calls the top-k probabilities usually sum to less than 1,
so you must normalize — Section 7 covers this.)

5.3 Three ways to select

Method A: integer only (argmax)

A = 8, B = 8, C = 9   → A and B are indistinguishable; C ranks first
Enter fullscreen mode Exit fullscreen mode

Method B: expected score

A 7.94 > B 7.70 > C 7.60   → select A
Enter fullscreen mode Exit fullscreen mode

Method C: expected score + confidence

A (7.94, 0.90)   → better on average and more certain
Enter fullscreen mode Exit fullscreen mode

The three methods disagree (A selects C; B and C select A).
This is not a contrived coincidence:

  • C emits 9, but only 50% of its mass is on 9 and 40% is on 6 — argmax reads a wavering distribution as a high score;
  • A and B both emit 8, one with 0.90 and one with 0.50 — argmax collapses two very different states into the same number.

The problem is not "whether the judge is accurate".
It is that the information argmax discards is exactly the information
you need to judge accuracy.


6. Why a Distribution Alone Is Not Enough: Reliability Protocol

The verifier fixes "scores are not comparable".
It does not fix "the judge itself is biased".
Even with a distribution, you cannot rely on a single judgment.

6.1 Four systematic biases

These are not occasional mistakes. They are repeatable:

  1. Position bias: swap A and B and the verdict can flip. A single judgment cannot support "A is better".
  2. Same-family self-preference: a model judging its own family's output applies a different standard. Same-family scores cannot be compared across families.
  3. Length and format bias: longer answers, more headings, and more "standard-answer" formatting raise scores regardless of content.
  4. Score saturation: most samples pile up at 8, 9, and 10. The scale has almost no discriminative power.

There is also a statistical trap: raw agreement is inflated.
It includes "random chance agreement"; subtract that and the real
consistency drops noticeably. Do not treat "85% agreement" as a conclusion —
look at kappa first.

6.2 Four tools

lc-evaluation/src/judge_reliability.rs provides:

Order swap

pub fn order_swap_consistent(original: Verdict, swapped: Verdict) -> Verdict {
    match (original, swapped) {
        (Verdict::AWins, Verdict::BWins) => Verdict::AWins,
        (Verdict::BWins, Verdict::AWins) => Verdict::BWins,
        _ => Verdict::Tie,
    }
}
Enter fullscreen mode Exit fullscreen mode

In the swapped round, "B wins" means the original A won.
A candidate wins only if both rounds point to the same original answer.

Cohen's kappa

let kappa = cohen_kappa(&judge_a, &judge_b);
Enter fullscreen mode Exit fullscreen mode

How much agreement remains after subtracting chance agreement.
Degenerate samples where everything is positive or everything is negative
return None instead of pretending to have a conclusion.

Same-family detection

assert!(same_family("gpt-5", "gpt-4o"));
assert!(!same_family("gpt-5", "claude-3"));
Enter fullscreen mode Exit fullscreen mode

Either reject same-family judging or mark it explicitly in the report.

Summary report

pub struct ReliabilityReport {
    pub n: usize,
    pub agreement: f64,
    pub kappa: Option<f64>,
    pub ties: usize,
}
Enter fullscreen mode Exit fullscreen mode

You need to know how many pairs became ties because the order disagreed.
A high tie rate means the judge has no discriminative power;
no matter how good the scores look, do not use them.


7. Implementing the Verifier: Three Steps and One Function

Step 1: constrain the output to one token

Do not let the model freely emit "8/10", "8 points", or "eight".
Make it emit exactly one character from 0 to 9.

Why this matters:

  • multi-character tokens depend on the tokenizer; parsing becomes unstable;
  • multi-digit scores (10) may be split into 1 and 0 by some providers;
  • once the output format drifts, the whole scoring pipeline breaks.

Step 2: request the top-k candidates and their log probabilities

let config = OpenAIConfig::from_env_result()?.with_top_logprobs(5);
let llm = OpenAIChat::new(config);
Enter fullscreen mode Exit fullscreen mode

The request body gains two fields:

{ "logprobs": true, "top_logprobs": 5 }
Enter fullscreen mode Exit fullscreen mode

(The field semantics and response shape are covered in Section 4;
here we only care about how to turn them on.)

They are off by default so that callers who do not need this feature
keep a byte-identical request body.

Step 3: compress the distribution into a score

The core function lives in lc-evaluation/src/verifier.rs:

pub fn score_from_logprobs(
    token_logprobs: &[TokenLogprob],
    max_score: u32,
) -> Option<VerificationScore> {
    for entry in token_logprobs {
        let mut mass = vec![0.0f64; (max_score + 1) as usize];

        // The generated token is often included in the candidate list;
        // adding it separately would double-count its probability.
        let mut add = |token: &str, logprob: f64| {
            if let Some(score) = parse_score_token(token, max_score) {
                mass[score as usize] += logprob.exp();
            }
        };
        let chosen_listed = entry.top_logprobs.iter().any(|alt| alt.token == entry.token);
        if !chosen_listed {
            add(&entry.token, entry.logprob);
        }
        for alt in &entry.top_logprobs {
            add(&alt.token, alt.logprob);
        }

        // The top-k probabilities do not sum to 1; normalize.
        let total: f64 = mass.iter().sum();
        if total <= 0.0 || !total.is_finite() {
            continue;
        }

        let expected: f64 = mass.iter().enumerate()
            .map(|(score, p)| score as f64 * (p / total))
            .sum();
        let confidence = mass.iter().cloned().fold(0.0f64, f64::max) / total;

        return Some(VerificationScore {
            score: expected,
            max_score: f64::from(max_score),
            confidence: confidence.clamp(0.0, 1.0),
            source: ScoreSource::Logprob,
        });
    }
    None
}
Enter fullscreen mode Exit fullscreen mode

There are three traps in this snippet. Get any one wrong and the score
will lie to you at some boundary:

  1. Double-counting the generated token: the candidate list usually contains the generated token. Adding it once directly and once while iterating the list doubles its probability.
  2. Non-digit tokens leaking in: The, score, and 42 must not contribute score mass. parse_score_token trims and rejects out-of-range values.
  3. No normalization: the top-k probabilities sum to less than 1. Without dividing by the total, the expected score is systematically low.

Keep two outputs separate:

  • score answers "how good on average";
  • confidence answers "how concentrated this judgment is".

A score without a confidence is just the distribution flattened again.


8. When to Use It, When Not To

8.1 Where to hook it into an agent

Three common integration points, ordered by cost:

Hook point What it does Recommendation
Before returning the final answer N candidates, return one On by default
Before a tool call Candidate arguments/actions, execute one Only for high-risk tools
Mid multi-step task Prune branches, keep the best k Only at key branch points

The corresponding framework call (lc-evaluation):

let verifier = LogprobVerifier::new(llm)
    .with_max_score(9)
    .with_top_logprobs(5);

let result = best_of_n(&verifier, task, &candidates).await?;

for (i, score) in result.scores.iter().enumerate() {
    println!(
        "{i}: score={:.2} confidence={:.2} source={:?}",
        score.score, score.confidence, score.source
    );
}
Enter fullscreen mode Exit fullscreen mode

Do not run best-of-N at every step.
An agent with M steps and N candidates per step costs
M × N generations plus M × N judgments.
Turn it on only where "choosing wrong is expensive".

Cost-control rules:

  • N defaults to 3, at most 5;
  • skip scoring when candidates are nearly identical;
  • keep the task prefix fixed to exploit prompt caching;
  • enable the verifier only for samples that need selection, not for everything.

8.2 Which scoring method to use

Scenario Preferred
Tests / compilation / static checks available Execution verification (hardest signal)
Logprobs available and you need selection Logprob verifier + best-of-N
No logprobs Integer judge + order swap + kappa (fallback)
Very low budget Single judge + manual spot checks
High-risk decision Execution verification + verifier + human review

8.3 Ties must be deterministic

best-of-N has a subtle trap: Iterator::max_by returns the last
element on a tie.

// Wrong: returns the last one on ties
let (best_index, _) = scores
    .iter()
    .enumerate()
    .max_by(|(_, a), (_, b)| a.score.partial_cmp(&b.score).unwrap())
    .unwrap();
Enter fullscreen mode Exit fullscreen mode

The same input can select different candidates on different runs.
In regression tests this appears as an intermittent failure that is very
hard to trace.

The correct version uses a strict greater-than and keeps the earliest:

let mut best_index = 0usize;
for (i, score) in scores.iter().enumerate() {
    if score.score > scores[best_index].score {
        best_index = i;
    }
}
Enter fullscreen mode Exit fullscreen mode

Failure mode: unstable best-of-N results; evaluation reports that
cannot be reproduced for the same candidate set.

8.4 Fallback when logprobs are unavailable

Not every provider exposes token log probabilities
(for example, Anthropic's public API).
The verifier needs two implementations:

pub struct LogprobVerifier<M> { ... }   // distribution available; use expectation
pub struct JudgeVerifier<M> { ... }     // no distribution; parse a single integer
Enter fullscreen mode Exit fullscreen mode

JudgeVerifier fixes the confidence at 0.25:

Ok(VerificationScore {
    score: f64::from(score),
    max_score: f64::from(self.max_score),
    confidence: 0.25,
    source: ScoreSource::Judge,
})
Enter fullscreen mode Exit fullscreen mode

This is not pretending it is as good as the logprob verifier.
It explicitly marks the result as low-confidence so downstream
consumers can decide whether to trust it.

8.5 Ordering principle

If execution results exist, do not use model scores.
The verifier belongs where there is no objective signal:
writing, analysis, option selection, open-ended QA.


9. Failure Modes Quick Reference

Symptom Root cause Fix
Scores are systematically high Generated token double-counted Deduplicate first
best-of-N changes between runs Tie resolved to the last element Strict greater-than
Score parsing fails Multi-digit / free text Allow only one token
Scores exceed the scale No normalization Divide by total probability
Same-family scores inflated Judge and candidate share a family Mark or reject
A/B verdict flips No order swap Merge both orders
Kappa is None All-positive or all-negative sample Change the sample; do not fake a conclusion

10. Production Checklist

If you are plugging model scores into an agent's automated decisions,
do these eight things in order:

  1. Let the judge emit exactly one legal digit token (0–9).
  2. Turn on logprobs and top_logprobs; do not take only the argmax.
  3. Deduplicate the generated token before accumulating probability mass.
  4. Normalize by the total probability.
  5. Record both score and confidence, not just the score.
  6. Make best-of-N ties deterministic: strict greater-than, keep the earliest.
  7. When logprobs are unavailable, use JudgeVerifier with confidence 0.25 and mark it explicitly.
  8. Run order swap, compute kappa, mark same-family judging, and cap N (default 3).

11. Conclusion

A judge model gives you an opinion. A verifier gives you a comparable measurement.

The difference is one sentence:

  • an opinion is "it chose 8";
  • a measurement is "with how much probability it chose 8, and how sure it was".

If you do only one thing, do this:

Stop taking just the integer. Take the distribution, and compute
the expected score and its confidence.


Project


Next: "From the Most Likely Token to the Expected Value: Building a Verifier"
— the implementation retrospective: the two small bugs that taught us the hard lessons.

Top comments (4)

Collapse
 
mateo_ruiz_6992b1fce47843 profile image
Mateo Ruiz •

The distinction between token confidence and answer quality is the most important part here. A judge can be highly confident in a poor evaluation, so logprobs make scores more informative, but they don't make the judge inherently reliable.

I'd add one more production safeguard: validate the verifier against a labeled set of human-reviewed examples and track calibration separately across task categories. A judge might be consistent on structured outputs but unreliable on open-ended reasoning, and a single aggregate confidence score can hide that difference.

Also, I like the recommendation to prefer execution results whenever they're available. If a candidate can be checked through a test, schema validation, or an actual tool outcome, that evidence should take precedence over another model's opinion.

The goal shouldn't be to make the judge sound more certain. It should be to establish when its judgment is dependable enough to influence an agent's next action.

Collapse
 
laolei profile image
laolei •

Thanks — this is a fair and important distinction, and I agree with all three points.

On calibration, the missing piece in the article is a procedure, not just a caveat:

  • Keep a human-labeled set per task category (structured extraction, closed QA, open-ended reasoning) and evaluate the verifier against it, not against the judge's own output.
  • Track calibration per category, not as one aggregate number. A reliability diagram plus ECE/Brier is ideal; a coverage-risk curve is the practical minimum: at a given confidence threshold, what fraction of actions are auto-approved, and what is their error rate?
  • Treat confidence as a routing signal, not a quality score. Auto-approve only above a threshold calibrated to a target precision; below it, escalate to execution verification or human review.
  • Where execution feedback exists (tests, schema validation, real tool results), validate the verifier against those outcomes and prefer them.

The goal is exactly what you said: decide when the judge's judgment is reliable
enough to affect the agent's next action. Logprobs make the signal more
informative; calibration is what makes it actionable.

Collapse
 
hannie8 profile image
Hai Anh •

G-Eval (Liu et al., 2023) used the same move, weighting each rating token by its probability, so there's prior evidence here. Their ablation is worth a look though: with GPT-4 the weighted version nudged average Spearman up (0.502 to 0.514) while Kendall's tau went down (0.446 to 0.418), which reads to me like the weighting mostly breaks ties. That fits your two-8s example nicely, and it backs up @mateo_ruiz_6992b1fce47843's point that a human-labeled set is still what tells you whether the judge is right.

Collapse
 
laolei profile image
laolei •

Thanks — good catch, that's G-Eval (Liu et al. 2023), I should have cited it.

Your Spearman-up / Kendall-down read is right: weighting mostly breaks ties,
it doesn't reorder. That maps straight onto my two-8s example — the win is
disambiguation, not better ranking.

So honestly: weighting buys comparability at the boundary, not accuracy. As
you and Mateo said, only a human-labeled set tells you whether the judge is
actually right. I'll fold G-Eval in and sharpen that caveat.