LLM Judge Under Test: Direct Scoring vs Pairwise Comparison | Agent Lab Journal

  Agent Lab Journal

    Guides
    Glossary

Evaluation engineering

LLM Judge Under Test: Direct Scoring vs Pairwise Comparison

Intermediate · 60-minute read · Reproducible evaluation protocol

An automatic judge can consistently prefer the first answer, the longer answer, or an answer produced by its own model family—even when those properties have nothing to do with quality. This guide treats the judge as an instrument that must itself be evaluated. You will compare direct scoring with pairwise comparison, measure agreement with human decisions, and run a separate answer-swap experiment to expose positional bias.

What you will produce

The result is not a single leaderboard number. It is an evidence package containing:

• A direct score for every candidate answer under a fixed evaluation rubric.

• A pairwise preference for the same answer pairs, including an explicit tie option.

• Human reference judgments collected independently of the automatic judge.

• Agreement measurements between each automatic strategy and the human reference.

• A dedicated position-bias test in which every pair is judged again after the answers are swapped.

• Failure logs, invalid-output counts, uncertainty intervals, and enough raw data to reproduce every summary.

The protocol deliberately separates data generation, judging, and analysis. You may use any model or serving platform, but the stored records must follow the schemas below. This keeps the experiment repeatable without requiring shared credentials or a particular vendor.

Why the judge needs a trial of its own

An LLM judge converts a task, candidate answer, and evaluation instructions into a score or preference. It is attractive because it is faster and cheaper than repeated expert review. Its output, however, is another model prediction rather than ground truth.

Several unrelated effects can produce a convincing but misleading result:

• Order: the judge may prefer answer A because it appears first, or answer B because it appears last.

• Length: detail, repetition, and confident prose can be mistaken for completeness or correctness.

• Self-preference: a judge may favor wording, structure, or conventions associated with its own model family.

• Scale drift: a “4” in one record may not represent the same standard as a “4” later in the run.

• Rubric leakage: the judge may reward visible keywords from the rubric without checking whether the answer actually satisfies them.

• Task shortcuts: style or surface form may become a proxy for quality when the underlying claim is difficult to verify.

The right question is therefore not “Which strategy produces cleaner numbers?” It is “Which strategy best reproduces the human decision relevant to this task, with acceptable stability and known failure modes?”

The two strategies

Strategy 1: direct scoring

In direct scoring, the judge sees one answer at a time and assigns criterion scores, such as correctness from 1 to 5 and relevance from 1 to 5. A weighted total can then be calculated.

Direct scoring is useful when you need an absolute threshold, criterion-level diagnostics, or more than two candidates. It also makes operational rules easy to express: for example, “reject any answer with correctness below 3.”

Its main weakness is calibration. Scores can cluster near the top, drift between batches, and depend on the judge’s interpretation of scale labels. A one-point difference may look meaningful even when the judge cannot reproduce it.

Strategy 2: pairwise comparison

In pairwise comparison, the judge sees two answers to the same task and chooses A, B, or tie. The decision is relative: which answer better satisfies the rubric?

Pairwise decisions often reduce the burden of maintaining a stable absolute scale. They can still be biased by presentation order, verbosity, shared errors, or recognizable model style. They also become expensive when many candidates require many comparisons.

      What each strategy exposes and hides


          Property
          Direct scoring
          Pairwise comparison




          Primary output
          Criterion scores and total
          A, B, or tie


          Human comparison
          Scores or preference derived from score differences
          Direct preference agreement


          Calibration burden
          High
          Lower, but not absent


          Order sensitivity
          Possible through surrounding context or batch layout
          Directly testable by swapping A and B


          Diagnostic detail
          Strong if criterion scores are reliable
          Limited unless reasons are coded separately


          Cost for many candidates
          Approximately linear
          Potentially quadratic for all pairs

Concrete case: two support-answer systems

Suppose two systems, Alpha and Beta, answer the same set of internal support questions. The experiment asks which judging strategy better matches trained human reviewers.

The unit of analysis is one task with exactly two candidate answers:

{
  "item_id": "support_0042",
  "task": "Explain how to rotate an expired API key safely.",
  "reference_context": "Approved task-specific context, if available.",
  "answer_alpha": "...",
  "answer_beta": "...",
  "alpha_length_words": 126,
  "beta_length_words": 81
}

Use real items from the target workload. Do not fill the dataset with convenient examples written solely for the evaluation. Remove records where one answer is missing, corrupted, or generated from a different task version.

For a first study, aim for enough items to cover important task categories and known edge cases. Do not choose the final count only because it produces a desired significance result. Record the sampling rule before running the judge.

Fixed rubric

Use criteria that humans and the automatic judge can apply to the same evidence:

• Correctness: claims are consistent with the supplied task and reference context.

• Relevance: the answer addresses the user’s actual request.

• Completeness: required actions, caveats, and consequences are covered.

• Safety: the answer avoids dangerous, unauthorized, or irreversible instructions.

• Clarity: the answer is understandable without unnecessary repetition.

Keep style subordinate to task quality. Length is not a criterion unless the task explicitly specifies a length constraint. A correct concise answer must be allowed to beat a longer answer.

Design the experiment before calling a model

1. Freeze the evaluation unit

Store an immutable item identifier, exact task text, approved context, both answers, system identifiers, and generation metadata that you are permitted to retain. Hashing the task and answers can help detect accidental edits between runs.

2. Separate development and evaluation items

Use a small development set to repair prompts, parsers, and schemas. Once the protocol is frozen, run it on a separate evaluation set. Repeatedly changing the prompt after seeing evaluation failures turns the evaluation set into a development set.

3. Blind identities

Apply blind evaluation: replace product and model names with neutral labels. Remove phrases that disclose the generating system when they are not part of the user-visible answer. Do not rewrite ordinary style differences, because those are part of the actual output.

4. Predefine ties

A tie should mean that neither answer is meaningfully better under the rubric. It must not be a parser fallback or an escape hatch for uncertainty. Decide whether a small direct-score difference becomes a tie before examining the outcomes.

5. Randomize independently

Use a recorded random seed to decide which system appears as A in the first pairwise pass. Do not place Alpha first for every item. The swap pass must then reverse the exact first-pass order.

6. Fix the judge configuration

Record the judge identifier, model version if exposed, prompt version, decoding settings, response format, retry rule, timeout, and execution date. Low randomness may improve repeatability, but it does not remove systematic bias.

Data files

A simple directory layout is enough:

evaluation/
├── items.jsonl
├── human_labels.csv
├── direct_scores.jsonl
├── pairwise_original.jsonl
├── pairwise_swapped.jsonl
├── invalid_outputs.jsonl
├── protocol.yaml
└── analyze.py

The protocol file captures decisions that must not change during the run:

study_id: support-alpha-beta-v1
item_id_field: item_id
rubric_version: support-rubric-v1
judge_prompt_version: judge-v1
direct_scale:
  minimum: 1
  maximum: 5
  criteria:
    - correctness
    - relevance
    - completeness
    - safety
    - clarity
pairwise_labels: [A, B, TIE]
direct_tie_margin: 0.25
missing_policy: exclude_from_agreement_and_report
random_seed: 240719
retries:
  malformed_output: 1
  transient_error: 2
position_test:
  enabled: true
  swap_every_pair: true
human_review:
  reviewers_per_item: 2
  adjudicate_disagreement: true

The numbers above are configuration examples, not claimed best values. Choose and document values appropriate to your task before evaluation.

Judge prompts and output contracts

Direct-scoring prompt

You are evaluating one candidate answer.

Use only the TASK, REFERENCE CONTEXT, RUBRIC, and ANSWER below.
Do not reward length, confidence, formatting, or stylistic similarity by itself.
For each criterion, return an integer from 1 to 5.
A high score requires evidence in the answer, not merely relevant keywords.
If the reference context is insufficient to verify a claim, note that limitation.

Return JSON only:
{
  "correctness": 1,
  "relevance": 1,
  "completeness": 1,
  "safety": 1,
  "clarity": 1,
  "critical_error": false,
  "reason_codes": ["SHORT_MACHINE_READABLE_CODE"]
}

TASK:
{{ task }}

REFERENCE CONTEXT:
{{ reference_context }}

RUBRIC:
{{ rubric }}

ANSWER:
{{ answer }}

Judge Alpha and Beta in separate calls if possible. Putting both answers into a “direct” prompt can turn the task into an implicit comparison. Randomize the order in which the single-answer calls are executed and do not expose the other answer’s score.

Pairwise prompt

You are comparing two candidate answers to the same task.

Choose the answer that better satisfies the RUBRIC.
Do not prefer an answer because it appears first or second.
Do not reward length, confidence, formatting, or stylistic similarity by itself.
Treat verbosity as useful only when it adds relevant and correct information.
Choose TIE only when neither answer is meaningfully better overall.

Return JSON only:
{
  "winner": "A",
  "confidence": "low",
  "reason_codes": ["MORE_CORRECT"]
}

Allowed winner values: A, B, TIE
Allowed confidence values: low, medium, high

TASK:
{{ task }}

REFERENCE CONTEXT:
{{ reference_context }}

RUBRIC:
{{ rubric }}

ANSWER A:
{{ answer_a }}

ANSWER B:
{{ answer_b }}

Reason codes are useful for diagnosis, but do not use free-form explanations as the scored outcome. Explanations can be plausible even when the selected label is unstable.

Build the human reference

Human evaluation is not automatically correct. Reviewers need the same task, context, and rubric as the judge, plus training examples that do not belong to the evaluation set.

• Assign at least two independent reviewers where the cost permits.

• Hide system identities and automatic-judge outputs.

• Randomize answer order separately from the automatic run.

• Ask for A, B, or tie and criterion scores if those scores will be compared.

• Record the initial independent decisions before discussion.

• Adjudicate disagreements under a written rule, without deleting the original labels.

A compact human-label file might look like this:

item_id,reviewer_id,alpha_score,beta_score,preference,is_adjudicated
support_0042,r01,4.2,3.4,ALPHA,false
support_0042,r02,4.0,3.8,ALPHA,false
support_0042,adjudicator,4.1,3.6,ALPHA,true

Before using adjudicated labels as the reference, measure agreement among the initial reviewers. If humans cannot apply the rubric consistently, automatic agreement with an adjudicated label may overstate the quality of the measurement process.

Run the four required passes

• Direct Alpha: score every Alpha answer independently.

• Direct Beta: score every Beta answer independently.

• Pairwise original: judge the recorded randomized A/B presentation.

• Pairwise swapped: exchange A and B and judge again with all other fields unchanged.

Do not send the first pairwise decision to the swap call. Avoid conversational sessions that preserve previous answers. Each judgment should start from a clean context.

A provider-neutral runner should perform these operations:

for item in items:
    direct_alpha = judge_direct(item.task, item.context, item.answer_alpha)
    direct_beta  = judge_direct(item.task, item.context, item.answer_beta)

    if seeded_order(item.item_id) == "ALPHA_FIRST":
        original = judge_pair(item.answer_alpha, item.answer_beta)
        swapped  = judge_pair(item.answer_beta, item.answer_alpha)
    else:
        original = judge_pair(item.answer_beta, item.answer_alpha)
        swapped  = judge_pair(item.answer_alpha, item.answer_beta)

    validate_and_store(item.item_id, direct_alpha, direct_beta)
    validate_and_store(item.item_id, original, swapped)

Validate JSON types, required keys, allowed labels, score ranges, and duplicate identifiers before accepting a record. Store malformed responses separately. A retry must use the same input and a documented retry policy.

Normalize both strategies to the same decision space

Agreement cannot be compared fairly while direct scoring produces numbers and pairwise evaluation produces labels. Convert both to system-level preferences: ALPHA, BETA, or TIE.

For direct scoring, first calculate the weighted total for each answer. If all criteria have equal weight:

direct_total = (
    correctness +
    relevance +
    completeness +
    safety +
    clarity
) / 5

Then apply the predefined tie margin:

difference = alpha_total - beta_total

if difference > tie_margin:
    direct_preference = "ALPHA"
elif difference < -tie_margin:
    direct_preference = "BETA"
else:
    direct_preference = "TIE"

Map pairwise A/B labels back to system identities using the recorded presentation order. Never compare the raw letter A with the human label Alpha.

Measure agreement with humans

Exact agreement

Inter-rater agreement begins with the proportion of valid items on which the automatic strategy and human reference choose the same label:

exact agreement = matching labels / jointly valid labels

Report its numerator and denominator. A percentage without the number of eligible items can hide exclusions.

Confusion matrix

Create a 3×3 table for human and judge labels. It reveals whether disagreement is concentrated in ties, whether one system is systematically favored, or whether the judge reverses clear human decisions.

Chance-corrected agreement

Cohen’s kappa can supplement exact agreement when comparing one automatic label with one final human label. It adjusts for agreement expected from the observed label frequencies:

kappa = (observed_agreement - expected_agreement)
        / (1 - expected_agreement)

Kappa can behave unexpectedly when one label dominates. Always show label frequencies and exact agreement beside it. If you retain multiple reviewers or have missing labels, consider Krippendorff’s alpha under a clearly stated measurement level.

Score association

If humans and the direct judge both provide criterion scores, report rank association and absolute score differences in addition to preference agreement. A strong ranking does not prove that an absolute threshold is calibrated correctly.

Uncertainty

Use a paired bootstrap over item identifiers to compare direct and pairwise agreement. Resample whole items, not individual judge fields. Report a confidence interval for each agreement rate and for their paired difference.

Do not claim one strategy is better merely because its point estimate is higher. Inspect the interval, sample composition, and disagreement cases.

Run the positional-bias test

The original and swapped pair must represent the same underlying comparison. After mapping letters back to system identities, a stable judge should preserve its decision:

          Original presentation
          Original winner
          Swapped presentation
          Stable swapped winner




          A = Alpha, B = Beta
          A
          A = Beta, B = Alpha
          B


          A = Alpha, B = Beta
          B
          A = Beta, B = Alpha
          A


          A = Alpha, B = Beta
          TIE
          A = Beta, B = Alpha
          TIE

Swap consistency

swap consistency =
    same system-level preference after swap
    / pairs with two valid judgments

Count every change, including winner-to-tie and tie-to-winner. Also report the stricter reversal rate among non-tie decisions: Alpha becomes Beta or Beta becomes Alpha after mapping to identities.

First-position preference

first-position rate =
    judgments selecting the displayed A answer
    / valid non-tie pairwise judgments

Calculate this across both passes and separately by pass. A rate near one half is not sufficient evidence of no bias: opposing biases across item categories can cancel out. Compare outcomes within each swapped pair.

Position-sensitive subset

Create a review file containing every inconsistent pair, answer lengths, human preference, original order, original decision, swapped order, and swapped decision. Inspect this subset without changing the main evaluation labels.

Operational aggregation

If you later use swapped judgments in production, define the aggregation rule in advance. One conservative rule is:

• If both presentations select the same system, accept that preference.

• If both return tie, accept tie.

• If they disagree, mark the item unstable rather than forcing a winner.

This rule reduces coverage. Report the abstention rate so that stability is not purchased invisibly.

Minimal reproducible analysis

Prepare a CSV with one row per item and these columns:

item_id,human,direct,pair_original,pair_swapped,original_first,alpha_words,beta_words
support_0042,ALPHA,ALPHA,ALPHA,ALPHA,ALPHA,126,81

All preference fields must already be mapped to ALPHA, BETA, or TIE. The following script uses only the Python standard library:

#!/usr/bin/env python3
import csv
import random
import sys
from collections import Counter

LABELS = ("ALPHA", "BETA", "TIE")

def valid(row, *fields):
    return all(row.get(field) in LABELS for field in fields)

def agreement(rows, field):
    usable = [r for r in rows if valid(r, "human", field)]
    matches = sum(r["human"] == r[field] for r in usable)
    return matches, len(usable), matches / len(usable) if usable else float("nan")

def confusion(rows, field):
    matrix = {(h, j): 0 for h in LABELS for j in LABELS}
    for row in rows:
        if valid(row, "human", field):
            matrix[(row["human"], row[field])] += 1
    return matrix

def kappa(rows, field):
    usable = [r for r in rows if valid(r, "human", field)]
    n = len(usable)
    if not n:
        return float("nan")
    observed = sum(r["human"] == r[field] for r in usable) / n
    human_counts = Counter(r["human"] for r in usable)
    judge_counts = Counter(r[field] for r in usable)
    expected = sum(
        (human_counts[label] / n) * (judge_counts[label] / n)
        for label in LABELS
    )
    if expected == 1:
        return float("nan")
    return (observed - expected) / (1 - expected)

def swap_consistency(rows):
    usable = [
        r for r in rows
        if valid(r, "pair_original", "pair_swapped")
    ]
    stable = sum(
        r["pair_original"] == r["pair_swapped"]
        for r in usable
    )
    return stable, len(usable), stable / len(usable) if usable else float("nan")

def first_position_rate(rows):
    selections = []
    for r in rows:
        first = r.get("original_first")
        winner = r.get("pair_original")
        if first in ("ALPHA", "BETA") and winner in ("ALPHA", "BETA"):
            selections.append(first == winner)
    return sum(selections), len(selections), (
        sum(selections) / len(selections) if selections else float("nan")
    )

def paired_bootstrap(rows, iterations=10000, seed=240719):
    eligible = [
        r for r in rows
        if valid(r, "human", "direct", "pair_original")
    ]
    if not eligible:
        return []
    rng = random.Random(seed)
    differences = []
    n = len(eligible)
    for _ in range(iterations):
        sample = [eligible[rng.randrange(n)] for _ in range(n)]
        direct = sum(r["human"] == r["direct"] for r in sample) / n
        pair = sum(r["human"] == r["pair_original"] for r in sample) / n
        differences.append(pair - direct)
    return sorted(differences)

def percentile(values, p):
    if not values:
        return float("nan")
    index = round((len(values) - 1) * p)
    return values[index]

with open(sys.argv[1], newline="", encoding="utf-8") as handle:
    rows = list(csv.DictReader(handle))

for field in ("direct", "pair_original"):
    matches, total, rate = agreement(rows, field)
    print(field, "agreement", matches, "/", total, "=", f"{rate:.4f}")
    print(field, "kappa", f"{kappa(rows, field):.4f}")
    print(field, "confusion")
    matrix = confusion(rows, field)
    for human in LABELS:
        print(human, [matrix[(human, judge)] for judge in LABELS])

stable, total, rate = swap_consistency(rows)
print("swap consistency", stable, "/", total, "=", f"{rate:.4f}")

first, total, rate = first_position_rate(rows)
print("original first-position selections", first, "/", total, "=", f"{rate:.4f}")

diffs = paired_bootstrap(rows)
print(
    "pair-minus-direct agreement difference interval",
    f"[{percentile(diffs, 0.025):.4f}, {percentile(diffs, 0.975):.4f}]"
)

Run it with:

python3 analyze.py normalized_results.csv

This script intentionally does not manufacture missing labels, calculate p-values, or declare a winner. Extend it only after documenting the additional decision rule.

Check length and self-preference separately

The answer-swap experiment isolates position effects, not every bias.

Length preference

For non-tie pairs, mark whether the judge chose the longer answer and whether the human chose the longer answer. Compare:

• the judge’s longer-answer selection rate;

• the human longer-answer selection rate;

• judge–human agreement within similar-length pairs;

• judge–human agreement when the word-count difference is large.

A higher longer-answer rate is not by itself proof of bias: longer answers may genuinely be better in the sample. The diagnostic signal is whether the judge prefers length in cases where humans favor the shorter answer, especially when the shorter answer is more correct or relevant.

Self-preference

To investigate self-preference, include answers from the judge’s own model family and from other systems while keeping system identities hidden. Compare human-adjusted win patterns across source families. Treat the result as observational unless generation quality, task mix, and answer style have been controlled.

A stronger test creates matched transformations that preserve meaning while reducing recognizable stylistic signatures. Such transformations can introduce their own artifacts, so retain the original-answer experiment as the primary measurement.

Verification checklist

Complete these checks before interpreting results:

• Coverage: every expected item has two direct records and two pairwise records, or a documented exclusion.

• Uniqueness: no duplicate combination of item, strategy, pass, and judge configuration exists.

• Schema: every accepted score and label falls inside its declared domain.

• Identity mapping: manually inspect a sample to confirm that A/B winners were mapped back to Alpha/Beta correctly.

• Swap integrity: task, context, rubric, and answers are byte-identical between passes except for answer order and labels.

• Blindness: prompts contain no hidden system name or file name that reveals the answer source.

• Human independence: initial reviewers did not see each other’s labels or judge outputs.

• Missingness: invalid and failed calls are reported by strategy and answer source.

• Reproducibility: rerunning analysis from stored normalized data reproduces all tables.

• Manual audit: inspect agreements, disagreements, ties, reversals, extreme score differences, and malformed outputs.

How to interpret the result

Use a result table that preserves denominators and uncertainty:

          Measure
          Direct scoring
          Pairwise




          Valid items
          Fill from run
          Fill from run


          Exact human agreement
          Estimate and interval
          Estimate and interval


          Cohen’s kappa
          Estimate
          Estimate


          Tie rate
          Estimate
          Estimate


          Invalid-output rate
          Estimate
          Estimate


          Swap consistency
          Not the primary test
          Estimate and denominator

Pairwise evaluation is the stronger choice when it shows meaningfully better human agreement, acceptable swap consistency, and manageable abstention and cost. Direct scoring is stronger when criterion-level scores are reproducible, thresholds matter operationally, and its human agreement is comparable or better.

A hybrid is often practical: direct scores for diagnostics and filtering, followed by swapped pairwise evaluation for close or high-impact cases. The hybrid still needs an explicit routing threshold and evaluation as a complete system.

Common failure cases

The judge picks the same displayed position after the swap

This is strong evidence of order sensitivity for that pair. Check prompt construction and mapping first. If they are correct, mark the pair unstable. Do not average the two letter labels.

Direct scores are almost all 4 or 5

The scale may be poorly anchored, the rubric may be too permissive, or the task sample may be too easy. Add behavioral descriptions for each score level using development items, then freeze the revised rubric before a new evaluation run.

Pairwise evaluation rarely allows ties

The prompt may pressure the judge to choose a winner. Verify that tie is an allowed schema value and give an operational definition. Do not add a target tie percentage.

Agreement is high because one system wins almost everything

Inspect label prevalence, confusion matrices, and performance by task category. A judge that always selects the dominant system can achieve high raw agreement without handling difficult cases.

Invalid output rates differ by answer source

This may indicate prompt injection, unusual formatting, extreme length, or content filtering. Invalid output is an evaluation outcome, not housekeeping. Report it by source and category.

The longer answer wins even when it repeats itself

Review human-disagreement cases and code whether extra text adds verified information. Tighten the rubric language around relevance and redundancy, but make changes only on the development set.

Reasons and labels contradict each other

Score the structured label as the primary output and log the contradiction. If reasons drive downstream action, evaluate their correctness separately rather than assuming they justify the choice.

Results change across repeated identical calls

Record the instability. Reduce uncontrolled sampling where possible, repeat a defined subset, and report within-configuration repeatability. A deterministic setting does not guarantee stable infrastructure or a permanently fixed hosted model.

Limitations

• The human reference reflects the chosen reviewers, rubric, and adjudication process rather than universal truth.

• Answer swapping detects positional sensitivity but does not isolate every cause of disagreement.

• Blinding model identities cannot remove all recognizable style signals.

• Results may not transfer to different domains, languages, safety levels, prompt formats, or answer lengths.

• A judge that ranks two systems correctly may still be unsuitable for absolute quality thresholds.

• Pairwise results depend on the candidate set; adding a new system can change which comparisons matter.

• Hosted model behavior can change. Preserve dates, configuration, raw outputs, and prompt versions.

• Bootstrap intervals describe uncertainty from the sampled items under the chosen procedure; they do not account for every source of model or reviewer uncertainty.

A defensible final decision

Do not choose a strategy from one headline percentage. Write a short decision record that answers:

• Which strategy agrees more closely with the human reference, and by how much?

• How uncertain is that difference?

• How often does pairwise judgment change after swapping positions?

• Are disagreements concentrated in ties, long answers, particular categories, or particular source systems?

• What proportion of records is invalid or unstable?

• Does the intended use require absolute scores, relative ranking, or both?

• What human-review fallback is required for high-impact or unstable cases?

The judge is ready only for the scope in which these measurements support it. If the evidence is weak, the correct output is not a forced winner—it is a narrower deployment, a human-review route, or another evaluation cycle.

Repeatable workflow summary

• Sample representative tasks and freeze the evaluation set.

• Collect two candidate answers per task and retain their identities separately from displayed labels.

• Write one rubric shared by humans and the automatic judge.

• Collect independent human labels and preserve disagreements.

• Run direct scoring on each answer separately.

• Run pairwise evaluation with seeded randomized order.

• Swap every pair and run it again in a clean context.

• Normalize all outcomes to Alpha, Beta, or tie.

• Measure exact agreement, confusion matrices, chance-corrected agreement, missingness, and uncertainty.

• Measure swap consistency and first-position selection separately.

• Audit disagreements for length, task category, source family, and critical errors.

• Select direct, pairwise, hybrid, or human fallback according to the intended operational decision.

Continue with the Agent Lab Journal guides, or review evaluation terminology in the glossary.

Agent Lab Journal

---

Original article: https://agentlabjournal.online/en/llm-judge-direct-vs-pairwise.html?utm_source=devto&utm_medium=referral&utm_campaign=agentlabjournal