DEV Community

Cover image for LLM-as-a-Judge: Evaluating RAG Systems Beyond Exact-Match Metrics
Nikhil raman K
Nikhil raman K

Posted on

LLM-as-a-Judge: Evaluating RAG Systems Beyond Exact-Match Metrics

Introduction

Building a Retrieval-Augmented Generation system is relatively straightforward.

Building a reliable RAG system is not.

A typical RAG pipeline looks like:

User Query

Query Processing

Retriever

Top-K Documents

Context Construction

LLM

Generated Answer

The difficult question is:

How do we know whether the answer is actually good?

Suppose a user asks:

"What is the company's policy for parental leave?"

Our retriever returns five documents.

The correct policy document is among them.

The LLM generates a fluent response.

Everything appears successful.

But perhaps the model:

ignored the relevant document,
mixed information from two conflicting documents,
invented a number,
omitted an important condition,
answered only part of the question,
or confidently generated information that does not exist in the retrieved context.

A retrieval metric alone cannot detect all of these failures.

This is where LLM-as-a-Judge becomes useful.

Instead of evaluating only whether a particular document was retrieved, we evaluate the complete interaction:

             RAG SYSTEM
                 │
                 ▼
            User Query
                 │
                 ▼
             Retriever
                 │
                 ▼
         Retrieved Context
                 │
                 ▼
                LLM
                 │
                 ▼
          Generated Answer
                 │
                 ▼
          LLM-as-a-Judge
                 │
    ┌────────────┼────────────┐
    ▼            ▼            ▼
Enter fullscreen mode Exit fullscreen mode

Faithfulness Relevance Completeness
│ │ │
└────────────┼────────────┘

Structured Scores


Human / Reference Check


Evaluation Report

The important engineering principle is:

An LLM judge is not ground truth. It is a scalable evaluator that must itself be validated.

That distinction is fundamental.

  1. Why Evaluating RAG Is Harder Than Evaluating a Normal LLM

A conventional LLM evaluation might look like:

Question

LLM

Answer

Compare with Reference

RAG introduces another entire failure surface.

Question

Retriever

Context

Generator

Answer

Now an incorrect answer can originate from multiple locations.

Failure type 1 — Retrieval failure

The correct information was never retrieved.

Question

Retriever

❌ Wrong documents

LLM

Incorrect answer
Failure type 2 — Context utilization failure

The correct information was retrieved, but the model failed to use it.

Question

Retriever

✅ Correct document

LLM

❌ Ignores evidence
Failure type 3 — Generation failure

The retrieved context is correct, but the model introduces unsupported information.

Retrieved Context

Correct evidence

LLM

❌ Hallucinated claim
Failure type 4 — Incomplete answer

The generated response may be factually correct but still fail the user.

For example:

Question:
"What are the eligibility requirements and application deadlines?"

Answer:
"The program is available to full-time employees."

The statement may be correct.

But the answer is incomplete.

This is why:

Correctness ≠ completeness ≠ faithfulness ≠ relevance.

  1. Retrieval Quality and Answer Quality Are Different

This is one of the most important distinctions in RAG evaluation.

Consider:

Query

Retriever

D1 ← irrelevant
D2 ← relevant
D3 ← irrelevant
D4 ← relevant
D5 ← irrelevant

The retrieval system may have performed reasonably well.

But now the LLM receives:

D1
D2
D3
D4
D5

and produces:

Generated Answer

The generation model can still:

misunderstand the evidence,
combine unrelated passages,
hallucinate,
omit critical details,
or answer the wrong interpretation of the question.

Therefore we should separate evaluation into layers.

Retrieval Metrics

Did we find useful evidence?

Generation Metrics

Did we answer the question well?

Grounding Metrics

Did we stay within the evidence?

Production Evaluation

Does the entire RAG system behave reliably?

This decomposition is extremely useful when debugging production systems.

  1. What Is LLM-as-a-Judge?

LLM-as-a-Judge means using one language model to evaluate the output of another model or system.

Instead of:

Human

Read 10,000 answers

Score manually

we can build:

RAG Output

Evaluation Prompt

Judge LLM

Structured Evaluation

For example:

{
"question": "What is the refund policy?",
"retrieved_context": [
"Customers may request a refund within 30 days..."
],
"generated_answer": "Customers can request a refund within 30 days."
}

The judge can evaluate:

{
"faithfulness": 0.98,
"answer_relevance": 0.96,
"context_relevance": 0.91,
"completeness": 0.88
}

This is powerful because the evaluator can reason about semantic properties, rather than requiring an exact string match.

Research such as G-Eval demonstrated that LLM-based evaluation can correlate meaningfully with human judgments, while also highlighting that LLM evaluators themselves have biases and limitations.

  1. The Five Dimensions I Would Evaluate

A serious RAG evaluation system should not reduce everything to:

"Is the answer good?"

That question is too vague.

Instead, break it down.

4.1 Faithfulness

Question:

Are the claims in the answer supported by the retrieved context?

Example:

Context:

The warranty period is 12 months.

Answer:

The warranty period is 24 months.

Faithfulness:

❌ Low

The answer contradicts the evidence.

4.2 Answer Relevance

Question:

Does the answer actually address the user's question?

Question:

What is the refund period?

Answer:

The company has operated since 1998 and serves customers globally.

The answer may contain perfectly valid information.

But it does not answer the question.

Therefore:

Faithfulness: potentially high
Answer relevance: very low

This demonstrates why metrics cannot be collapsed into a single concept.

4.3 Context Relevance

Question:

Did the retriever provide useful evidence for answering the query?

Suppose the retriever returns:

Document 1 → Relevant
Document 2 → Relevant
Document 3 → Marketing history
Document 4 → Employee benefits
Document 5 → Office locations

The answer might still be correct because Documents 1 and 2 contained the necessary information.

But the retrieval system is wasting context capacity.

This matters because poor retrieval can eventually hurt generation through:

context dilution,
irrelevant evidence,
conflicting information,
increased token cost,
and reduced model attention.

RAGAS explicitly treats retrieval and generation as separate evaluation dimensions, including context-related metrics and answer-level metrics.

4.4 Completeness

A response can be relevant and faithful while still being incomplete.

Question:

What are the eligibility requirements and application deadlines?

Context contains:

Eligibility:

  • Full-time employees
  • Minimum 12 months tenure

Deadline:

  • Applications must be submitted by December 15

Answer:

The program is available to full-time employees.

The answer is:

Relevant → Yes
Faithful → Yes
Complete → No

This distinction becomes extremely important in enterprise applications.

4.5 Correctness

Correctness asks:

Is the answer factually correct?

When a trusted reference answer exists, it can be used.

For example:

Reference:
"The warranty period is 12 months."

Generated:
"The warranty period is 12 months."

Correctness:
1.0

But correctness and faithfulness are not always identical.

Consider:

Context:
"The warranty period is 12 months."

Answer:
"The warranty period is 12 months for all products."

The answer might be factually true in the larger knowledge base.

But if the retrieved context does not establish "all products", the answer has a grounding problem.

That is why production evaluation should retain both:

Correctness
+
Faithfulness

  1. Groundedness vs Faithfulness

These terms are often used interchangeably, but an engineering evaluation framework should define them explicitly.

I prefer this operational interpretation:

Faithfulness

Are the answer's claims entailed by the provided context?

Groundedness

Can important claims be traced back to available evidence?

For production systems, we can go one step further.

Instead of only producing:

{
"faithfulness": 0.92
}

we can request:

{
"unsupported_claims": [
"The policy applies to contractors."
]
}

Now evaluation becomes actionable.

The system isn't merely saying:

"Your score is 0.72."

It is saying:

"This specific claim is unsupported."

That is far more useful for debugging.

  1. A Production Evaluation Input

A judge should receive structured information.

For example:

{
"question": "What is the company's parental leave policy?",
"retrieved_context": [
"Employees are eligible for 16 weeks of parental leave.",
"Leave must be requested through the HR portal."
],
"generated_answer": "Employees receive 16 weeks of parental leave and must request it through the HR portal."
}

And return:

{
"faithfulness": 0.98,
"answer_relevance": 0.97,
"context_relevance": 0.91,
"completeness": 0.94,
"correctness": null,
"unsupported_claims": [],
"reason": "The answer directly addresses the question and each claim is supported by the retrieved context."
}

Notice something important.

correctness can be null.

Why?

Because we may not have a trusted reference answer.

That leads to another important design decision.

  1. Reference-Free vs Reference-Based Evaluation

There are two broad evaluation modes.

Reference-based
Question

├── Reference Answer

└── Generated Answer

Judge

Useful when we have:

verified answers,
domain experts,
benchmark datasets,
deterministic expected outputs.
Reference-free
Question

├── Retrieved Context

└── Generated Answer

Judge

This is extremely useful for enterprise RAG because manually creating reference answers for thousands of questions is expensive.

RAGAS was explicitly designed around reference-free evaluation of RAG pipelines.

But reference-free does not mean ground-truth-free.

It means the evaluator is using the available evidence and criteria rather than requiring a manually written answer for every example.

  1. Designing the Judge Prompt

A weak judge prompt might say:

Is this answer good?
Give a score from 1 to 10.

This is not a robust evaluation protocol.

The judge has too much freedom.

Instead, define:

evaluation criteria,
scoring scale,
evidence requirements,
output schema,
failure conditions.

For example:

You are evaluating a RAG system.

Evaluate the generated answer using only the supplied
question and retrieved context.

Faithfulness:
Determine whether every factual claim in the answer
is supported by the retrieved context.

Answer relevance:
Determine whether the answer directly addresses
the user's question.

Completeness:
Determine whether the answer covers the important
information required to answer the question.

Do not award a high score simply because the answer
sounds fluent or confident.

Identify unsupported claims explicitly.

Return valid JSON only.

This is much more reproducible.

  1. Use Structured Outputs

A production evaluator should not return arbitrary prose.

Bad:

The answer seems mostly correct but there are
some concerns...

Better:

{
"faithfulness": 0.84,
"answer_relevance": 0.93,
"completeness": 0.76,
"unsupported_claims": [
"The policy applies to contractors."
]
}

Now the output can enter:

Evaluation Pipeline

JSON Parser

Database

Metrics Aggregation

Dashboard

Regression Tests

This turns qualitative evaluation into machine-readable telemetry.

  1. Scoring

Suppose each evaluation produces:

Faithfulness = 0.92
Answer Relevance = 0.88
Context Relevance = 0.76
Completeness = 0.84

We could calculate:

Overall Score

0.30 × Faithfulness
+
0.25 × Relevance
+
0.20 × Context Relevance
+
0.25 × Completeness

But there is an important warning.

Do not automatically assume a weighted average is the correct production metric.

Averages can hide catastrophic failures.

Consider:

System A

Faithfulness = 0.99
Relevance = 0.98
Completeness = 0.97

versus:

System B

Faithfulness = 0.99
Relevance = 0.98
Completeness = 0.97

They appear equivalent.

But if System B produces:

2% responses with severe unsupported claims

that tail behavior may matter much more than the average.

Therefore production evaluation should track:

Mean
Median
P90 / P95
Failure rate
Critical failure rate
Unsupported claim rate

  1. Pairwise Evaluation

Absolute scoring isn't the only useful approach.

We can ask a judge:

Which answer is better?

Answer A
Answer B

For example:

            Same Question
                 │
        ┌────────┴────────┐
        ↓                 ↓
    RAG Version A     RAG Version B
        ↓                 ↓
     Answer A          Answer B
        └────────┬────────┘
                 ↓
             Judge LLM
                 ↓
         A / B / Tie
Enter fullscreen mode Exit fullscreen mode

This is particularly useful when comparing:

chunking strategies,
embedding models,
retrievers,
rerankers,
prompts,
LLMs,
query rewriting,
hybrid search,
RAG architectures.

Instead of asking:

"Is version B a 7.8?"

we ask:

"Is version B better than version A?"

That is often easier for a judge to reason about.

  1. But LLM Judges Are Not Neutral

This is where the topic becomes genuinely interesting.

An LLM judge can have biases.

Research on LLM-as-a-Judge has documented issues including:

position bias,
verbosity bias,
self-enhancement bias,
limited reasoning capability,
sensitivity to evaluation prompt design.

The MT-Bench/Chatbot Arena work found strong agreement between GPT-4 judges and human preferences in their experiments, but also explicitly studied these limitations.

Other research demonstrated that simply changing the order of candidate responses can influence evaluation outcomes.

Therefore:

Never treat an LLM judge as an infallible oracle.

  1. Position Bias

Imagine:

Question

Answer A
Answer B

The judge selects:

A wins

Now swap them:

Question

Answer B
Answer A

If the judge suddenly selects:

B wins

we have discovered a problem.

The evaluation is sensitive to position.

A simple mitigation is:

Evaluation 1:
A vs B

Evaluation 2:
B vs A

Then aggregate the results.

For example:

A wins first comparison
B wins reversed comparison

→ potentially ambiguous.

This kind of calibration is far more robust than blindly trusting one judge call. Position bias and calibration strategies have been explicitly studied in the literature.

  1. Verbosity Bias

Suppose we have two answers.

Answer A
The warranty period is 12 months.
Answer B
The warranty period is 12 months.

This means that customers who purchase the product
are entitled to warranty coverage for a full twelve-month
period following their purchase date.

The company introduced this policy to ensure...

Answer B sounds more sophisticated.

But it might contain irrelevant or unsupported content.

A judge can accidentally associate:

More words

More quality

which is not necessarily true.

Therefore judge prompts should explicitly say:

Do not reward verbosity unless additional detail improves correctness, relevance, or completeness.

  1. Self-Preference and Model Bias

Suppose:

Generator = Model X
Judge = Model X

The judge may have preferences for outputs resembling its own generation style.

This is another reason why:

LLM judge = ground truth

is the wrong mental model.

Instead:

LLM judge
+
Human calibration
+
Deterministic metrics
+
Reference checks
+
Multiple evaluation methods

creates a much stronger evaluation framework.

  1. Judge Calibration

Before trusting an evaluator in production, build a human-labelled validation set.

For example:

1,000 RAG examples

100–200 carefully reviewed by experts

Human scores

LLM judge scores

Compare

Measure:

Human vs Judge

using appropriate agreement/correlation measures.

Then inspect disagreements.

For example:

Human: 0.30
Judge: 0.90

That example is valuable.

Why did the judge think the answer was excellent?

Perhaps the answer was fluent but unsupported.

That reveals a judge failure.

ARES takes a particularly interesting approach here: it combines automated judges with a relatively small amount of human-annotated data and prediction-powered inference to improve evaluation reliability and provide confidence intervals.

  1. Multiple Judges

For high-value applications, one judge may not be enough.

We can use:

             RAG Answer
                 │
    ┌────────────┼────────────┐
    ↓            ↓            ↓
 Judge A      Judge B      Judge C
    │            │            │
    └────────────┼────────────┘
                 ↓
            Aggregator
                 ↓
          Final Evaluation
Enter fullscreen mode Exit fullscreen mode

Different judges can specialize.

Judge A → Faithfulness
Judge B → Relevance
Judge C → Completeness

Or:

Judge 1 → General evaluator
Judge 2 → Domain evaluator
Judge 3 → Safety evaluator

The final system can combine their results.

This is effectively an evaluation ensemble.

  1. When Deterministic Metrics Are Better

LLM judges are powerful.

But they should not replace deterministic evaluation where deterministic evaluation is appropriate.

For example:

Classification
Accuracy
Precision
Recall
F1
Retrieval
Recall@K
Precision@K
MRR
nDCG
Hit Rate
Structured extraction
Exact Match
Schema Validity
Field Accuracy
Code generation
Unit Tests
Compilation
Execution
SQL generation
Query Execution
Result Equivalence

If a deterministic test can answer the question reliably, use it.

Don't ask an LLM:

"Did this JSON contain the required field?"

when a JSON parser can answer that deterministically.

The best evaluation systems are therefore hybrid.

  1. Building a RAG Evaluation Dataset

Evaluation quality depends heavily on evaluation data.

A production dataset should contain representative queries.

For example:

Easy queries
Hard queries
Ambiguous queries
Multi-hop queries
Long-context queries
No-answer queries
Out-of-domain queries
Adversarial queries

Also include known failure cases.

For example:

Query:
"What is the refund policy for international orders?"

Expected behavior:
Retrieve international refund policy.

Another:

Query:
"What is the refund policy for a product that
is not covered by the policy?"

Expected behavior:
Explicitly state that the available evidence
does not establish an answer.

This second category is particularly important.

A good RAG system should know when not to answer.

  1. Regression Testing for RAG

This is where evaluation becomes engineering rather than experimentation.

Imagine version 1:

Chunk size = 500
Retriever = Dense
Top-K = 5

Evaluation:

Faithfulness = 0.91

Then version 2 changes chunking:

Chunk size = 1000

The system looks better in manual testing.

But automated evaluation shows:

Faithfulness = 0.84

Now we know the change caused regression.

A mature workflow becomes:

Code Change

Build

Run RAG Evaluation Set

LLM Judge

Deterministic Metrics

Compare Against Baseline

Pass / Fail

This can become part of CI/CD.

  1. LangChain / LangGraph Perspective

A RAG graph can be represented as:

START

retrieve

rerank

generate

evaluate

┌───────────────┐
│ Score Quality │
└───────┬───────┘

┌────┴─────┐
↓ ↓
PASS FAIL
│ │
↓ ↓
END retry/retrieve

This is where evaluation becomes especially interesting with LangGraph.

The evaluator can become an actual control node.

For example:

def evaluate(state):

result = judge(
    question=state["question"],
    context=state["context"],
    answer=state["answer"]
)

state["evaluation"] = result

return state
Enter fullscreen mode Exit fullscreen mode

Then:

evaluate

├── score >= threshold → END

└── score < threshold

retry_retrieval

Now evaluation is no longer just a dashboard.

It becomes part of the agentic control loop.

  1. Evaluation as a Feedback Loop

This gives us a much more powerful architecture.

            ┌───────────────────┐
            │    User Query     │
            └─────────┬─────────┘
                      ↓
                ┌───────────┐
                │ Retriever │
                └─────┬─────┘
                      ↓
                ┌───────────┐
                │  Context  │
                └─────┬─────┘
                      ↓
                ┌───────────┐
                │    LLM    │
                └─────┬─────┘
                      ↓
                ┌───────────┐
                │  Answer   │
                └─────┬─────┘
                      ↓
             ┌─────────────────┐
             │  LLM-as-Judge   │
             └────────┬────────┘
                      ↓
             ┌─────────────────┐
             │ Evaluation      │
             │ Metrics         │
             └────────┬────────┘
                      ↓
             ┌─────────────────┐
             │ Pass / Retry /  │
             │ Human Review    │
             └─────────────────┘
Enter fullscreen mode Exit fullscreen mode

This creates a closed-loop RAG system.

  1. Production Observability

A production RAG system should log more than:

request_id
response
latency

For evaluation, capture:

query
query_type
retriever
retrieved_documents
document_scores
reranker_scores
prompt_version
model_version
generated_answer
latency
token_usage
evaluation_scores
unsupported_claims
failure_category
timestamp

Then dashboards can expose:

Average Faithfulness
Average Relevance
Context Relevance
Completeness
Retrieval Recall
Latency
Cost
Failure Rate

But the most useful dashboard isn't necessarily:

Overall Score = 87.4%

It is something like:

                RAG HEALTH
Enter fullscreen mode Exit fullscreen mode

Faithfulness █████████░ 92%
Answer Relevance █████████░ 89%
Completeness ████████░░ 84%
Context Relevance ███████░░░ 76%

Unsupported Claims: 4.8%
Retrieval Failures: 11.2%
Generation Failures: 3.1%

Now engineers know where to investigate.

  1. Failure Analysis

Aggregate scores tell us that something is wrong.

Failure analysis tells us why.

Suppose:

Faithfulness dropped from 0.93 → 0.81

We should classify failures.

Retrieval failure 38%
Unsupported generation 31%
Incomplete answer 18%
Question ambiguity 8%
Other 5%

Now we know where to focus.

This leads to a powerful engineering loop:

Evaluation

Failure Classification

Root Cause

System Change

Evaluation

That is how RAG systems should actually be improved.

  1. A Practical End-to-End Evaluation Object

Here is a structure I would actually persist in an evaluation database:

{
"evaluation_id": "eval_001",
"question": "What is the parental leave policy?",
"retrieved_context": [
{
"document_id": "hr_104",
"text": "Employees are eligible for 16 weeks..."
},
{
"document_id": "hr_108",
"text": "Requests must be submitted through HR..."
}
],
"generated_answer": "Employees are eligible for 16 weeks...",
"metrics": {
"context_relevance": 0.91,
"faithfulness": 0.97,
"answer_relevance": 0.95,
"completeness": 0.89
},
"unsupported_claims": [],
"failure_category": null,
"judge_model": "evaluation-model",
"prompt_version": "judge_v3",
"rag_version": "rag_v17"
}

Notice the version fields.

They matter enormously.

Without:

model_version
prompt_version
rag_version

evaluation results become difficult to reproduce.

  1. The Most Important Architectural Principle

We can now separate the system into four layers.

┌─────────────────────────────┐
│ Retrieval Metrics │
│ │
│ Did we find useful evidence?│
└──────────────┬──────────────┘

┌─────────────────────────────┐
│ Generation Metrics │
│ │
│ Did we answer effectively? │
└──────────────┬──────────────┘

┌─────────────────────────────┐
│ Grounding Metrics │
│ │
│ Did we stay within evidence?│
└──────────────┬──────────────┘

┌─────────────────────────────┐
│ Production Evaluation │
│ │
│ Is the entire system │
│ reliable over time? │
└─────────────────────────────┘

This is much more meaningful than saying:

"Our RAG accuracy is 91%."

Accuracy of what?

Retrieval?

Generation?

Grounding?

Human preference?

Reference correctness?

The metric must correspond to the failure mode.

  1. A Mature RAG Evaluation Stack

A production architecture might therefore look like:

                RAG APPLICATION
                      │
        ┌─────────────┴─────────────┐
        ↓                           ↓
   RETRIEVAL                    GENERATION
        │                           │
   Recall@K                      Answer
   Precision@K                      │
   MRR                              │
   nDCG                             │
        │                           │
        └─────────────┬─────────────┘
                      ↓
                LLM EVALUATORS
                      │
         ┌────────────┼────────────┐
         ↓            ↓            ↓
    Faithfulness   Relevance   Completeness
         │            │            │
         └────────────┼────────────┘
                      ↓
              HUMAN CALIBRATION
                      │
                      ↓
             REGRESSION TESTING
                      │
                      ↓
              OBSERVABILITY
                      │
                      ↓
                PRODUCTION
Enter fullscreen mode Exit fullscreen mode

Frameworks such as RAGAS and ARES demonstrate different approaches to automating this layer, while ARES additionally incorporates human-labelled examples and statistical calibration.

  1. What LLM-as-a-Judge Should and Shouldn't Do Use it for: semantic relevance, faithfulness, completeness, qualitative answer quality, pairwise comparison, nuanced failure detection, scalable evaluation. Don't use it blindly for: exact numerical calculations, JSON validity, deterministic business rules, unit tests, schema validation, executable code correctness, simple string matching.

The strongest evaluation architecture combines:

Deterministic Metrics
+
LLM Evaluation
+
Human Evaluation
+
Reference-Based Tests

rather than choosing one methodology.

  1. The Senior-Level Mental Model

A beginner asks:

"Does my RAG answer correctly?"

An intermediate engineer asks:

"Did retrieval return the correct documents?"

A stronger engineer asks:

"Is the generated answer relevant and grounded in those documents?"

A production AI engineer asks:

"Can I continuously measure retrieval quality, grounding, correctness, completeness, cost, latency, and failure modes—and detect regressions when the system changes?"

That is the difference between building a RAG demo and engineering a RAG system.

  1. Final Perspective

RAG evaluation should not be treated as an afterthought.

It should be treated as an engineering subsystem.

The progression looks like:

RAG

├── Retrieve knowledge


Adaptive RAG

├── Choose retrieval strategy


Corrective RAG

├── Detect poor retrieval


RAG Evaluation

├── Measure system behavior


LLM-as-a-Judge

└── Scale semantic evaluation

And the final architecture becomes:

         BUILD
           ↓
         TEST
           ↓
         EVALUATE
           ↓
      FIND FAILURES
           ↓
         FIX
           ↓
       REGRESSION
           ↓
       DEPLOY
           ↓
      OBSERVE
           ↓
      EVALUATE AGAIN
Enter fullscreen mode Exit fullscreen mode

The key lesson is simple:

A RAG system isn't production-ready because it can generate impressive answers.

It is production-ready when we can systematically measure whether those answers are relevant, grounded, complete, correct, and reliable—and detect when a change makes them worse.

And perhaps the most important principle of all:

LLM-as-a-Judge should not replace evaluation discipline. It should make evaluation scalable.

The judge itself must be tested.

Its prompts must be versioned.

Its biases must be measured.

Its scores should be calibrated against humans.

And whenever a deterministic metric can answer a question more reliably, we should use the deterministic metric.

That is how LLM-as-a-Judge moves from an interesting prompting technique to a serious production evaluation architecture.

Research foundation

This approach is aligned with several important research directions:

RAGAS — EACL 2024: reference-free evaluation across retrieval, generation, and faithfulness dimensions.
ARES — NAACL 2024: automated RAG evaluation for context relevance, answer faithfulness, and answer relevance, with human calibration and prediction-powered inference.
MT-Bench / LLM-as-a-Judge — NeurIPS 2023: large-scale investigation of LLM judges, including their agreement with human preferences and known biases.
Large Language Models Are Not Fair Evaluators — ACL 2024: empirical analysis of positional bias and calibration strategies for LLM-based evaluation.
G-Eval: investigation of LLM-based evaluation and its relationship with human judgments, including evaluator bias.

Top comments (0)