LLM Evaluation Pipelines: Golden Sets, Cosine Similarity, LLM-as-Judge for Data Teams

llm evaluation is the discipline that separates a demoable prototype from an LLM feature you can put in front of paying customers — and, in 2026, it is squarely a data-engineering problem, not a research one. Every product team ships at least one language-model feature, every prompt change silently reshapes the answer distribution, and every retrieval index update touches the same measured quality surface. Without a versioned golden set, a numeric metric, and a llm eval pipeline that runs on every pull request, "did we make it worse?" becomes a Slack argument instead of a graph. The data-engineering signal an interviewer looks for is exactly the one that keeps a product shippable: rigorous inputs, deterministic scoring, cost-aware judge design, and CI hooks that block a regression before it ever reaches production.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through how you'd design a llm as judge scorer without letting the judge outvote your labels" or "when do you trust cosine similarity and when do you escalate to a rubric-scoring judge?" or "how do you keep a rag evaluation harness green through a text-embedding upgrade?" It walks through why LLM evaluation is a data-eng problem (golden-set discipline, drift monitoring, metric hygiene), how to curate and version a golden set the way you version code, the mechanics of cosine similarity scoring with text-embedding-3 and threshold calibration, the LLM-as-judge rubric with structured JSON output and cost control, and the CI + production monitoring loop that catches regressions before they ship. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works. ragas, deepeval, and promptfoo show up where they earn their place; the rest is the plumbing you'll build yourself.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the data-analysis practice library →, and sharpen the tuning axis with the optimization practice library →.

---

On this page

• Why LLM evaluation is a data-eng problem
• Golden sets — curation and versioning
• Cosine similarity + embedding-based scoring
• LLM-as-judge — GPT-4-class rubric scorer
• CI integration + production monitoring
• Cheat sheet — LLM eval recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why LLM evaluation is a data-eng problem

Evaluation, not modeling, is what keeps LLM features shippable — and it is pipelines, not papers

The one-sentence invariant: an LLM feature without a versioned golden set and a nightly llm eval pipeline is a research prototype pretending to be a product, and every prompt change is a silent regression risk until the evaluation harness proves otherwise. In 2026, most teams do not train models — they compose a base model with a prompt, a retrieval index, and a set of tools. The engineering surface that decides whether the product stays good is not "which model did you fine-tune," it is "which inputs did you preserve, which metric did you compute, and did the CI harness block yesterday's bad PR." That is a data-engineering problem end to end: curated inputs, deterministic scoring, versioned artefacts, and observable outputs.

The four "must-answer" axes interviewers actually probe.

• Golden set. A curated, versioned set of (input, expected_output, metadata) tuples that the harness scores against on every change. The senior signal is how you curate it: SME-labelled, class-balanced, git-hashed, growing every week. "We use the same 20 examples we tested with in the demo" is a fail.
• Metric. Which numeric score are you computing per row and how do you aggregate it? Exact match is too strict; free-form BLEU is meaningless; cosine similarity on embeddings is a strong baseline; llm as judge rubric scores are the modern default. The senior signal is knowing which metric for which task and how they fail.
• LLM-as-judge. A GPT-4-class model scoring the prediction against the expected answer using a rubric, returning a structured score plus a rationale. Cheap enough to run per PR, expensive enough that you cache and sample intelligently. The senior signal is understanding judge bias, prompt sensitivity, and how you'd guard against them.
• CI integration. The harness runs on every pull request, blocks the merge if the aggregate score drops by more than a configured threshold (typically 5 percent), and posts the diff into the PR comment. In production, you sample live traffic through the same harness and alert on drift. The senior signal is treating quality as a build gate, not a lagging indicator.

Why this is a data-engineering problem, not a modeling one.

• The scarce resource is data, not compute. The bottleneck to shipping a better LLM feature is not a bigger model — it is a bigger, cleaner, better-labelled golden set plus a metric you trust. Both are data problems.
• The pipeline shape is DAG-native. Load golden set → run predictions → embed → compare to expected → score → aggregate → alert. That is a DAG. It runs on the same Airflow / Dagster / Prefect scheduler you already have. The primitives (idempotent tasks, retry-on-failure, backfill by run_id) are exactly the data-engineering primitives you use for every other pipeline.
• The failure modes are drift, freshness, and schema. Embedding-model version changes silently reshape cosine distributions; retrieval index refreshes drop coverage; prompt edits change the answer surface. These are drift and schema-evolution problems — the same problems you handle in every production data platform.
• The observability story is metric surfaces. Per-row scores go into a warehouse table; nightly aggregates roll into a dashboard; alerts fire on regression. This is the "metric with a runbook" pattern you already run for every SLO. LLM-as-judge is a new signal source; the observability shape is the same.

Why "eval on demo examples" is the default failure mode.

• The demo set is 20 rows. It is what the founder used to prove the product to the first ten customers. It has no class balance, no edge cases, no adversarial inputs. Scoring against it produces a green graph that promises nothing about production behaviour.
• The demo set is not versioned. Rows get added ad hoc, expected answers get edited when a stakeholder pushes back, provenance is lost. The metric on Tuesday is not comparable to the metric on Friday. You cannot regress against a moving target.
• The demo set has no metadata. No difficulty tag, no topic label, no "who wrote this expected answer." When the aggregate score drops, you cannot slice by anything and cannot find the regression's centre of mass.
• The demo set is not calibrated. If cosine similarity of 0.87 counts as "correct," the team invented that threshold once, in a Colab notebook, three months ago, and never revisited. The threshold no longer matches the underlying embedding distribution — the metric drifts silently.

What senior interviewers listen for.

• Do you frame LLM evaluation as "versioned inputs plus a deterministic metric plus a CI hook" rather than "we manually spot-check outputs"? — senior signal.
• Do you name cosine similarity, LLM-as-judge, and exact/regex as the metric family and know which for which task? — required answer.
• Do you push back on "eval on demo examples" with the class-balance and version-control argument? — required answer.
• Do you treat evaluation cost (judge dollars per row × golden-set size × runs per week) as a first-class engineering constraint? — senior signal.

Worked example — the "demo set" failure mode quantified

Detailed explanation. The archetypal LLM-eval failure: a team ships a RAG chatbot backed by a 20-row demo set. Six weeks later they roll out a prompt tweak, the demo score stays green at 0.94 cosine, and a customer support flood reveals the tweak has cratered the answer quality on a whole family of questions the demo set never covered. Walk an interviewer through why the 20-row set could not catch this and what the correct golden-set size looks like.

• The symptom. Support tickets spike from 5/day to 45/day within 24 hours of a prompt deploy. Metric dashboards are green.
• The naive assumption. "The demo passes, therefore the prompt is fine."
• The actual failure. The demo covers "what are your business hours" style questions; the prompt regression touched "how do I cancel a subscription" style questions. Zero overlap.
• The fix. A golden set of at least 200–500 rows, class-balanced across question topics, with per-row topic metadata so slicing by class catches localized regressions.

Question. A team has a 20-row demo set with cosine 0.94 average on the current prompt. They plan to roll out a prompt change. Quantify the statistical power of scoring the change against 20 rows vs 200 rows, and derive the minimum golden-set size for a 5 percent regression to be detectable.

Input.

Parameter	Demo set	Golden set	Notes
Rows	20	200	10× fanout
Cosine standard deviation per row	0.08	0.08	Empirically measured
Standard error of mean	0.018	0.006	σ / √N
Detectable regression (2σ)	0.036	0.011	95% CI half-width
5% regression threshold	0.047	0.047	Business-defined
Detectable at 5% threshold?	barely	comfortably	2× vs 4× margin

Code.

import numpy as np

def detectable_regression(sigma_per_row: float, n_rows: int, confidence: float = 0.95) -> float:
    """
    Return the smallest regression (in the same units as sigma_per_row) that a
    golden set of size n_rows can reliably detect at the given confidence level.

    Uses a two-sided normal approximation: half-width = z * sigma / sqrt(n).
    """
    z = 1.96 if confidence == 0.95 else 2.58   # 95% or 99%
    sem = sigma_per_row / np.sqrt(n_rows)
    return z * sem


# The team's numbers
sigma = 0.08                # per-row cosine std dev
threshold = 0.05            # 5% business-defined regression threshold

for n in (20, 100, 200, 500, 1000):
    delta = detectable_regression(sigma, n)
    verdict = "yes" if delta < threshold else "no"
    print(f"n={n:>5d}  detectable={delta:.4f}  <threshold? {verdict}")

# Output:
# n=   20  detectable=0.0350  <threshold? yes  (barely, uses full CI budget)
# n=  100  detectable=0.0157  <threshold? yes
# n=  200  detectable=0.0111  <threshold? yes
# n=  500  detectable=0.0070  <threshold? yes
# n= 1000  detectable=0.0050  <threshold? yes

Step-by-step explanation.

1. The statistical setup: score deltas are approximately normal by CLT; the two-sided 95 percent confidence half-width on the mean is 1.96 × σ / √N. That half-width is the smallest regression the harness can distinguish from noise.
2. At N=20, the half-width is 0.035 — meaning any regression under 3.5 percent is indistinguishable from run-to-run variance. A 5 percent regression is barely detectable and only against a single quiet baseline; a 3 percent regression sails through unnoticed.
3. At N=200, the half-width drops to 0.011 — the harness can distinguish a 1.1 percent regression from noise. A 5 percent regression is caught with a 4× margin, and per-class slices (topic buckets) each have enough rows to be statistically meaningful.
4. The bigger reason 200 rows beats 20 is coverage. Statistical power is the floor; class coverage is the actual protection. A 200-row set with 10 topic buckets × 20 rows each catches localized regressions the 20-row demo never sees. A 20-row set has no coverage at all.
5. The team's failure was both: statistical power too low to distinguish a real change from noise, and coverage too narrow to see the class where the regression actually landed. Both problems dissolve at 200+ rows with balanced classes.

Output.

Golden-set size	Half-width (95% CI)	Detects 5% regression?	Detects 1% regression?	Practical use
20	0.035	barely	no	Demo only
100	0.016	yes	no	Development
200	0.011	yes	barely	Production baseline
500	0.007	yes	yes	Multi-slice production
1000	0.005	yes	yes	Regression-forensics grade

Rule of thumb. Golden-set size ≥ 200 rows, class-balanced across the top 10 question topics, is the minimum threshold for shipping an LLM feature to real users. Below 200 rows, the harness is decoration — the metric graph is green because the noise is louder than the signal.

Worked example — the four axes on a single interview slide

Detailed explanation. A senior interviewer often asks "walk me through your LLM evaluation architecture in one minute." The four-axis framing is the tightest possible response: golden set, metric, judge, CI. Practise deriving them from a concrete task — "customer support RAG chatbot" — so the answer is a mini design doc, not a list of buzzwords.

• The four axes as questions. Which inputs? Which metric? Who judges? When does it run?
• The concreteness test. Every axis should have a numeric answer for the task at hand.
• The senior signal. Naming the metric family (cosine + judge, or judge + regex, or embedding + regex), not just "we evaluate."

Question. Fill in the four-axis architecture for a customer-support RAG chatbot: 500 daily support conversations, historical ticket-plus-resolution corpus, GPT-4o-mini as production model, GPT-4o as available judge, PR CI budget of 5 minutes total. Show the numbers.

Input.

Parameter	Value
Production model	GPT-4o-mini
Judge model (available)	GPT-4o
Daily live conversations	500
Historical ticket-plus-resolution corpus	~40,000 rows
PR CI budget	5 minutes
Judge cost per row	~$0.006 (GPT-4o at ~1500 tokens)
Judge budget per run	$2.00

Code.

# eval-config.yaml — the four axes for the support chatbot
golden_set:
  path: golden/support_v3.jsonl
  version: v3.2-sha-e1f2c9
  rows: 300
  class_balance:
    billing: 60
    account: 60
    product: 60
    shipping: 60
    other: 60
  curation: SME-labelled + LLM-augmented, weekly refresh
  growth_rate: +20 rows/week

metric:
  primary: llm_judge_correctness   # 0.0 to 1.0
  secondary: cosine_similarity      # sanity check, threshold 0.85
  aggregation: mean_over_rows

judge:
  model: gpt-4o
  prompt_hash: c93f11
  rubric:
    - correctness
    - completeness
    - style
  output_schema: {score: float, rationale: string, breakdown: {correctness: float, completeness: float, style: float}}
  cost_per_row_usd: 0.006
  runs_per_pr: 1
  budget_per_run_usd: 2.00

ci:
  trigger: on pull_request
  gate: aggregate_score >= previous_baseline - 0.05
  timeout_minutes: 5
  parallelism: 20
  slack_channel: '#llm-quality'

Step-by-step explanation.

1. The golden set is 300 rows split evenly across 5 topic classes — the class-balance chip means a topic regression cannot be masked by aggregate improvement elsewhere. Version stamp v3.2-sha-e1f2c9 is the git hash of the golden-set repo commit; every metric row is joined to it in the warehouse.
2. The metric mix is llm_judge_correctness (primary) plus cosine_similarity (secondary sanity check). The judge is expensive but authoritative; cosine is cheap and catches "the model hallucinated a URL that has zero embedding overlap." Together they cover both semantic and structural failure.
3. The judge is GPT-4o with a rubric of correctness / completeness / style. Prompt hash c93f11 versions the judge prompt itself — a prompt edit bumps the hash and the harness knows to re-run baseline scores rather than compare across judge-prompt versions.
4. CI cost math: 300 rows × $0.006 per row = $1.80 per PR run, comfortably under the $2.00 budget. With 20-way parallelism and per-row latency around 2 seconds, wall-clock time is (300 / 20) × 2s = 30s — well within the 5-minute PR budget.
5. The gate blocks the merge if the aggregate correctness drops by more than 5 percent below the previous baseline. "Previous baseline" is the last main-branch merge, not an all-time high — otherwise a one-time high score locks the team out of every future PR.

Output.

Axis	Numeric answer
Golden set	300 rows, v3.2-sha-e1f2c9, 5 classes × 60 rows
Metric	LLM-judge correctness (primary) + cosine 0.85 threshold (secondary)
Judge	GPT-4o with 3-criterion rubric, $0.006/row, $1.80/run
CI	PR gate, block on ≥ 5% aggregate drop vs previous baseline, 5-min budget

Rule of thumb. If any of the four axes has a qualitative answer ("we spot-check," "we use vibes," "we run it weekly"), the architecture is not ready to ship. Every axis should have a number an interviewer can quote back at you.

Worked example — evaluation cost as a first-class constraint

Detailed explanation. A common mistake in the first evaluation build-out: no one runs the numbers on what the judge actually costs at scale. The team ships a 2000-row golden set with an LLM-as-judge that costs $0.03 per row, and the CI bill lands at $60 per PR. At 40 PRs per week, that is $10,000 per year on eval alone — and the team has zero budget for the production monitoring layer that is the actual point. Walk through the cost model and the levers that keep it sane.

• Cost per PR. golden_set_size × judge_cost_per_row.
• Cost per year. cost_per_PR × PRs_per_week × 52.
• The three levers. Reduce set size, reduce judge cost, reduce runs per PR.

Question. Given a 2000-row golden set, $0.03 per-row judge cost, and 40 PRs per week, quantify the annual eval bill and propose three optimizations that cut it by at least 5×.

Input.

Parameter	Baseline	After optimization
Golden set size	2000	400 (core) + 1600 (sample)
Judge cost per row	$0.03	$0.006 (smaller judge on 90% of rows)
Runs per PR	1 full	1 sample (400 rows) + 1 full nightly
PRs per week	40	40
Cost per full run	$60	$12 (nightly)
Cost per PR run	$60	$2.40 (sample only)
Cost per week	$2,400	$96 (PRs) + $84 (nightly) = $180
Cost per year	$124,800	$9,360

Code.

from dataclasses import dataclass

@dataclass
class EvalCostModel:
    golden_set_size: int
    judge_cost_per_row_usd: float
    prs_per_week: int
    runs_per_pr: int = 1
    nightly_full_runs: int = 0
    nightly_row_multiplier: float = 1.0   # 1.0 = full set; 0.2 = 20% sample
    weeks_per_year: int = 52

    def cost_per_pr_run(self) -> float:
        rows = int(self.golden_set_size * self.nightly_row_multiplier)
        return rows * self.judge_cost_per_row_usd

    def annual_cost_usd(self) -> float:
        pr_cost = self.cost_per_pr_run() * self.runs_per_pr * self.prs_per_week
        nightly_cost = self.golden_set_size * self.judge_cost_per_row_usd * self.nightly_full_runs * 7
        return (pr_cost + nightly_cost) * self.weeks_per_year


baseline = EvalCostModel(
    golden_set_size=2000,
    judge_cost_per_row_usd=0.03,
    prs_per_week=40,
)
print(f"Baseline: ${baseline.annual_cost_usd():,.0f}/yr")   # $124,800

# Optimization 1 — sample 20% on PR, full run nightly
opt1 = EvalCostModel(
    golden_set_size=2000,
    judge_cost_per_row_usd=0.03,
    prs_per_week=40,
    nightly_row_multiplier=0.2,
    nightly_full_runs=1,
)
print(f"Opt 1 (sample + nightly): ${opt1.annual_cost_usd():,.0f}/yr")

# Optimization 2 — cheaper judge (GPT-4o-mini instead of GPT-4)
opt2 = EvalCostModel(
    golden_set_size=2000,
    judge_cost_per_row_usd=0.006,
    prs_per_week=40,
    nightly_row_multiplier=0.2,
    nightly_full_runs=1,
)
print(f"Opt 2 (+cheap judge): ${opt2.annual_cost_usd():,.0f}/yr")

Step-by-step explanation.

1. The baseline math is brutal: 2000 rows × $0.03 × 40 PRs × 52 weeks = $124,800 per year, all on evaluation before a single dollar of production monitoring. The finance team will ask why.
2. Optimization 1 splits the harness into a fast PR run (20 percent sample = 400 rows, still statistically powerful) and a thorough nightly run over the full 2000. PR cost drops 5×; the nightly cost is 7 runs per week at full size, but only one team pays for it, not every PR author.
3. Optimization 2 stacks a cheaper judge on top. GPT-4o-mini is roughly 5× cheaper than GPT-4o for judge tasks and, for well-designed rubric prompts, correlates with GPT-4o judgements at r > 0.9 (measured with a 200-row A/B). Use the cheap judge on the PR sample; keep the expensive judge on the nightly full run for calibration.
4. The combined effect: $124,800 → $9,360, a 13× reduction, without cutting the golden set. Coverage is preserved via the nightly full run; PR sensitivity is preserved via the sample-based gate; the judge-cost sensitivity is spent where it matters (calibration), not where it does not (every PR).
5. The pattern that generalizes: sample for speed, calibrate on the full set, use a cheaper judge for the sample. It is the exact same pattern as "sample for AB tests, batch for correctness reports" — an old data-engineering pattern, applied to eval.

Output.

Configuration	Cost per PR	Cost per year	Coverage
Baseline (full set, expensive judge)	$60	$124,800	Full every PR
Opt 1 (sample 20% + nightly full)	$12	$16,320	Full nightly, sample per PR
Opt 2 (opt 1 + cheap judge)	$2.40	$9,360	Full nightly (expensive), sample per PR (cheap)
Opt 3 (opt 2 + reduce PRs to weekly nightly only)	n/a	$2,184	Nightly only, no PR gate

Rule of thumb. The judge bill is a data-engineering budget item, not a research afterthought. Model it before you build the harness; sample-on-PR + full-nightly + cheap-judge-with-expensive-calibration is the default architecture that scales.

Senior interview question on the LLM evaluation architecture

A senior interviewer often opens with: "Design an end-to-end LLM evaluation system for a mid-size product team shipping a RAG chatbot. Cover golden set, metric, judge, CI, production monitoring, and cost. Walk me through the concrete numbers and the trade-offs you'd make."

Solution Using a four-axis pipeline with golden set + cosine + judge + PR gate

# eval-arch.yaml — the reference architecture
version: v1.0

golden_set:
  storage: git-versioned JSONL
  path: golden/support_v3.jsonl
  rows: 300
  balance: 5 classes × 60 rows
  growth: +20 rows/week from production sampling
  provenance: SME-labelled + LLM-augmented
  version_stamp: v3.2-sha-e1f2c9

metrics:
  primary:
    name: llm_judge_correctness
    scale: 0.0-1.0
    judge_model: gpt-4o-mini      # sample judge
    calibration_model: gpt-4o     # nightly full-set judge
  secondary:
    name: cosine_similarity
    embedding_model: text-embedding-3-large
    threshold: 0.85               # calibrated per class
    role: sanity check + hallucination detector

harness:
  execution: DAG (Dagster / Airflow / Prefect)
  parallelism: 20
  timeout: 5 minutes
  storage: warehouse table eval_runs (run_id, golden_set_hash, prompt_hash, model, row_id, score, rationale)

ci:
  trigger: on pull_request touching prompts/, retrieval/, or model_config/
  sample: 400 rows (20% of full set)
  gate: mean_correctness >= previous_baseline - 0.05
  comment: post per-class delta table to PR
  budget: $2.40 per PR

nightly:
  trigger: cron 02:00 UTC daily
  scope: full 2000-row set with expensive judge (gpt-4o)
  gate: none (reports only, feeds baseline)
  budget: $12 per run

production_monitor:
  sampler: 1% of live conversations
  frequency: hourly aggregation
  metric: llm_judge_correctness (cheap judge)
  alert: 24h rolling mean drops > 5% vs last-week baseline
  cost: ~$50/week

Step-by-step trace.

Layer	Input	Output	Cost
Golden set	git commit v3.2-sha-e1f2c9	300 rows loaded	$0
Model inference	300 predictions	prediction column	~$0.30
Cosine scoring	prediction + expected	cosine column	$0.03 (embeddings)
Judge scoring (sample)	400 rows	judge score + rationale	$2.40
Aggregation	per-class + overall means	dashboard row + PR comment	$0
CI gate	mean vs baseline	pass or block	$0
Nightly	full 2000	reports	$12
Production sampler	1% traffic	continuous score	~$50/week

After the rollout, every PR touching prompts or retrieval runs a 400-row sample eval in ~30 seconds, posts a per-class delta table into the PR, and blocks the merge if aggregate correctness drops by more than 5 percent. The nightly full-set run recalibrates the baseline and catches slow drift the PR sample cannot see. Production monitoring samples 1 percent of live traffic through the same judge and alerts on 24-hour drops.

Output:

Metric	Value
PR eval wall-clock	~30 seconds (parallelism = 20)
PR eval cost	$2.40
Nightly eval wall-clock	3 minutes
Nightly eval cost	$12
Production monitor cost	~$50/week
Annual total	~$9,400
Detectable regression at PR gate	~1% (400-row sample)
Detectable regression nightly	~0.5% (2000-row full)

Why this works — concept by concept:

• Golden set as the versioned input — every score is joined to golden_set_hash so the metric is comparable across runs. Editing the golden set bumps the hash and the harness knows to re-baseline rather than compare across incompatible inputs. This is the same "immutable-input, versioned-artifact" pattern that dbt uses for models.
• Cosine as the cheap sanity check — cosine similarity on text-embedding-3-large costs about $0.0001 per row and catches structural failures (hallucinated URLs, wrong entities, missing named entities) at essentially zero cost. It is the cheap first line of defence before the expensive judge fires.
• Judge with sample-on-PR + full-nightly — the sample keeps PR feedback fast and cheap; the nightly keeps the baseline anchored on the full set. The pattern is exactly the CI-vs-CD split from every mature data-engineering platform: fast on the write path, thorough on the batch path.
• CI gate at 5% below previous baseline — the gate is relative to the previous baseline, not an all-time high, so the team can ratchet quality upward without accidentally trapping themselves under an unreproducible peak. The 5 percent tolerance leaves room for judge noise while still catching real regressions.
• Cost — total annual eval cost lands at ~$9,400 for a mid-size product, dominated by production monitoring rather than PR gating. The alternative (unmeasured regressions shipping to prod) costs a single incident — the eval stack pays for itself the first time it catches a bad prompt.

ETL
Topic — etl
ETL problems on LLM evaluation pipelines

Practice →

Optimization Topic — optimization Optimization problems on eval cost and coverage

Practice →

---

2. Golden sets — curation and versioning

A golden set is versioned code, not a spreadsheet — curation discipline is the single biggest lever on eval quality

The mental model in one line: a golden set is (input, expected_output, metadata) tuples treated with the same versioning, review, and observability rigour you apply to database migrations — SME-labelled, class-balanced, git-hashed, growing weekly, joined to every downstream metric row by its content hash. Everything else in the LLM eval stack is downstream of the golden set; a bad golden set produces green metrics that promise nothing. This is the single highest-leverage discipline in llm evaluation.

The four axes of a golden set worth trusting.

• Tuples. Every row is (input, expected_output, metadata). input is the exact string that would go into your production model. expected_output is the SME-labelled canonical answer. metadata is the rich sidecar: topic, difficulty, source ticket id, expected style, forbidden patterns, retrieval-index snapshot at label time.
• Versioned. The golden set lives in a git repo. Every edit is a PR, gets code review, gets a content hash. Downstream metric rows join to the hash — comparing scores across hashes is disallowed by construction.
• Balanced. Rows are distributed across the classes your product actually serves: topic buckets, difficulty tiers, language / locale / persona. Class imbalance in the golden set means the aggregate score is dominated by whichever class has the most rows, and localized regressions in under-represented classes vanish into the mean.
• Growing. Golden sets are not static. Every week, sample fresh production conversations, SME-label them, add to the set. A stagnant golden set drifts away from the live traffic distribution and stops being predictive within months.

The curation loop.

• Source. Sample from live production traffic — the golden set must reflect the actual input distribution, not an idealized version of it. For a chatbot: sample real user turns. For extraction: sample real documents.
• Label. Route to an SME queue. For customer support: a senior support engineer. For medical Q&A: a clinician. For code review: a staff engineer. The SME writes the expected answer; a peer reviews it; the reviewed row is merged.
• Augment. LLM-augmented golden sets — where a strong model generates the expected answer and an SME reviews it — are the compromise for scale. Purely LLM-generated golden sets are self-referential garbage; SME-reviewed LLM-generated sets are 5–10× cheaper than SME-authored ones while retaining most of the quality.
• Balance. After every batch, re-check the class distribution. If billing questions are now 40 percent of the set and shipping questions are 5 percent, rebalance before the next merge.
• Hash + release. The final artefact is a JSONL file with a content hash. v3.2-sha-e1f2c9 means "golden set version 3.2, content hash e1f2c9." Every downstream metric row carries this hash.

Version control patterns that work.

• Storage. JSONL in a git repo (fast to diff, streams row-by-row into the harness). Alternative: parquet with an accompanying MANIFEST.json. Never a Google Sheet.
• Review. PRs against the golden-set repo. Every edit is reviewed by at least one SME. Bulk merges (>50 rows) get double review.
• Hash. Content hash computed on canonicalized JSON (sorted keys, no trailing whitespace) — SHA-256 first 6 hex is enough. Store in a MANIFEST.json alongside the file; harness verifies at load time.
• Rollback. Git tags per release (golden-v3.2). If a bad batch is merged, revert the merge commit and re-tag. Downstream metric rows keyed by hash never confuse pre- and post-revert.

Common curation failures and their fixes.

• "We use production traffic samples directly." Fails on labeler bias — the model's own output becomes the expected answer. Fix: SME must write the expected answer independently, without seeing the model's response.
• "We only add rows when the model gets something wrong." Fails on class balance — the golden set becomes an adversarial edge-case set with no representative easy rows, and the aggregate metric is meaningless. Fix: add rows from all outcome buckets (correct, partially-correct, wrong) at fixed ratios.
• "We edit expected answers when the SME disagrees with the score." Fails on immutability — the metric is no longer comparable across time. Fix: expected-answer edits go through the same PR-and-hash-bump process as adding new rows.
• "We don't rebalance." Fails on drift — organic growth is never balanced, and the metric quietly gets dominated by whichever class the SMEs happen to label most. Fix: automatic imbalance detection with a warning below 20% deviation from target ratios.

What senior interviewers listen for.

• Do you talk about the golden set as a first-class artefact with version control and PR review, rather than as a file someone maintains? — senior signal.
• Do you volunteer the class-balance chip without prompting? — required answer.
• Do you distinguish SME-labelled from LLM-augmented and defend both use cases? — senior signal.
• Do you name content-hash + downstream-join as the mechanism that keeps the metric comparable across time? — senior signal.

Worked example — schema for a 300-row customer-support RAG golden set

Detailed explanation. Design the JSONL schema for a customer-support RAG golden set. Every row must carry enough metadata to slice by topic, difficulty, language, expected style, and provenance. Show the actual JSON with three representative rows and walk through why every field earns its place.

• Row shape. input, expected_output, metadata.
• Metadata fields. topic, difficulty, language, source_ticket_id, sme_labeler, reviewer, retrieval_snapshot_id, forbidden_patterns, style_tag.
• The senior test. Every field must have a downstream consumer (dashboard slice, regression forensic, harness gate).

Question. Produce the JSONL schema plus three example rows for a customer-support RAG golden set covering billing, account, and shipping topics, with an SME-labeler chain and a retrieval-snapshot pin.

Input.

Field	Type	Purpose
input	string	The user turn as it enters the RAG chatbot
expected_output	string	SME-authored canonical answer
metadata.topic	enum	Class bucket for balance + slice
metadata.difficulty	enum	Bucket for slice + power analysis
metadata.language	enum	Locale slice
metadata.source_ticket_id	string	Provenance to the original support ticket
metadata.sme_labeler	string	The SME who wrote the expected answer
metadata.reviewer	string	The peer who approved
metadata.retrieval_snapshot_id	string	Retrieval index at label time
metadata.forbidden_patterns	list	Regexes that must not appear in prediction
metadata.style_tag	enum	Expected tone (formal, empathetic, terse)

Code.

{"input": "How do I update my billing address on file?", "expected_output": "You can update your billing address by opening Settings → Billing → Address, entering the new address, and clicking Save. The change takes effect on your next invoice. If a payment is currently processing, contact support before saving.", "metadata": {"topic": "billing", "difficulty": "easy", "language": "en", "source_ticket_id": "T-8821", "sme_labeler": "alice@support", "reviewer": "bob@support", "retrieval_snapshot_id": "idx-2026-06-15", "forbidden_patterns": ["credit card", "bank account"], "style_tag": "formal"}}
{"input": "My charge from last month is wrong — I was billed twice.", "expected_output": "I'm sorry about the duplicate charge — that's frustrating. Please email [e-mail] with your account email and the two transaction IDs. Our team can reverse one of the charges within 3 business days. If you'd like, I can also open a ticket for you now.", "metadata": {"topic": "billing", "difficulty": "hard", "language": "en", "source_ticket_id": "T-8834", "sme_labeler": "alice@support", "reviewer": "carol@support", "retrieval_snapshot_id": "idx-2026-06-15", "forbidden_patterns": ["we'll refund", "guaranteed"], "style_tag": "empathetic"}}
{"input": "Where is my order? Tracking says shipped but I haven't received it.", "expected_output": "I understand — a shipped order that hasn't arrived is stressful. Please check the tracking link in your shipment email for the latest carrier update. If the tracking has been stagnant for more than 5 business days, reply here with your order number and I'll open a carrier claim on your behalf.", "metadata": {"topic": "shipping", "difficulty": "medium", "language": "en", "source_ticket_id": "T-8912", "sme_labeler": "dave@support", "reviewer": "alice@support", "retrieval_snapshot_id": "idx-2026-06-15", "forbidden_patterns": ["lost forever", "impossible"], "style_tag": "empathetic"}}

# harness/load_golden.py — canonicalized load + hash verification
import hashlib
import json
from pathlib import Path


def content_hash(rows: list[dict]) -> str:
    """SHA-256 first 6 hex over canonicalized JSON."""
    canonical = "\n".join(json.dumps(r, sort_keys=True, ensure_ascii=False) for r in rows)
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:6]


def load_golden(path: Path, expected_hash: str | None = None) -> list[dict]:
    rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
    actual_hash = content_hash(rows)
    if expected_hash and actual_hash != expected_hash:
        raise ValueError(f"golden set hash mismatch: expected {expected_hash}, got {actual_hash}")
    return rows


rows = load_golden(Path("golden/support_v3.jsonl"), expected_hash="e1f2c9")
print(f"loaded {len(rows)} rows, hash e1f2c9 verified")

Step-by-step explanation.

1. Every row carries input, expected_output, and a rich metadata block. The metadata is not optional — it is what turns a golden set into a diagnostic tool. Without topic and difficulty, regressions can only be described at the aggregate; with them, you can point to "billing / hard rows dropped 12 percent" in the PR comment.
2. forbidden_patterns is the row-level guardrail: regexes the prediction must not contain. Row 1's ["credit card", "bank account"] means the model should never invite the user to email their credit card number — an easily-detectable structural failure that pure cosine similarity might miss.
3. style_tag distinguishes expected tone. Row 2's "empathetic" means the judge rubric should score empathy explicitly; row 1's "formal" means the same phrase would score lower. The metric responds to the semantic and the pragmatic axis.
4. retrieval_snapshot_id pins the retrieval index state at label time — critical for RAG evaluation. When the retrieval index refreshes and the expected answer no longer matches what a correct retrieval would return, the row is flagged for re-labelling rather than silently failing.
5. The content_hash function computes SHA-256 over canonicalized JSON (sorted keys, one row per line). Downstream metric rows join to this hash; the harness verifies the hash on load. If the file was edited without a PR review, the hash mismatches and the run aborts.

Output.

Field	Purpose	Downstream consumer
input / expected_output	Core tuple	Model + judge inputs
topic	Class balance + slice	Dashboard slice
difficulty	Slice + power	Regression forensics
language	Locale slice	Multilingual dashboards
source_ticket_id	Provenance	Row-level audit
sme_labeler / reviewer	Trust chain	SME QA reports
retrieval_snapshot_id	RAG pinning	Index-refresh guard
forbidden_patterns	Row-level guardrail	Structural failure check
style_tag	Style discipline	Judge rubric input

Rule of thumb. Every metadata field must have a downstream consumer. If you cannot name what a field is used for in the dashboard, harness, or forensics report, drop it — noise metadata makes the schema harder to evolve without making the metric better.

Worked example — the class-balance rebalancer

Detailed explanation. Golden sets grow organically as SMEs add rows. Without discipline, the distribution drifts toward whatever the SMEs happen to work on that week. Build the class-balance rebalancer that runs on every merge to the golden-set repo, warns on drift, and refuses to merge if drift exceeds a threshold.

• Target ratios. 5 topic classes at 20 percent each (0.2 target per class).
• Warning threshold. ±5 percent deviation from target — flags the row but allows the merge.
• Blocking threshold. ±10 percent deviation — CI check fails.
• Fix. Rebalance by removing over-represented rows or by generating replacements in the under-represented classes.

Question. Build the class-balance check as a pre-merge CI script that reads the JSONL, computes per-class ratios, prints a warning or fails, and outputs the recommended rebalance action.

Input.

Class	Current rows	Target ratio	Current ratio	Delta
billing	78	0.20	0.26	+0.06
account	55	0.20	0.18	-0.02
product	60	0.20	0.20	0.00
shipping	45	0.20	0.15	-0.05
other	62	0.20	0.21	+0.01
total	300	1.00	1.00	—

Code.

# harness/check_balance.py
import json
import sys
from collections import Counter
from pathlib import Path


TARGET = {"billing": 0.20, "account": 0.20, "product": 0.20, "shipping": 0.20, "other": 0.20}
WARN = 0.05
BLOCK = 0.10


def check_balance(path: Path) -> int:
    rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
    n = len(rows)
    counts = Counter(r["metadata"]["topic"] for r in rows)

    exit_code = 0
    print(f"golden set has {n} rows")
    print(f"{'topic':<10} {'rows':>5} {'ratio':>7} {'target':>7} {'delta':>7}")
    for topic, target in TARGET.items():
        c = counts.get(topic, 0)
        ratio = c / n if n else 0.0
        delta = ratio - target
        marker = ""
        if abs(delta) >= BLOCK:
            marker = "  BLOCK"
            exit_code = 1
        elif abs(delta) >= WARN:
            marker = "  WARN"
        print(f"{topic:<10} {c:>5d} {ratio:>7.3f} {target:>7.3f} {delta:>+7.3f}{marker}")

    # Suggested action
    over = [t for t, target in TARGET.items() if counts.get(t, 0) / n - target > 0.05]
    under = [t for t, target in TARGET.items() if target - counts.get(t, 0) / n > 0.05]
    if over or under:
        print()
        print("Suggested rebalance:")
        for t in under:
            need = int((TARGET[t] - counts.get(t, 0) / n) * n)
            print(f"  add {need:+d} rows for {t}")
        for t in over:
            drop = int((counts.get(t, 0) / n - TARGET[t]) * n)
            print(f"  drop {drop:+d} rows for {t}")

    return exit_code


if __name__ == "__main__":
    sys.exit(check_balance(Path(sys.argv[1])))

# .github/workflows/golden-check.yml
name: golden-set-check
on:
  pull_request:
    paths: ['golden/**']
jobs:
  balance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: python harness/check_balance.py golden/support_v3.jsonl

Step-by-step explanation.

1. The check iterates over the 5 target classes and computes the current ratio versus the target. Deviations under ±5 percent are silent, ±5–10 percent print a WARN marker (informational), ±10 percent or worse print BLOCK and exit non-zero.
2. In the input scenario, billing is at 26 percent (+6 percent over target) — WARN. shipping is at 15 percent (-5 percent) — WARN. No class hits the BLOCK threshold, so the check exits 0 and the merge is allowed, but the PR author sees the warnings inline.
3. The suggested-rebalance block computes concrete row counts: shipping needs +15 more rows to hit target; billing needs to drop 18 rows. The suggestion drives the next batch of SME curation work.
4. The check runs on every PR that touches golden/** via GitHub Actions. It is a 30-second job on a 300-row set; even a 10,000-row set runs in under 5 seconds. The cost is negligible; the guardrail is absolute.
5. The pattern extends: add checks for difficulty distribution, language distribution, and sme_labeler diversity (no single SME should label more than 40 percent of any class — labeller bias). All of them are the same shape.

Output.

golden set has 300 rows
topic       rows   ratio  target   delta
billing       78   0.260   0.200  +0.060  WARN
account       55   0.183   0.200  -0.017
product       60   0.200   0.200  +0.000
shipping      45   0.150   0.200  -0.050  WARN
other         62   0.207   0.200  +0.007

Suggested rebalance:
  add +15 rows for shipping
  drop -18 rows for billing

Rule of thumb. Automate class-balance checks the same way you automate lint. The check is cheap, the failure mode is expensive; the moment a golden set is not automatically balanced, drift wins and the metric decays silently.

Worked example — versioning golden sets like code

Detailed explanation. Walk through the git workflow for a golden-set repo: PRs, review, hash, tag, harness pin. Show what happens when a batch of 20 new rows is added — from the SME's local edit through the merged tag to the downstream metric join.

• Repo shape. golden/support_v3.jsonl, golden/MANIFEST.json (with hash + row count), golden/CHANGELOG.md.
• PR flow. Branch → edit → local check → PR → review → merge → tag.
• Hash discipline. Content hash goes in MANIFEST + git tag + downstream metric rows.

Question. Walk through the PR flow for adding 20 new billing-topic rows to a 280-row golden set, ending with the new hash and tag, and show how the harness pins to the tag.

Input.

Step	Actor	Artefact
1. Sample	SME	20 real support tickets from the last week
2. Label	SME	expected_output for each
3. Peer review	Reviewer	approval or edits
4. Local check	SME	check_balance.py + hash regen
5. PR	SME	branch feature/golden-billing-batch-42
6. CI	GitHub Actions	balance check + row-count diff
7. Merge	Maintainer	squash to main
8. Tag	Maintainer	golden-v3.3
9. Harness pin	DE	eval-config.yaml → version: golden-v3.3

Code.

# 1-2. Sample + label (SME's local flow)
git checkout -b feature/golden-billing-batch-42
python harness/sample_from_prod.py --topic billing --n 20 > tmp/samples.jsonl
# SME edits tmp/samples.jsonl to write expected_output + metadata for each row

# 3. Peer review — SME opens PR, reviewer proposes edits, SME applies

# 4. Local check
cat golden/support_v3.jsonl tmp/samples.jsonl > golden/support_v3.jsonl.new
mv golden/support_v3.jsonl.new golden/support_v3.jsonl
python harness/check_balance.py golden/support_v3.jsonl   # WARN or BLOCK check
python harness/hash_manifest.py golden/support_v3.jsonl   # writes MANIFEST.json
git diff golden/MANIFEST.json                              # confirms hash bump

// golden/MANIFEST.json (after the merge)
{
  "version": "v3.3",
  "sha": "b83a11",
  "rows": 300,
  "class_balance": {
    "billing": 78,
    "account": 55,
    "product": 60,
    "shipping": 45,
    "other": 62
  },
  "generated_at": "2026-07-04T09:12:33Z",
  "generator": "harness/hash_manifest.py"
}

# eval-config.yaml — harness pins to a specific tag
golden_set:
  repo: [e-mail]:acme/golden-sets.git
  path: golden/support_v3.jsonl
  tag: golden-v3.3
  expected_sha: b83a11    # harness aborts if mismatch

-- Every metric row joins by (run_id, golden_set_sha)
SELECT
  m.run_id,
  m.golden_set_sha,
  m.row_id,
  m.prediction,
  m.judge_score,
  g.topic,
  g.difficulty
FROM   eval_metric_rows m
JOIN   golden_set_rows g
  ON   m.golden_set_sha = g.sha
  AND  m.row_id         = g.row_id
WHERE  m.run_id = :run_id;

Step-by-step explanation.

1. The SME branches off main, samples 20 real production tickets via a helper, and labels each with an expected answer + metadata. The samples are typed into JSONL in a local scratch file, then concatenated onto the existing golden set.
2. Peer review happens in the PR. The reviewer proposes edits to expected answers (typos, style, missing empathy), the SME applies them, and the PR is re-pushed. Only reviewer-approved rows land in main.
3. The local check runs check_balance.py — in this case billing moves from 58/280 (21 percent) to 78/300 (26 percent), triggering a WARN but not a BLOCK. The SME chooses to keep the imbalance for now with a plan to add shipping rows next batch.
4. hash_manifest.py computes the new SHA (b83a11) and writes MANIFEST.json. The diff on MANIFEST.json is what the reviewer signs off on — a mechanical audit trail of "row count went from 280 to 300, hash went from e1f2c9 to b83a11."
5. The harness pins to tag: golden-v3.3 + expected_sha: b83a11. On load, it verifies both — if the tag has been re-pointed or the file has been edited without a hash bump, the harness aborts and the run does not corrupt the metric table. Every metric row also carries the SHA so downstream analysis can never confuse pre- and post-v3.3 scores.

Output.

Artefact	Before	After
Rows	280	300
SHA	e1f2c9	b83a11
Tag	golden-v3.2	golden-v3.3
CHANGELOG.md entry	—	"v3.3: +20 billing rows from prod week 27"
Harness pin	v3.2	v3.3
Downstream metric join key	(run_id, e1f2c9)	(run_id, b83a11)

Rule of thumb. Golden sets deserve the same git rigour as production migrations: branch, PR, review, tag, hash-pin. Skipping any of these is how a Monday score becomes incomparable to a Friday score without anyone noticing until an interview.

Senior interview question on golden-set design

A senior interviewer might ask: "You inherit a team where the golden set is a shared Google Sheet with 40 rows that everyone edits. Walk me through the migration to a versioned, class-balanced golden set of 300 rows in 6 weeks, without stalling the team's evaluation cadence."

Solution Using a six-week golden-set migration plan with a hybrid harness

# migration-plan.yaml — 6 weeks from Google Sheet to versioned golden set
version: v1.0

week_1:
  goal: freeze current state + baseline metric
  actions:
    - snapshot the sheet as-is into git as golden/support_v0.jsonl
    - compute content_hash + tag as golden-v0-baseline
    - run the existing harness against v0-baseline for a 4-week baseline
  deliverable: golden-v0-baseline in git; baseline metric on dashboard

week_2:
  goal: schema migration
  actions:
    - migrate 40 rows to (input, expected_output, metadata) schema
    - add topic + difficulty + language metadata (SME retroactively fills)
    - stand up check_balance.py + GitHub Actions
  deliverable: golden-v1.0 in git (40 rows, schema-conformant)

week_3:
  goal: first curation batch — sample 60 fresh production rows
  actions:
    - build sample_from_prod.py (random sample from live conversations)
    - SME labels 60 rows across 5 topic classes (12 each)
    - peer review + merge
  deliverable: golden-v2.0 (100 rows, mostly balanced)

week_4:
  goal: second curation batch — 100 more rows
  actions:
    - sample another 100 rows, focusing on under-represented classes
    - LLM-augmented labels + SME review (cheaper labelling)
  deliverable: golden-v2.1 (200 rows, balance within ±5%)

week_5:
  goal: hybrid harness — score against both v0-baseline AND v2.1
  actions:
    - harness runs against both sets on every PR
    - dashboard shows both metrics side-by-side
    - team gets used to the new numbers before cutover
  deliverable: hybrid dashboard live

week_6:
  goal: cutover
  actions:
    - final +100 rows to reach 300
    - retire v0-baseline; harness gates on v3.0 only
    - CHANGELOG + team announcement + runbook
  deliverable: golden-v3.0 (300 rows) as sole gating set

Step-by-step trace.

Week	Activity	Rows	Metric availability
1	Freeze sheet, tag v0	40	Old-sheet baseline
2	Schema migration	40	v1.0 available
3	+60 rows	100	v2.0 available
4	+100 rows	200	v2.1 available
5	Hybrid harness	200	Both v0 and v2.1
6	Cutover to v3.0	300	v3.0 is the gate

After 6 weeks, the team has a 300-row, class-balanced, git-versioned golden set with automatic balance checks on every PR, a downstream metric table joined by SHA, and a runbook for the next migration when the schema changes again. The old sheet is archived; no one edits it.

Output:

Metric	Week 1	Week 6
Golden-set rows	40	300
Storage	Google Sheet	Git JSONL
Class balance	unmeasured	±5% per class
Version control	none	git tag + SHA
PR review	none	reviewer required
Downstream metric join	none	(run_id, SHA)
Detectable regression	~2.5%	~1.1%

Why this works — concept by concept:

• Freeze first, migrate second — the week-1 baseline lets the team detect regressions during the migration itself; without it, a schema change might silently mask a real quality drop. This is the same "capture golden state, then refactor" pattern you use for legacy pipeline migrations.
• Schema migration before curation — moving to the (input, expected_output, metadata) shape unblocks every future capability (slice by topic, class-balance check, forbidden-pattern guard). Trying to curate before the schema is stable produces rows you have to re-edit later.
• Hybrid harness in week 5 — running both metrics side-by-side is the safety net for cutover week. The team gets to compare "old sheet said 0.87, new golden set says 0.82" and understand the delta before it becomes a gate. Big-bang cutovers are how you break trust in the metric.
• LLM-augmented labelling in week 4 — SMEs write 100 rows in weeks 2–3, then LLM-augmented labelling with SME review handles the next 100. The 5–10× speedup on the second batch is what makes the 6-week timeline realistic on a mid-size team.
• Cost — 6 weeks of one SME's part-time attention plus roughly 40 senior-engineer hours for schema, harness, and CI plumbing. The alternative (running on a 40-row sheet forever) has an unbounded downside; the migration cost is paid once and the golden set compounds in value every week thereafter.

ETL
Topic — etl
ETL problems on versioned dataset curation

Practice →

Data analysis Topic — data-analysis Data-analysis problems on class balance and slicing

Practice →

---

3. Cosine similarity + embedding-based scoring

Cosine similarity is the cheap, fast, structural sanity check — and a lousy standalone truth metric

The mental model in one line: cosine similarity on embeddings of prediction and expected_output is the fastest, cheapest, most-deterministic LLM-eval metric — perfect as a sanity check and a hallucination detector, terrible as a standalone truth metric because it cannot distinguish paraphrase from contradiction. Every llm eval pipeline should compute cosine as the first metric on every row, and every serious rag evaluation stack should also compute an LLM-judge score on top for the semantic axis cosine misses.

The four axes of cosine-based scoring.

• Embedding model. The cosine score is a function of the embedding — text-embedding-3-large produces different distributions than text-embedding-3-small or bge-large. Pin the model, version-stamp the score column, and never change models mid-experiment.
• Threshold. A cosine of 0.85 might be "correct" on one embedding model and "irrelevant" on another. The threshold must be calibrated per embedding model per task against the golden set, not carried across from a tutorial.
• Failure modes. Cosine treats paraphrase and contradiction as similar. "Yes, you can cancel anytime" and "No, you cannot cancel" embed close to each other; cosine sails through, the judge (and the customer) does not.
• Aggregation. Per-row cosine is a float in -1, 1. Aggregate as mean over rows for the summary; also compute per-class means for the dashboard slice.

How embeddings become a metric.

• Step 1. Embed the expected_output string with the chosen embedding model; store as a float32[dim] vector (dim=3072 for text-embedding-3-large).
• Step 2. Embed the model's prediction with the same embedding model. The vectors live in the same space; cosine is defined.
• Step 3. Compute cosine similarity: dot(a, b) / (norm(a) × norm(b)). For normalized embeddings this is a dot product.
• Step 4. Compare to the threshold. Above threshold = "correct" (cheap-check pass); below = "flagged" (escalate to judge for the semantic axis).

Threshold calibration — the numeric discipline.

• Gather. 200 golden-set rows plus 200 known-wrong predictions (paraphrased edits, contradiction injections).
• Score. Compute cosine for all 400 pairs.
• Choose. Pick the threshold that maximizes F1 on the "correct vs wrong" classification. Typical values land at 0.75–0.90 for text-embedding-3-large on medium-length answers.
• Version-stamp. The threshold is a function of the embedding model + task; store it in the eval config with the model version. Bumping the embedding model always requires re-calibration.

Failure modes cosine cannot catch alone.

• Contradiction. "Yes" and "No" embed as nearby short strings. Cosine misses the polarity flip.
• Paraphrase. "Update via Settings → Billing" and "Change in the Billing settings menu" are semantically identical but embed differently on some models — cosine can under-count paraphrase correctness.
• Style mismatch. A perfectly correct answer in a formal tone will score high cosine against an empathetic expected answer even though the tone-brand is off.
• Long-form dilution. A 500-token answer embeds "average"-ish; a small hallucinated URL inside it barely moves the vector. Cosine is blind to needle-in-haystack failure.

When cosine actually shines.

• Structural hallucination detection. Predictions with wrong entities, wrong URLs, wrong dates score dramatically lower than paraphrased-but-correct alternatives. Cosine is a great first-pass anomaly detector.
• Regression detection. A prompt change that drops mean cosine by 0.10 across 300 rows is always a regression worth investigating, even if individual rows look fine.
• Speed. Embeddings cost fractions of a cent; cosine is a matrix multiply. Perfect for every-PR gating where an LLM-judge would be too slow or expensive.
• Determinism. Cosine is deterministic for a fixed embedding model. Unlike judges, the score is reproducible bit-for-bit — great for baseline math.

Common cosine pitfalls in the wild.

• Un-normalized embeddings. Forgetting to normalize the vectors turns cosine into "dot product times garbage." Always L2-normalize before comparing.
• Model version drift. OpenAI silently ships new embedding checkpoints under the same public name; pin to a specific model version and log it with every score row.
• Mixed languages. Cross-lingual cosine is much noisier than same-language. Slice the metric by language before drawing conclusions.
• Threshold from a Colab notebook. The threshold everyone quotes came from someone's exploration script and was never re-calibrated. Re-run calibration on every embedding-model bump.

What senior interviewers listen for.

• Do you name cosine as a sanity check, not a truth metric, and pair it with a judge? — required answer.
• Do you volunteer paraphrase vs contradiction as the canonical failure modes? — senior signal.
• Do you name threshold calibration as a per-embedding-model discipline rather than a constant? — senior signal.
• Do you distinguish normalized vs unnormalized and defend the L2 normalization step? — required answer.

Worked example — cosine scoring a 300-row golden set with text-embedding-3

Detailed explanation. Build the cosine scorer against a 300-row golden set using OpenAI's text-embedding-3-large. Batch the embedding calls for cost, L2-normalize the vectors, compute per-row cosine, aggregate by topic class, and produce the summary table with the calibrated 0.85 threshold.

• Embedding model. text-embedding-3-large (dim=3072).
• Batch size. 100 rows per API call.
• Threshold. 0.85 (previously calibrated).
• Aggregation. Per-class mean + overall mean + fraction ≥ threshold.

Question. Implement the cosine scorer end-to-end. Show the batch embedding, normalization, cosine computation, and per-class aggregation. Report the summary table.

Input.

Parameter	Value
Golden set	300 rows, 5 topic classes × 60
Embedding model	text-embedding-3-large (dim=3072)
Batch size	100
Threshold	0.85
Expected cost	300 × 2 embeddings × $0.00013 / 1K tokens × ~200 tokens = ~$0.016

Code.

# cosine_scorer.py
import json
from pathlib import Path
from typing import Iterable

import numpy as np
import pandas as pd
from openai import OpenAI

EMBED_MODEL = "text-embedding-3-large"
BATCH = 100
client = OpenAI()


def embed_batch(texts: list[str]) -> np.ndarray:
    """Return an (n, dim) float32 array of L2-normalized embeddings."""
    resp = client.embeddings.create(model=EMBED_MODEL, input=texts)
    vecs = np.asarray([d.embedding for d in resp.data], dtype=np.float32)
    norms = np.linalg.norm(vecs, axis=1, keepdims=True)
    return vecs / np.clip(norms, 1e-9, None)


def batched(items: list, size: int) -> Iterable[list]:
    for i in range(0, len(items), size):
        yield items[i : i + size]


def cosine_score(rows: list[dict], predictions: dict[str, str]) -> pd.DataFrame:
    """
    rows        : golden set (must include row_id, expected_output, metadata.topic)
    predictions : row_id -> model prediction string
    """
    expected = [r["expected_output"] for r in rows]
    predicted = [predictions[r["row_id"]] for r in rows]

    exp_vecs = np.vstack([embed_batch(b) for b in batched(expected, BATCH)])
    pred_vecs = np.vstack([embed_batch(b) for b in batched(predicted, BATCH)])

    # Dot product of L2-normalized vectors == cosine similarity
    cos = np.sum(exp_vecs * pred_vecs, axis=1)

    return pd.DataFrame({
        "row_id":    [r["row_id"] for r in rows],
        "topic":     [r["metadata"]["topic"] for r in rows],
        "cosine":    cos,
        "pass_0.85": cos >= 0.85,
    })


def summarize(df: pd.DataFrame) -> pd.DataFrame:
    by_topic = df.groupby("topic").agg(
        n=("row_id", "count"),
        mean_cosine=("cosine", "mean"),
        pass_rate=("pass_0.85", "mean"),
    ).round(4).reset_index()
    overall = pd.DataFrame([{
        "topic": "OVERALL",
        "n": len(df),
        "mean_cosine": df["cosine"].mean().round(4),
        "pass_rate": df["pass_0.85"].mean().round(4),
    }])
    return pd.concat([by_topic, overall], ignore_index=True)


if __name__ == "__main__":
    rows = [json.loads(l) for l in Path("golden/support_v3.jsonl").read_text().splitlines()]
    predictions = json.loads(Path("runs/run_2026-07-04.json").read_text())
    df = cosine_score(rows, predictions)
    print(summarize(df).to_markdown(index=False))

Step-by-step explanation.

1. embed_batch sends 100 strings at a time to OpenAI's embeddings endpoint, receives 100 3072-dim vectors, stacks them into a NumPy array, and L2-normalizes so cosine reduces to a plain dot product.
2. cosine_score embeds the 300 expected outputs and the 300 predictions in parallel batches (3 calls each), then computes cosine as an element-wise multiply-and-sum on the L2-normalized vectors. Total wall-clock is dominated by the embedding calls (~2 seconds each) — roughly 6 seconds for the full run.
3. Every row emits a (row_id, topic, cosine, pass_0.85) record. The pass_0.85 boolean is the threshold check; downstream aggregation treats it as a per-class pass rate.
4. summarize groups by topic to expose per-class means and pass rates, then appends an overall row. This is the table that goes into the PR comment — one row per topic plus overall.
5. Cost math: 300 × 2 embeddings × ~200 tokens/row = 120,000 tokens; at $0.00013 / 1K tokens for text-embedding-3-large, the scorer costs about $0.016 per full run. Effectively free at PR gate frequency.

Output.

topic	n	mean_cosine	pass_rate
account	60	0.9215	0.9500
billing	60	0.8842	0.8500
other	60	0.9051	0.9167
product	60	0.9127	0.9333
shipping	60	0.8776	0.8000
OVERALL	300	0.9002	0.8900

Rule of thumb. Cosine scoring is a pandas.groupby on top of a NumPy dot product. Keep it deterministic, cheap, and always paired with a judge for the semantic axis it cannot see.

Worked example — threshold calibration with a contradiction-injected set

Detailed explanation. Cosine thresholds are meaningless without calibration. Build the calibrator: take 200 golden-set pairs (correct) plus 200 contradiction-injected pairs (wrong), compute cosine on all 400, sweep the threshold to maximize F1, and produce the calibration curve.

• Positive class. Correct predictions (from the golden set itself, expected as prediction).
• Negative class. Contradiction-injected predictions (LLM-generated wrong-polarity answers reviewed by an SME).
• Sweep. Thresholds from 0.5 to 0.99 in 0.01 steps; report precision, recall, F1.
• Choose. Threshold that maximizes F1, unless business context prefers precision (fewer false-positive passes) or recall (fewer false-negative fails).

Question. Build the calibrator, sweep the threshold, and pick the operating point for a chatbot where a missed wrong answer is 3× more costly than a false-flagged correct answer.

Input.

Parameter	Value
Correct pairs	200 (from golden set)
Contradiction pairs	200 (LLM-generated, SME-reviewed)
Embedding model	text-embedding-3-large
Sweep	0.50 to 0.99 in 0.01 steps
Cost ratio (miss : false-flag)	3 : 1

Code.

# calibrate_threshold.py
import json
from pathlib import Path

import numpy as np
import pandas as pd

from cosine_scorer import embed_batch, BATCH


def cosine_pairs(a: list[str], b: list[str]) -> np.ndarray:
    va = np.vstack([embed_batch(a[i:i+BATCH]) for i in range(0, len(a), BATCH)])
    vb = np.vstack([embed_batch(b[i:i+BATCH]) for i in range(0, len(b), BATCH)])
    return np.sum(va * vb, axis=1)


def sweep(cos_pos: np.ndarray, cos_neg: np.ndarray, cost_miss: float = 3.0, cost_ff: float = 1.0) -> pd.DataFrame:
    rows = []
    for t in np.arange(0.50, 1.00, 0.01):
        tp = (cos_pos >= t).sum()
        fn = (cos_pos <  t).sum()
        fp = (cos_neg >= t).sum()
        tn = (cos_neg <  t).sum()
        precision = tp / (tp + fp) if (tp + fp) else 0.0
        recall    = tp / (tp + fn) if (tp + fn) else 0.0
        f1        = 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0
        # cost-weighted score: penalize false-flags (fn on positives) and missed-wrongs (fp on negatives)
        cost = cost_ff * fn + cost_miss * fp
        rows.append({
            "threshold": round(float(t), 2),
            "tp": int(tp), "fn": int(fn), "fp": int(fp), "tn": int(tn),
            "precision": round(precision, 3),
            "recall":    round(recall, 3),
            "f1":        round(f1, 3),
            "cost":      int(cost),
        })
    return pd.DataFrame(rows)


if __name__ == "__main__":
    pos_pairs = json.loads(Path("calibration/correct_pairs.json").read_text())    # 200 pairs
    neg_pairs = json.loads(Path("calibration/contradiction_pairs.json").read_text())  # 200 pairs

    cos_pos = cosine_pairs([p["expected"] for p in pos_pairs], [p["prediction"] for p in pos_pairs])
    cos_neg = cosine_pairs([p["expected"] for p in neg_pairs], [p["prediction"] for p in neg_pairs])

    df = sweep(cos_pos, cos_neg, cost_miss=3.0, cost_ff=1.0)
    df_best_f1  = df.loc[df["f1"].idxmax()]
    df_best_cost = df.loc[df["cost"].idxmin()]
    print("Best F1 threshold:", df_best_f1.to_dict())
    print("Best cost-weighted threshold:", df_best_cost.to_dict())
    print(df.to_markdown(index=False))

Step-by-step explanation.

1. Positive pairs are 200 rows where prediction == expected_output (perfect answers). Negative pairs are 200 rows where a strong LLM was prompted to contradict the expected answer, then an SME reviewed to ensure each is a genuine wrong-polarity flip. Both are embedded and cosine'd.
2. The sweep computes confusion-matrix entries at each threshold: true positives (correct passes), false negatives (correct-but-flagged), false positives (wrong-but-passed), true negatives (wrong-and-flagged). Precision, recall, and F1 fall out of the confusion matrix.
3. The cost function encodes the business preference: a missed wrong answer (FP on negatives) is 3× the pain of a false-flagged correct answer (FN on positives). The cost-optimal threshold is higher than the F1-optimal threshold because higher thresholds catch more wrongs at the cost of flagging more corrects.
4. Empirically on text-embedding-3-large: F1 peaks around 0.82 (threshold 0.86); cost-optimal at 3:1 miss weight lands around 0.89. Both numbers are model-specific and answer-length-specific; they should be re-derived on your data.
5. The chosen operating point (0.89) goes into eval-config.yaml as cosine.threshold: 0.89 with a comment linking to the calibration report and the calibration set's git hash. Any future embedding-model bump requires a fresh calibration.

Output.

threshold	tp	fn	fp	tn	precision	recall	f1	cost
0.80	195	5	42	158	0.823	0.975	0.893	131
0.85	190	10	22	178	0.896	0.950	0.922	76
0.86	187	13	15	185	0.926	0.935	0.930	58
0.89	175	25	8	192	0.956	0.875	0.914	49
0.90	168	32	5	195	0.971	0.840	0.901	47
0.92	148	52	2	198	0.987	0.740	0.846	58

Rule of thumb. Calibrate the cosine threshold against a labeled positive+negative set, sweep with a cost function that reflects your business's miss/false-flag ratio, and pin the chosen threshold in config with a link back to the calibration report. Never quote a threshold "from the paper" or "from a blog post" — it will not match your data.

Worked example — the paraphrase vs contradiction blind spot

Detailed explanation. Demonstrate the canonical cosine failure mode with a synthetic 20-row set: 10 paraphrases (semantically identical, superficially different) and 10 contradictions (semantically opposite, superficially similar). Show cosine scores for each and prove why an LLM-judge is required on top.

• Paraphrases. Same meaning, different words. Cosine will under-count these as false negatives.
• Contradictions. Opposite meaning, similar surface form. Cosine will over-count these as false positives.
• Judge job. Distinguish paraphrase from contradiction where cosine cannot.

Question. Build the demonstrator: score 10 paraphrase and 10 contradiction pairs with cosine, then with an LLM-as-judge, and compare their agreement + disagreement.

Input.

Pair type	Example expected	Example prediction	Ground truth
Paraphrase	Yes, you can cancel anytime.	You may cancel at any time.	correct
Paraphrase	The refund arrives in 3-5 business days.	Refunds land within 3 to 5 working days.	correct
Contradiction	Yes, you can cancel anytime.	No, cancellation is not permitted.	wrong
Contradiction	The refund arrives in 3-5 business days.	Refunds are not available.	wrong

Code.

# cosine_vs_judge_demo.py
import json
import numpy as np

from cosine_scorer import embed_batch
from llm_judge import judge_row       # defined in Section 4


pairs = [
    # Paraphrases (correct)
    ("Yes, you can cancel anytime.",                  "You may cancel at any time.",                          "correct"),
    ("The refund arrives in 3-5 business days.",      "Refunds land within 3 to 5 working days.",             "correct"),
    ("Please email [e-mail] for help.",    "For assistance, contact [e-mail].",         "correct"),
    ("Your order will ship tomorrow.",                "The shipment is scheduled for tomorrow.",              "correct"),
    ("We accept Visa and Mastercard.",                "Visa and Mastercard payments are accepted.",           "correct"),
    ("This feature is available on Pro plans.",       "Pro plans include this feature.",                      "correct"),
    ("You'll need admin access to change settings.",  "Admin permissions are required to modify settings.",   "correct"),
    ("The trial lasts 14 days.",                      "You get a 14-day trial period.",                       "correct"),
    ("Data exports are in CSV format.",               "Exported data is provided as CSV files.",              "correct"),
    ("The API rate limit is 1000 requests/min.",      "You can make up to 1000 API calls per minute.",        "correct"),
    # Contradictions (wrong)
    ("Yes, you can cancel anytime.",                  "No, cancellation is not permitted.",                   "wrong"),
    ("The refund arrives in 3-5 business days.",      "Refunds are not available.",                           "wrong"),
    ("Please email [e-mail] for help.",    "Do not email support; they don't respond.",            "wrong"),
    ("Your order will ship tomorrow.",                "Your order has been cancelled.",                       "wrong"),
    ("We accept Visa and Mastercard.",                "We do not accept credit cards.",                       "wrong"),
    ("This feature is available on Pro plans.",       "This feature is not available on any plan.",           "wrong"),
    ("You'll need admin access to change settings.",  "Anyone can change settings without any permission.",   "wrong"),
    ("The trial lasts 14 days.",                      "There is no trial period offered.",                    "wrong"),
    ("Data exports are in CSV format.",               "Data cannot be exported.",                             "wrong"),
    ("The API rate limit is 1000 requests/min.",      "There is no API access available.",                    "wrong"),
]


def demo():
    exp = embed_batch([p[0] for p in pairs])
    pred = embed_batch([p[1] for p in pairs])
    cos = np.sum(exp * pred, axis=1)

    rows = []
    for (e, p, truth), c in zip(pairs, cos):
        judge = judge_row(input_text="(hidden)", expected=e, prediction=p)   # returns 0 or 1
        rows.append({
            "type":     "paraphrase" if truth == "correct" else "contradiction",
            "cosine":   round(float(c), 3),
            "cos_pass": bool(c >= 0.85),
            "judge":    judge,
            "truth":    truth,
        })

    import pandas as pd
    df = pd.DataFrame(rows)
    print(df.to_markdown(index=False))

    cos_acc = ((df["cos_pass"] & (df["truth"] == "correct")) | (~df["cos_pass"] & (df["truth"] == "wrong"))).mean()
    jud_acc = ((df["judge"] == 1) & (df["truth"] == "correct") | (df["judge"] == 0) & (df["truth"] == "wrong")).mean()
    print(f"\ncosine  agreement with truth: {cos_acc:.2%}")
    print(f"judge   agreement with truth: {jud_acc:.2%}")


demo()

Step-by-step explanation.

1. The 20 pairs are hand-designed: 10 paraphrases (semantically identical, different surface form) and 10 contradictions (opposite polarity, otherwise similar surface form). This is a stress test built to break a naïve cosine gate.
2. Cosine scores paraphrases in the 0.75–0.85 range (below the calibrated 0.85 threshold — false negatives on the correct class) and contradictions in the 0.80–0.90 range (some above the threshold — false positives on the wrong class). The threshold cannot separate the two populations cleanly.
3. The LLM-judge is called with the full (expected, prediction) pair plus a rubric. For paraphrases it recognizes the semantic equivalence and returns 1 (pass); for contradictions it recognizes the polarity flip and returns 0 (fail). Judge accuracy on this stress set is typically 95–100 percent.
4. The comparison table exposes the exact gap: cosine agreement with ground truth is around 55–65 percent; judge agreement is 95–100 percent. On real production data the gap narrows (real predictions are less adversarial), but the direction is always the same — judge on top of cosine.
5. The lesson is not "throw cosine away." It is "cosine as first-pass filter, judge as truth arbiter." Cosine catches structural hallucinations cheaply; the judge catches semantic errors expensively. Use both, and know which failure mode each catches.

Output.

type	cosine	cos_pass	judge	truth
paraphrase	0.816	False	1	correct
paraphrase	0.892	True	1	correct
paraphrase	0.751	False	1	correct
contradiction	0.828	False	0	wrong
contradiction	0.874	True	0	wrong
contradiction	0.845	False	0	wrong

Cosine agreement with truth: ~60 percent. Judge agreement with truth: ~100 percent.

Rule of thumb. Any metric that cannot distinguish "yes" from "no" on similar surface forms is a false-security metric. Cosine is a great cheap gate; the judge is the truth signal that keeps the metric honest.

Senior interview question on cosine-based scoring

A senior interviewer might ask: "Walk me through your cosine similarity scorer for a RAG chatbot. How do you pick the embedding model, calibrate the threshold, handle multilingual traffic, and pair it with a judge without doubling your eval cost?"

Solution Using pinned embedding + calibrated threshold + judge-on-miss escalation

# cosine_plus_judge.py — the reference cosine + judge orchestrator
import json
from pathlib import Path
from typing import Optional

import numpy as np
import pandas as pd

from cosine_scorer import embed_batch, EMBED_MODEL, BATCH
from llm_judge import judge_row

# Pinned in config; bump forces recalibration
EMBEDDING_VERSION = "text-embedding-3-large@2024-11-01"
THRESHOLD_BY_LANGUAGE = {"en": 0.86, "es": 0.83, "fr": 0.83, "default": 0.82}


def score_row(expected_vec: np.ndarray, prediction_vec: np.ndarray) -> float:
    return float(np.dot(expected_vec, prediction_vec))


def threshold_for(language: str) -> float:
    return THRESHOLD_BY_LANGUAGE.get(language, THRESHOLD_BY_LANGUAGE["default"])


def score_batch(rows: list[dict], predictions: dict[str, str]) -> pd.DataFrame:
    exp_vecs = np.vstack([
        embed_batch([r["expected_output"] for r in rows[i:i+BATCH]])
        for i in range(0, len(rows), BATCH)
    ])
    pred_vecs = np.vstack([
        embed_batch([predictions[r["row_id"]] for r in rows[i:i+BATCH]])
        for i in range(0, len(rows), BATCH)
    ])

    out = []
    for r, ev, pv in zip(rows, exp_vecs, pred_vecs):
        lang = r["metadata"].get("language", "en")
        c = score_row(ev, pv)
        thr = threshold_for(lang)
        cos_pass = c >= thr
        # Judge-on-miss: escalate only when cosine says "wrong"
        judge = judge_row(
            input_text=r["input"],
            expected=r["expected_output"],
            prediction=predictions[r["row_id"]],
            rubric=["correctness"],
        ) if not cos_pass else None
        out.append({
            "row_id":         r["row_id"],
            "topic":          r["metadata"]["topic"],
            "language":       lang,
            "cosine":         round(c, 4),
            "cos_threshold":  thr,
            "cos_pass":       cos_pass,
            "judge_score":    judge,
            "final_pass":     cos_pass or (judge is not None and judge >= 0.7),
            "embedding":      EMBEDDING_VERSION,
        })
    return pd.DataFrame(out)


if __name__ == "__main__":
    rows = [json.loads(l) for l in Path("golden/support_v3.jsonl").read_text().splitlines()]
    predictions = json.loads(Path("runs/run_2026-07-04.json").read_text())
    df = score_batch(rows, predictions)
    print(df.groupby(["topic", "language"]).agg(
        n=("row_id", "count"),
        mean_cosine=("cosine", "mean"),
        cos_pass_rate=("cos_pass", "mean"),
        final_pass_rate=("final_pass", "mean"),
    ).round(3).to_markdown())

Step-by-step trace.

Step	Input	Output
Pin embedding	text-embedding-3-large@2024-11-01	version stamp on every row
Embed expected + prediction	300 rows	600 3072-dim vectors
Compute cosine	dot(ev, pv) after L2-norm	300 floats in [0, 1]
Threshold by language	en=0.86, es/fr=0.83	300 bool pass flags
Escalate on miss	~40 rows below threshold	40 judge calls
Aggregate final pass	cos_pass OR judge≥0.7	300 final bool

After the rollout, cosine handles ~87 percent of rows for free (below-threshold rows escalate to the judge); judge is called on only the ~13 percent that cosine flagged. Total judge cost drops by ~7× vs judging every row, while the final metric is judge-quality on the disputed subset.

Output:

Layer	Cost per full run	Detects paraphrase	Detects contradiction	Detects structural
Cosine only	$0.02	partial	no	yes
Judge only	$2.00	yes	yes	partial
Cosine + judge-on-miss	$0.30	yes (via judge)	yes (via judge)	yes (cosine)

Why this works — concept by concept:

• Pinned embedding version — text-embedding-3-large@2024-11-01 is written into every score row so the metric is comparable across time. Bumping the embedding model is a first-class change that requires re-calibrating the threshold; the version stamp makes it impossible to forget.
• Per-language thresholds — cross-lingual embeddings noisy differently per language pair; carrying a single "0.85" threshold across en/es/fr silently penalizes non-English rows. Per-language calibration is a one-hour exercise per language, paid once.
• Judge-on-miss escalation — cosine handles the majority of rows for essentially free; the judge is called only when cosine says "wrong." This preserves judge-quality decisions on the disputed subset while cutting judge cost by roughly 7× on a well-performing model.
• final_pass logic — a row passes if cosine says yes OR (cosine said no AND judge says ≥0.7). The judge acts as an appeal for cosine's false negatives on paraphrases; the OR-logic ensures paraphrases don't get punished, while contradictions still fail because the judge does not overturn cosine's passes.
• Cost — per-run: ~$0.02 embeddings + ~$0.30 judge on the escalated 13 percent = ~$0.32. The all-judge alternative is ~$2.00 (6× more expensive) with only marginal quality gain on rows cosine already handled correctly. Judge-on-miss is the price-performance sweet spot.

Data analysis
Topic — data-analysis
Data-analysis problems on similarity and threshold calibration

Practice →

Optimization Topic — optimization Optimization problems on scorer cost and coverage

Practice →

---

4. LLM-as-judge — GPT-4-class rubric scorer

The judge is a rubric + a strong model + a structured JSON schema — anything less is vibes

The mental model in one line: llm as judge is the pattern where a strong model (GPT-4-class or better) scores the production model's prediction against the golden expected answer using an explicit rubric, returning a structured JSON with a numeric score plus a rationale — and its power comes entirely from the rubric quality, the model choice, and the output schema, not from any magic in "the LLM said so". A judge without a rubric is vibes; a rubric without a strong judge is a rubric being ignored; a judge with a rubric returning free-form prose is unusable as a metric. Engineer all three.

The four axes of a serious judge.

• Rubric. An explicit list of criteria with clear pass/fail language. Typical rubrics carry 2–4 criteria (correctness, completeness, style, safety); more than 4 dilutes the judge's attention; fewer than 2 collapses the metric to a single dimension.
• Model. GPT-4o, Claude Opus, or equivalent. Judge quality tracks base-model reasoning quality; using the same model as production for judgement risks correlated failure modes.
• Output schema. Structured JSON with typed fields (score: float, rationale: string, optional breakdown per criterion). Function-calling or JSON-mode is mandatory — free-form text is unparseable.
• Cost + sampling. Judge cost per row × golden-set size × runs per week is the finance-visible number. Sample intelligently (per-PR 20 percent sample + nightly full-set) to keep the bill sane.

The rubric — write it like an SLA.

• Explicit criteria. correctness (is the factual content right?), completeness (are the required facts included?), style (is the tone appropriate?). Each criterion gets its own explicit language.
• Scored on a scale. 0.0–1.0 or 1–5. Continuous scales let you aggregate as means; discrete scales are easier for judges to calibrate but coarser downstream.
• Grounded in the expected answer. The judge always has access to the expected answer as ground truth; without it, the judge is just re-answering the question rather than scoring.
• Failure modes named. The rubric explicitly names the failure modes to look for: "a wrong URL", "a wrong dollar amount", "an empathy mismatch". This forces the judge to check specifics rather than hand-wave.

The output schema — never take free-form text.

• Use JSON mode / function calling. OpenAI's response_format={"type": "json_object"}, Anthropic's tool use, or Gemini's structured output. The judge returns typed fields; parsing is trivial.
• Type every field. score: float in [0, 1], rationale: string, breakdown: {correctness: float, completeness: float, style: float}. The schema is verified with pydantic on receive.
• Bounded output length. Cap rationale at 500 tokens. Longer rationales cost more without improving the score signal.
• Idempotence. Set temperature=0 for reproducible judgements. Non-zero temperature turns the judge into a random noise generator.

The judge model — do not use the production model.

• Same-family risk. Using GPT-4o-mini as judge for a GPT-4o-mini production model risks correlated failure modes — the judge fails to notice the same errors the production model made.
• Recommendation. Judge with a stronger or different-family model. GPT-4o judges GPT-4o-mini production; Claude Opus judges GPT-4o production; Gemini Ultra judges Claude Sonnet.
• Cost trade-off. A stronger judge costs more per row. Balance with sample-on-PR + full-nightly (cheap judge on the sample, stronger judge on nightly calibration).
• Calibrate cross-judge agreement. Run 100 golden rows through two judges (GPT-4o and Claude Opus). Compute agreement (Cohen's κ or Pearson r on scores). Above 0.85 means either judge is fine; below 0.7 means the rubric needs tightening.

Failure modes of the judge itself.

• Judge bias. Judges have systematic preferences (length, formality, structure). A verbose but wrong answer may score higher than a concise correct one. Counter with explicit anti-bias language in the rubric.
• Prompt sensitivity. Small phrasing changes in the rubric shift scores. Version-hash the rubric and treat rubric edits like migrations.
• Position bias. Some judges systematically favor the first option in a comparison; randomize the order in pairwise comparison setups.
• Rubric drift. Over time the SME's mental rubric drifts from the written rubric. Refresh the rubric every quarter and re-calibrate.

What senior interviewers listen for.

• Do you name the rubric as a first-class artefact with version control? — senior signal.
• Do you insist on structured JSON output rather than free-form? — required answer.
• Do you use a stronger or different-family judge than production? — senior signal.
• Do you talk about judge bias and prompt sensitivity as engineering problems, not model flaws? — senior signal.

Worked example — a 3-criterion rubric + JSON schema for a support chatbot

Detailed explanation. Design the judge for the customer-support chatbot. Rubric: correctness, completeness, style. Model: GPT-4o. Output: JSON with per-criterion scores plus an overall + rationale. Show the exact prompt, the pydantic schema, and a worked scoring on a representative row.

• Judge model. GPT-4o (temperature=0).
• Rubric criteria. correctness (0.0-1.0), completeness (0.0-1.0), style (0.0-1.0).
• Overall score. Weighted mean: 0.5*correctness + 0.3*completeness + 0.2*style.
• Output. JSON via response_format={"type": "json_object"}.

Question. Write the judge prompt, the pydantic schema, and the client code. Score a representative row and interpret.

Input.

Field	Value
Input	"How do I cancel my subscription?"
Expected	"You can cancel anytime by opening Settings → Billing → Cancel. The cancellation takes effect at the end of your current billing period."
Prediction	"You may cancel any time from the billing settings; the change is effective at the end of the current cycle."

Code.

# llm_judge.py
import json
from typing import Literal, Optional

from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError

client = OpenAI()

RUBRIC_HASH = "c93f11"     # bumps whenever the prompt changes


class JudgeBreakdown(BaseModel):
    correctness: float = Field(..., ge=0.0, le=1.0)
    completeness: float = Field(..., ge=0.0, le=1.0)
    style: float = Field(..., ge=0.0, le=1.0)


class JudgeResult(BaseModel):
    score: float = Field(..., ge=0.0, le=1.0)
    rationale: str
    breakdown: JudgeBreakdown


SYSTEM_PROMPT = """
You are an expert evaluator for a customer support chatbot. You score the
`prediction` against the `expected` answer on THREE criteria:

- correctness (0.0-1.0): factually accurate; no wrong dates, URLs, entities,
  or polarity flips. If the prediction contradicts the expected answer in
  any material way, correctness is 0.0.
- completeness (0.0-1.0): includes the essential facts from the expected
  answer. Missing a required step, disclaimer, or call-to-action lowers
  completeness proportionally.
- style (0.0-1.0): tone matches the expected style (empathetic where the
  expected is empathetic, formal where the expected is formal); grammar
  and phrasing are natural.

You are BLIND to answer length: a shorter correct answer scores as high as
a longer correct answer. You do NOT reward verbosity.

Return a JSON object with the schema:
{
  "score": <weighted mean: 0.5 * correctness + 0.3 * completeness + 0.2 * style>,
  "rationale": "<= 500 tokens explaining the criterion scores",
  "breakdown": {"correctness": ..., "completeness": ..., "style": ...}
}

Do not include any other fields; do not include markdown; return raw JSON.
""".strip()


def judge_row(
    input_text: str,
    expected: str,
    prediction: str,
    rubric: Optional[list[str]] = None,
    model: str = "gpt-4o",
) -> JudgeResult:
    user_prompt = (
        f"INPUT:\n{input_text}\n\n"
        f"EXPECTED:\n{expected}\n\n"
        f"PREDICTION:\n{prediction}\n"
    )
    resp = client.chat.completions.create(
        model=model,
        temperature=0.0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user",   "content": user_prompt},
        ],
    )
    raw = resp.choices[0].message.content
    try:
        return JudgeResult.model_validate_json(raw)
    except ValidationError as e:
        raise ValueError(f"judge returned invalid JSON: {raw!r}") from e


if __name__ == "__main__":
    result = judge_row(
        input_text="How do I cancel my subscription?",
        expected=(
            "You can cancel anytime by opening Settings → Billing → Cancel. "
            "The cancellation takes effect at the end of your current billing period."
        ),
        prediction=(
            "You may cancel any time from the billing settings; the change "
            "is effective at the end of the current cycle."
        ),
    )
    print(json.dumps(result.model_dump(), indent=2))

Step-by-step explanation.

1. The system prompt spells out the three criteria with explicit language, calls out the specific failure modes (wrong URLs, polarity flips, missing steps), and tells the judge not to reward verbosity. This is the rubric-as-SLA discipline in one prompt.
2. The output schema is a pydantic model: score in [0, 1], rationale string, breakdown per-criterion. response_format={"type": "json_object"} forces JSON; temperature=0.0 makes the judgement deterministic; RUBRIC_HASH versions the prompt.
3. On the sample row, the prediction is a paraphrase of the expected: same steps, same effect, slightly different wording. The judge scores correctness at 1.0 (all facts match), completeness at 0.9 (mentions cancel + end-of-cycle, misses the explicit menu path), style at 0.85 (informal but appropriate). Overall = 0.5×1.0 + 0.3×0.9 + 0.2×0.85 = 0.94.
4. The rationale is included for auditability. When a PR author sees a per-row miss, they can read the rationale and understand why — "the prediction says 'billing settings' but the expected says 'Settings → Billing → Cancel'; the menu path is less precise" — turning the metric from a black box into a diagnostic tool.
5. The client caches on (input_hash, expected_hash, prediction_hash, rubric_hash, model). Deterministic judgement means identical inputs always return identical scores; cache hits are free.

Output.

{
  "score": 0.94,
  "rationale": "The prediction correctly identifies the cancellation location (billing settings) and confirms the effective date (end of current cycle), matching the factual content of the expected. It omits the specific menu path (Settings → Billing → Cancel) which lowers completeness slightly. Style is informal but appropriate for a support context; the tone is helpful.",
  "breakdown": {
    "correctness": 1.0,
    "completeness": 0.9,
    "style": 0.85
  }
}

Rule of thumb. Every judge call returns structured JSON validated by pydantic; every prompt is version-hashed; every judgement is deterministic (temperature=0). Free-form judge outputs are unusable as a metric — engineer for structure.

Worked example — cross-judge agreement calibration

Detailed explanation. Before trusting a judge, measure it against another judge. Take 100 golden-set rows plus their predictions, score with two independent judges (GPT-4o and Claude Opus), and compute agreement. If the two disagree systematically, the rubric needs work.

• Judge A. GPT-4o with the 3-criterion rubric.
• Judge B. Claude Opus with the same rubric.
• Metrics. Pearson correlation on continuous scores; Cohen's κ on discretized pass/fail at 0.7.
• Decision rule. r > 0.85 → either judge is fine; r < 0.7 → tighten the rubric or narrow the criteria.

Question. Build the cross-judge calibrator, run it on 100 rows, and interpret. What do you do if r = 0.62?

Input.

Parameter	Value
Rows	100 (from golden set)
Judge A	GPT-4o
Judge B	Claude Opus 4
Prompt hash	c93f11 (same for both)
Pass threshold	score ≥ 0.7

Code.

# cross_judge_agreement.py
import json
from pathlib import Path

import numpy as np
import pandas as pd
from scipy.stats import pearsonr, spearmanr

from llm_judge import judge_row


def score_all(rows: list[dict], predictions: dict[str, str], model: str) -> pd.Series:
    scores = []
    for r in rows:
        j = judge_row(
            input_text=r["input"],
            expected=r["expected_output"],
            prediction=predictions[r["row_id"]],
            model=model,
        )
        scores.append({"row_id": r["row_id"], "score": j.score})
    return pd.DataFrame(scores).set_index("row_id")["score"]


def cohens_kappa(a: pd.Series, b: pd.Series, threshold: float = 0.7) -> float:
    a_pass = (a >= threshold).astype(int)
    b_pass = (b >= threshold).astype(int)
    # 2x2 confusion
    n = len(a)
    tt = int(((a_pass == 1) & (b_pass == 1)).sum())
    ff = int(((a_pass == 0) & (b_pass == 0)).sum())
    tf = int(((a_pass == 1) & (b_pass == 0)).sum())
    ft = int(((a_pass == 0) & (b_pass == 1)).sum())
    p_obs = (tt + ff) / n
    p_a_pos = (tt + tf) / n
    p_b_pos = (tt + ft) / n
    p_exp = p_a_pos * p_b_pos + (1 - p_a_pos) * (1 - p_b_pos)
    return (p_obs - p_exp) / (1 - p_exp) if p_exp < 1 else 1.0


if __name__ == "__main__":
    rows = [json.loads(l) for l in Path("golden/support_v3.jsonl").read_text().splitlines()][:100]
    predictions = json.loads(Path("runs/run_2026-07-04.json").read_text())

    a = score_all(rows, predictions, model="gpt-4o")
    b = score_all(rows, predictions, model="claude-opus-4")

    r_pearson, p1 = pearsonr(a, b)
    r_spearman, p2 = spearmanr(a, b)
    kappa = cohens_kappa(a, b, threshold=0.7)

    print(f"Pearson r = {r_pearson:.3f}  (p = {p1:.4f})")
    print(f"Spearman r = {r_spearman:.3f}")
    print(f"Cohen's κ  = {kappa:.3f}")

    if r_pearson >= 0.85:
        print("PASS: either judge is fine; pick the cheaper one for PR sample.")
    elif r_pearson >= 0.70:
        print("MARGINAL: use the more expensive judge; re-examine rubric criteria.")
    else:
        print("FAIL: rubric ambiguous. Tighten criteria; re-write style axis; re-run.")

Step-by-step explanation.

1. Score the same 100 rows through both judges with the identical rubric. Both judges are temperature=0, so within-judge reruns are stable; the delta between judges is the "rubric ambiguity plus model bias" signal.
2. Compute Pearson r (linear correlation of continuous scores) and Spearman r (rank correlation, robust to scale differences). Compute Cohen's κ on the discretized pass/fail at 0.7 — this captures how well the judges agree on the decision level.
3. Interpret: r ≥ 0.85 means the judges agree strongly; either can be used, so pick the cheaper for high-frequency runs. r in [0.7, 0.85) is marginal — the expensive judge should stay in the gate, and the rubric deserves another look. r < 0.7 is a rubric problem: the criteria are ambiguous enough that reasonable judges disagree.
4. If r = 0.62, look at the per-row scores where the judges disagree. Common causes: the style axis is subjective (empathy is culture-dependent), the completeness axis is underspecified (which facts are required?), or the rubric conflates axes that need to be separated. Rewrite the ambiguous criteria and re-run.
5. Once the rubric is stable and r > 0.85, freeze the rubric hash and pin the cheaper judge for the PR sample. The expensive judge stays on the nightly calibration run as an anchor — if the cheap judge starts drifting from the expensive one, the calibration diff flags it.

Output.

Pearson r = 0.891  (p = 0.0000)
Spearman r = 0.867
Cohen's κ  = 0.782
PASS: either judge is fine; pick the cheaper one for PR sample.

Rule of thumb. Cross-judge calibration is the acceptance test for a rubric. A rubric that two strong judges score inconsistently is a rubric your PR author will not trust. Tighten first, then judge.

Worked example — the length-bias trap and how to instrument against it

Detailed explanation. LLM judges systematically prefer longer answers even when the shorter answer is factually complete. Instrument the judge output with per-row prediction length, plot score-vs-length, detect bias, and inject an anti-bias instruction into the rubric.

• Signal. Positive correlation between len(prediction) and judge_score, controlling for correctness.
• Detection. Regress judge_score on len_pred while holding expected_correctness fixed; a positive slope is length bias.
• Fix. Add "you are blind to length" language + a paired-example demonstration.

Question. Build the length-bias detector against a 300-row scored set, quantify the bias slope, and update the rubric to counteract it.

Input.

Parameter	Value
Scored rows	300
Judge scores	judge_score column from prior run
Prediction length	len(prediction) in chars
Baseline (correct-answer) length	180 chars mean

Code.

# length_bias_detector.py
import json
from pathlib import Path

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression


def detect_length_bias(scored_df: pd.DataFrame) -> dict:
    """
    scored_df must have: judge_score (float 0-1), prediction (str), correctness (float 0-1).
    Regresses judge_score on len_pred while conditioning on correctness.
    """
    df = scored_df.copy()
    df["len_pred"] = df["prediction"].str.len()
    df["len_pred_z"] = (df["len_pred"] - df["len_pred"].mean()) / df["len_pred"].std()

    # Partial regression: judge_score ~ len_pred_z + correctness
    X = df[["len_pred_z", "correctness"]].values
    y = df["judge_score"].values
    model = LinearRegression().fit(X, y)

    slope_len = float(model.coef_[0])          # per-standard-deviation of length
    slope_correct = float(model.coef_[1])
    r_partial = float(np.corrcoef(y - model.intercept_ - slope_correct * X[:, 1], X[:, 0])[0, 1])
    return {
        "n": len(df),
        "slope_length_per_sd": round(slope_len, 4),
        "slope_correctness": round(slope_correct, 4),
        "partial_r_length": round(r_partial, 4),
        "interpretation": (
            "BIAS: judge rewards length" if slope_len > 0.02
            else "OK: no meaningful length bias"
        ),
    }


if __name__ == "__main__":
    df = pd.read_parquet("runs/run_2026-07-04_scored.parquet")
    print(detect_length_bias(df))


# --- Anti-bias rubric update (append to SYSTEM_PROMPT) ---
ANTI_BIAS_ADDENDUM = """
CRITICAL: You must be blind to answer length. Two answers that convey the same
factual content should receive the same score, regardless of length. A 40-word
correct answer scores identically to a 200-word correct answer.

Consider this pair:
- Prediction A (short): "Cancel from Settings → Billing → Cancel; effective at cycle end."
- Prediction B (long): "You can absolutely cancel your subscription at any time! To do so, please navigate to Settings, then Billing, then Cancel. The cancellation will take effect at the end of your current billing cycle. If you have any questions, feel free to reach out to our support team..."

Both should score identically (correctness=1.0, completeness=1.0), because both
contain the same factual information. Prediction B's length adds no signal.
""".strip()

Step-by-step explanation.

1. Regress judge_score on standardized len_pred while controlling for correctness (which is itself judge-labelled — a limitation the harness acknowledges). A positive slope on len_pred_z while correctness is fixed is length bias by construction.
2. On the reference set, the slope typically lands in the range 0.03–0.08 for GPT-4-class judges without an explicit anti-bias instruction — meaning a 1-standard-deviation increase in length (say, +200 chars) raises the judge score by 3–8 points on the 0-100 scale. Significant, and directionally wrong.
3. The interpretation triggers a rubric update: append the anti-bias language plus a paired-example demonstration. The demonstration is what actually moves the needle — instructing the judge to "consider two predictions of the same content" is far more effective than telling it to "ignore length."
4. After the rubric update, re-run the detector on the same 300 rows. Slope typically drops to 0.005–0.015 — small residual bias, but small enough that the metric is not systematically favoring verbose answers.
5. The detector runs on every nightly full-set run. If the slope creeps back up (rubric drift, model update on the judge side), the alert fires and the team re-examines the rubric before the bias corrupts weeks of PR gating.

Output.

Rubric version	Slope (length)	Slope (correctness)	Interpretation
v1.0 (no anti-bias)	0.062	0.71	BIAS: judge rewards length
v1.1 (+ anti-bias)	0.008	0.79	OK: no meaningful length bias

Rule of thumb. Judge bias detection is a nightly check, not a one-time calibration. Rubric drift is real; new judge model versions ship weekly; the harness that watches the judge is the harness the team trusts. Never assume the judge stays honest without an instrument.

Senior interview question on the judge design

A senior interviewer might ask: "Design an LLM-as-judge for a code-explanation assistant where correctness must be exact, style is 'friendly and precise,' and the judge must be robust against style drift when the production model changes. Walk me through the rubric, model choice, output schema, calibration, and the anti-bias instrumentation."

Solution Using a versioned rubric + stronger-judge + cross-model calibration + length-bias check

# code_assistant_judge.py — the reference judge for a code-explanation task
import json
from typing import Literal

from openai import OpenAI
from pydantic import BaseModel, Field

client = OpenAI()

RUBRIC_HASH = "cx-42a9"       # bump on any prompt edit
JUDGE_MODEL = "gpt-4o"         # production is Claude Sonnet; judge is different family


class CodeExplainBreakdown(BaseModel):
    factual_correctness: float = Field(..., ge=0.0, le=1.0)
    code_accuracy: float = Field(..., ge=0.0, le=1.0)  # cited APIs / syntax right
    style: float = Field(..., ge=0.0, le=1.0)


class CodeExplainResult(BaseModel):
    score: float = Field(..., ge=0.0, le=1.0)
    rationale: str
    breakdown: CodeExplainBreakdown


SYSTEM = """
You evaluate a code-explanation assistant. You compare a prediction to a
golden expected answer on THREE criteria:

1. factual_correctness (0.0-1.0): Does the explanation match the expected
   behavior? Wrong function names, wrong parameters, wrong return types, or
   wrong Big-O = 0.0. Partial matches score in between.

2. code_accuracy (0.0-1.0): Are the code snippets in the prediction runnable
   and match the expected API? Wrong imports, wrong syntax, or wrong API
   surface = 0.0. Minor style issues that do not affect runnability = 0.9-1.0.

3. style (0.0-1.0): Is the tone "friendly and precise"? Precise means using
   the exact technical term (say "generator" not "iterator" when generator
   is correct). Friendly means direct, non-condescending, no filler. Verbosity
   is NOT a positive signal.

CRITICAL: You are blind to answer length. Two explanations of the same content
score identically. A 30-word correct answer = a 200-word correct answer.

Return raw JSON:
{
  "score": <0.5*factual + 0.3*code + 0.2*style>,
  "rationale": "<= 400 tokens",
  "breakdown": {"factual_correctness": ..., "code_accuracy": ..., "style": ...}
}
""".strip()


def judge_code_row(input_text: str, expected: str, prediction: str) -> CodeExplainResult:
    user = f"INPUT:\n{input_text}\n\nEXPECTED:\n{expected}\n\nPREDICTION:\n{prediction}\n"
    resp = client.chat.completions.create(
        model=JUDGE_MODEL,
        temperature=0.0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user",   "content": user},
        ],
    )
    return CodeExplainResult.model_validate_json(resp.choices[0].message.content)


# --- Nightly instrumentation harness ---
def nightly_calibration(rows, predictions):
    a = [judge_code_row(r["input"], r["expected_output"], predictions[r["row_id"]]).score for r in rows]
    b = [judge_code_row_with_alt_judge(r["input"], r["expected_output"], predictions[r["row_id"]]).score for r in rows]
    from scipy.stats import pearsonr
    return pearsonr(a, b)[0]

Step-by-step trace.

Step	Choice	Rationale
Production model	Claude Sonnet	Team's product choice
Judge model	GPT-4o	Different family; avoids correlated failures
Rubric criteria	factual_correctness, code_accuracy, style	Three axes cover this task
Weights	0.5 / 0.3 / 0.2	Factual dominates; code accuracy matters; style is tiebreaker
Output schema	pydantic-validated JSON	Structured, typed, cacheable
Rubric hash	cx-42a9	Frozen on merge; bumps invalidate cache
Length bias check	nightly	Detects rubric drift
Cross-judge calibration	weekly	Pin agreement r > 0.85

After the rollout, the judge scores every prediction on three axes with a stable weighted overall, produces per-criterion breakdowns for forensics, and is version-controlled at the rubric level. The nightly length-bias check catches drift; the weekly cross-judge calibration keeps the judge honest against a different-family alternate.

Output:

Instrument	Frequency	Alert threshold	Owner
Judge score (per PR)	on merge	<baseline - 5%	PR author
Rubric hash	on rubric edit	new hash	Reviewer
Length-bias slope	nightly	> 0.02	ML infra on-call
Cross-judge Pearson r	weekly	< 0.85	ML infra on-call
Judge cost/PR	on run	> $3	Finance ops

Why this works — concept by concept:

• Different-family judge — Claude Sonnet in production, GPT-4o as judge. Correlated failure modes are the biggest hidden risk with same-family judging; switching families breaks the correlation and gives you an independent view of quality. The 5-10 percent cost premium is worth the independence.
• Weighted rubric with typed schema — 0.5 factual + 0.3 code + 0.2 style is task-specific; the weights encode "what matters most for this product." Pydantic validation on receive means malformed judge output surfaces immediately as an error instead of silently poisoning the metric.
• Anti-bias instruction with paired example — telling the judge "you are blind to length" is worth about 3× more with an explicit paired example than without. LLMs are pattern-matchers; showing the pattern beats declaring it.
• Nightly length-bias check — the judge is an artefact under active development. Model updates, prompt drift, and rubric edits all reintroduce bias. The nightly detector is the judge's CI; without it, the harness measuring quality drifts silently.
• Cost — GPT-4o judge at ~$0.006/row × 300 golden rows = $1.80 per PR. Weekly cross-judge calibration with Claude Opus adds ~$3 per week. Nightly length-bias check is a regression on existing score data — $0. Total judge cost < $100/month for a mature setup.

Data analysis
Topic — data-analysis
Data-analysis problems on scorer calibration and bias

Practice →

Optimization Topic — optimization Optimization problems on judge cost and sampling

Practice →

---

5. CI integration + production monitoring

Evaluation is a build gate and a production sensor — one signal, two schedulers

The mental model in one line: a mature llm ci stack runs the same eval harness in two places — on every PR against the golden set (as a build gate that blocks regressions) and on a sample of live production traffic (as a drift detector that alerts on quality decay) — and the frameworks that glue the pieces together are Ragas for RAG-specific metrics, DeepEval for pytest-native regression tests, and promptfoo for lightweight prompt A/B. The engineering discipline is exactly the pattern you already run for every other pipeline: same code path, two schedulers, two alerting surfaces.

The four axes of CI-plus-production.

• PR gate. The harness runs on every PR touching prompts, retrieval, or model config. It scores the change against the golden set, compares to the previous baseline, and blocks the merge if aggregate score drops more than the configured threshold (typically 5 percent).
• Nightly full-set. A cron job runs the full golden set with the stronger judge every 24 hours. Feeds the baseline the PR gate compares to; catches slow drift that PR-level sampling cannot see.
• Production sampler. A sidecar samples 0.5–2 percent of live production conversations, routes them through the same harness (cheap judge + cosine), and writes scores to a rolling metrics table. This is your "did the metric decay in prod even though CI is green?" sensor.
• Alerting. Regression alerts (5 percent aggregate drop) page on-call; production drift alerts (24-hour rolling mean drops 5 percent vs last-week baseline) page on-call; per-class alerts (any class drops 10 percent) page on-call. All feed the same runbook.

The PR gate — the anatomy.

• Trigger. GitHub Actions on pull_request with path filter (prompts/**, retrieval/**, model_config/**).
• Sample. 20 percent of the golden set (typically 300-400 rows) for a 30-second wall clock.
• Score. Cosine + judge-on-miss for cost efficiency; per-row scores land in a metrics table with the PR SHA and golden-set SHA.
• Compare. Load the previous main-branch baseline (last-merged run) and compute the delta. Aggregate delta ≥ -5 percent = pass; less than that = block.
• Comment. Post the per-class delta table into the PR comment. The author sees exactly which class regressed.

The nightly full-set — the anatomy.

• Trigger. Cron 02:00 UTC daily. Runs against main at HEAD.
• Scope. Full golden set (2000 rows) with the stronger judge (GPT-4o).
• Store. Metric row per (run_id, golden_set_sha, row_id, judge_score, cosine_score).
• Baseline update. Aggregate mean over the run becomes the "previous baseline" the next PR run compares to.
• Report. Slack post with per-class means + week-over-week deltas.

The production sampler — the anatomy.

• Sample. 1 percent of live conversations, uniformly random. Higher sample for lower-traffic tenants to keep the metric statistically meaningful.
• Ground truth. For sampled conversations, there is no expected answer. Score cosine against retrieved context and judge against the retrieved answer (RAG-style self-consistency), or use the LLM-judge with a "no reference" rubric ("was this answer useful given the question?").
• Alert. 24-hour rolling mean drops 5 percent vs the same day-of-week last week. Rolling-window baseline handles weekly seasonality.
• Cost. ~$50/week for a mid-size product; the cheapest reliable prod-quality sensor.

The frameworks — when to reach for each.

• Ragas. Purpose-built for RAG evaluation. Metrics: answer_relevancy, faithfulness (does the answer contradict the retrieved context?), context_precision, context_recall. Uses LLM-as-judge under the hood. Great starter kit if you're building RAG and don't want to write the judge from scratch.
• DeepEval. pytest-native LLM eval library. Write assert_test(TestCase(input=..., expected=..., metrics=[GEval(...)])) in your test files; run with pytest. Fits the "make it a pytest" instinct of most Python teams.
• promptfoo. CLI + YAML for prompt A/B. promptfoo eval runs matrix of prompts × models × inputs and produces an HTML report. Best for lightweight prompt tuning, not production monitoring.
• Roll your own. Once you have a mature golden set + judge + CI hook, the framework overhead is often not worth it. Ragas and DeepEval win on time-to-first-metric; a hand-rolled harness wins on flexibility once you know exactly what you want.

Common CI + production failure modes.

• Baseline drift. The "previous baseline" is the last-merged score, which can be a lucky high or an unlucky low. Fix: use a 7-day moving average as the baseline, not a single run.
• Flake tolerance. A one-time judge hiccup causes a false-fail PR. Fix: retry the harness once on transient judge errors; fail only on the second attempt.
• Silent judge upgrade. OpenAI ships a new GPT-4o checkpoint under the same name; scores shift 2 percent overnight. Fix: pin the judge model to a dated version and log the version on every run.
• Prod sampler skew. The 1 percent sample is over-representative of one tenant. Fix: stratified sample per tenant + per topic class.

What senior interviewers listen for.

• Do you frame CI + prod monitoring as "same harness, two schedulers" rather than as two separate systems? — senior signal.
• Do you volunteer rolling-baseline as the answer to flaky one-run baselines? — senior signal.
• Do you name Ragas / DeepEval / promptfoo and know when to reach for each? — senior signal.
• Do you distinguish PR gate (block regression) from production monitor (detect drift) as different problems with a shared implementation? — required answer.

Worked example — pytest-style regression harness with pass/fail gate

Detailed explanation. Build the PR-gate harness as a pytest suite. Each test case is a golden-set row; the metric is the judge score; the gate is aggregate mean vs baseline. The suite runs in the PR CI, blocks the merge on a 5 percent aggregate drop, and posts a per-class delta table into the PR comment.

• Test shape. One pytest parameterization per row (or one aggregate test with a summary).
• Baseline. Loaded from a JSON artefact stored in the CI cache, updated on nightly.
• Gate. assert mean_score >= baseline - 0.05.
• Comment. Formatted markdown table via gh pr comment.

Question. Implement the pytest harness plus the CI YAML plus the PR comment formatter. Show the exact fail behaviour for a 6 percent aggregate drop.

Input.

Parameter	Value
Framework	pytest + custom fixtures
Golden set	400 rows (20% sample of 2000)
Judge	GPT-4o (temperature=0)
Baseline	last-nightly aggregate = 0.84
Gate threshold	-0.05 (block if mean < 0.79)
PR result (simulated)	mean = 0.78 (6 percent drop)

Code.

# tests/eval/test_regression.py
import json
import statistics
from pathlib import Path

import pytest

from harness.load_golden import load_golden
from harness.predict import predict_batch
from harness.llm_judge import judge_row
from harness.cosine import cosine_batch


GATE_DELTA = 0.05
BASELINE_PATH = Path(".ci/eval_baseline.json")


@pytest.fixture(scope="session")
def golden():
    return load_golden(Path("golden/support_v3.jsonl"), expected_hash="b83a11")


@pytest.fixture(scope="session")
def predictions(golden):
    inputs = [r["input"] for r in golden]
    return dict(zip((r["row_id"] for r in golden), predict_batch(inputs)))


@pytest.fixture(scope="session")
def baseline():
    return json.loads(BASELINE_PATH.read_text()) if BASELINE_PATH.exists() else {"mean": None}


def _sample(golden, k: int = 400, seed: int = 42):
    import random
    rng = random.Random(seed)
    return rng.sample(golden, k=min(k, len(golden)))


def test_aggregate_regression(golden, predictions, baseline, record_property):
    sample = _sample(golden, k=400)
    scores = []
    per_class = {}
    for row in sample:
        j = judge_row(
            input_text=row["input"],
            expected=row["expected_output"],
            prediction=predictions[row["row_id"]],
        ).score
        scores.append(j)
        topic = row["metadata"]["topic"]
        per_class.setdefault(topic, []).append(j)

    mean_score = statistics.mean(scores)
    delta = mean_score - (baseline["mean"] or mean_score)

    record_property("mean", mean_score)
    record_property("baseline", baseline["mean"])
    record_property("delta", delta)
    record_property("per_class", {t: statistics.mean(s) for t, s in per_class.items()})

    if baseline["mean"] is not None:
        assert delta >= -GATE_DELTA, (
            f"regression: mean {mean_score:.3f} < baseline {baseline['mean']:.3f} - {GATE_DELTA}"
        )

# .github/workflows/eval-gate.yml
name: eval-gate
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'retrieval/**'
      - 'model_config/**'
      - 'golden/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install -r requirements-eval.txt
      - name: restore baseline
        uses: actions/cache@v4
        with:
          path: .ci/eval_baseline.json
          key: eval-baseline-${{ github.base_ref }}
      - name: run eval suite
        run: pytest tests/eval/ -q --junitxml=eval-report.xml
      - name: post PR comment
        if: always()
        run: python harness/pr_comment.py eval-report.xml

# harness/pr_comment.py
import os
import sys
import xml.etree.ElementTree as ET
import subprocess


def load_report(path):
    root = ET.parse(path).getroot()
    props = {p.get("name"): p.get("value") for p in root.iter("property")}
    return props


def markdown(props):
    mean = float(props["mean"])
    base = props.get("baseline") or "n/a"
    delta = float(props["delta"])
    per_class = eval(props["per_class"])
    verdict = "PASS" if delta >= -0.05 else "BLOCK"
    tbl = "\n".join(f"| {t} | {s:.3f} |" for t, s in sorted(per_class.items()))
    return f"""### LLM Eval Gate: {verdict}

| metric | value |
|---|---|
| mean | {mean:.3f} |
| baseline | {base} |
| delta | {delta:+.3f} |
| threshold | -0.050 |

**Per-class means**

| topic | mean |
|---|---|
{tbl}

_Judge: gpt-4o • Rubric: c93f11 • Golden: b83a11_
"""


if __name__ == "__main__":
    props = load_report(sys.argv[1])
    body = markdown(props)
    pr = os.environ["GITHUB_REF"].split("/")[-2]
    subprocess.run(["gh", "pr", "comment", pr, "--body", body], check=True)

Step-by-step explanation.

1. The pytest suite loads the golden set, samples 400 rows deterministically (seed=42), scores each with the judge, aggregates the mean, and asserts delta >= -0.05 against the baseline. The record_property calls emit per-run metadata into the JUnit XML for the PR comment step to read.
2. The CI workflow triggers only on PRs touching the eval-relevant paths, saving budget when someone edits the README. The actions/cache step restores the eval_baseline.json from the base branch's cache; the nightly job is responsible for writing it there.
3. When the PR run reports mean=0.78 against baseline 0.84 (a 6 percent drop, exceeding the 5 percent gate), the assertion fails, pytest exits non-zero, GitHub Actions marks the eval-gate check as failed, and the merge is blocked by the branch-protection rule.
4. The post PR comment step runs on always() so a failed run still posts the diagnostic. The comment shows aggregate + per-class means; the author sees "shipping dropped from 0.86 to 0.72, product held at 0.88" and knows immediately where to look.
5. Rolling the baseline: the nightly full-set run writes a fresh eval_baseline.json (aggregated over 2000 rows with the stronger judge) into the CI cache. The next day's PR runs compare against the fresh baseline; the "moving target" concern is bounded because the nightly is deterministic (fixed golden set, fixed judge prompt, temperature 0).

Output.

=========================== test session starts ===========================
tests/eval/test_regression.py::test_aggregate_regression FAILED [100%]

================================ FAILURES ================================
______________________ test_aggregate_regression ______________________
> assert delta >= -GATE_DELTA
E    AssertionError: regression: mean 0.780 < baseline 0.840 - 0.05
E    assert -0.06 >= -0.05

------- PR comment posted -------
### LLM Eval Gate: BLOCK

| metric | value |
|---|---|
| mean | 0.780 |
| baseline | 0.840 |
| delta | -0.060 |
| threshold | -0.050 |

Per-class means:
| topic | mean |
|---|---|
| account | 0.815 |
| billing | 0.782 |
| other | 0.821 |
| product | 0.883 |
| shipping | 0.723  ← centre of regression |

Rule of thumb. The gate is a pytest — the same test runner your Python devs use for unit tests. Do not build a bespoke CI system; the moment the harness feels like an isolated framework, the team stops trusting it.

Worked example — production sampler with rolling-baseline alert

Detailed explanation. Build the production sampler as an Airflow task that runs hourly, pulls a 1 percent stratified sample from the last hour of live conversations, scores each via the same cosine + judge harness, and writes rows into a metrics warehouse table. A downstream alert compares the 24-hour rolling mean against the same-time-last-week baseline and pages on-call if the drop exceeds 5 percent.

• Sampler. Hourly cron; 1 percent stratified by tenant + topic.
• Scoring. Cosine against retrieved context + LLM-judge with "no reference" rubric.
• Storage. prod_eval_scores(conversation_id, ts, tenant, topic, cosine, judge_score, run_id).
• Alert. 24-hour rolling mean vs same day-of-week last week; > 5% drop = page.

Question. Build the Airflow DAG, the scoring task, and the alert query. Show how the alert fires on a synthetic 6 percent drop.

Input.

Parameter	Value
Sample rate	1% stratified
Cadence	hourly
Judge	gpt-4o-mini (cheap; sample is large)
Storage	Snowflake analytics.prod_eval_scores
Alert query	24h rolling mean vs same-day last week
Alert threshold	-5%

Code.

# dags/prod_eval_sampler.py
from datetime import datetime, timedelta

from airflow import DAG
from airflow.decorators import task
from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook

default_args = {"owner": "ml-infra", "retries": 2, "retry_delay": timedelta(minutes=5)}


with DAG(
    dag_id="prod_eval_sampler",
    schedule="@hourly",
    start_date=datetime(2026, 6, 1),
    catchup=False,
    default_args=default_args,
    tags=["llm-eval", "prod"],
) as dag:

    @task
    def sample_conversations(**ctx) -> list[dict]:
        window_end = ctx["logical_date"]
        window_start = window_end - timedelta(hours=1)
        hook = SnowflakeHook(snowflake_conn_id="analytics")
        return hook.get_records("""
            WITH candidates AS (
              SELECT conversation_id, tenant, topic, user_turn, model_answer,
                     retrieved_context, ts
              FROM   analytics.conversations
              WHERE  ts >= %s AND ts < %s
            ),
            stratified AS (
              SELECT *,
                     ROW_NUMBER() OVER (PARTITION BY tenant, topic ORDER BY random()) AS rn,
                     COUNT(*) OVER (PARTITION BY tenant, topic) AS n
              FROM   candidates
            )
            SELECT conversation_id, tenant, topic, user_turn, model_answer,
                   retrieved_context, ts
            FROM   stratified
            WHERE  rn <= GREATEST(1, ROUND(n * 0.01));
        """, parameters=(window_start, window_end))

    @task
    def score_and_write(rows: list[dict], **ctx):
        from harness.cosine import cosine_batch
        from harness.llm_judge import judge_no_reference

        scores = []
        for r in rows:
            cos = cosine_batch([r["retrieved_context"]], [r["model_answer"]])[0]
            j = judge_no_reference(
                question=r["user_turn"],
                context=r["retrieved_context"],
                answer=r["model_answer"],
            )
            scores.append({
                "conversation_id": r["conversation_id"],
                "ts": r["ts"],
                "tenant": r["tenant"],
                "topic": r["topic"],
                "cosine": float(cos),
                "judge_score": float(j.score),
                "run_id": ctx["run_id"],
            })

        hook = SnowflakeHook(snowflake_conn_id="analytics")
        hook.insert_rows(table="analytics.prod_eval_scores", rows=[list(s.values()) for s in scores])

    score_and_write(sample_conversations())

-- alerts/prod_drift.sql — 24h rolling vs same-day last week
WITH today AS (
  SELECT AVG(judge_score) AS mean_today
  FROM   analytics.prod_eval_scores
  WHERE  ts >= CURRENT_TIMESTAMP - INTERVAL '24 hours'
),
last_week AS (
  SELECT AVG(judge_score) AS mean_last_week
  FROM   analytics.prod_eval_scores
  WHERE  ts >= CURRENT_TIMESTAMP - INTERVAL '8 days'
    AND  ts <  CURRENT_TIMESTAMP - INTERVAL '7 days'
)
SELECT today.mean_today,
       last_week.mean_last_week,
       today.mean_today - last_week.mean_last_week AS delta,
       CASE
         WHEN today.mean_today - last_week.mean_last_week <= -0.05 THEN 'PAGE'
         WHEN today.mean_today - last_week.mean_last_week <= -0.03 THEN 'WARN'
         ELSE 'OK'
       END AS status
FROM   today, last_week;

Step-by-step explanation.

1. The sampler queries the last hour of live conversations, stratified 1 percent per (tenant, topic) pair to prevent one high-volume tenant from dominating the sample. The stratification uses ROW_NUMBER() + COUNT() per partition to compute the target sample size in-database.
2. Each sampled conversation carries the retrieved context + the model answer. Cosine is computed between context and answer (a proxy for faithfulness — is the answer grounded in what was retrieved?). Judge uses a "no reference" rubric that asks "given the question and the retrieved context, was the answer useful and faithful?"
3. Rows land in analytics.prod_eval_scores with run_id for lineage. Downstream dashboards slice by tenant, topic, and time; the alert query is a straight aggregation on the same table.
4. The alert query compares the 24-hour rolling mean against the same-day-last-week window. The last-week baseline handles weekly seasonality (Monday morning traffic ≠ Sunday evening). A -0.03 warn triggers a Slack post; a -0.05 page fires PagerDuty.
5. On the synthetic scenario (today mean=0.78, last week same window=0.85), delta=-0.07 → PAGE. The on-call receives a link to the drift dashboard, the per-tenant breakdown, and the runbook that walks them from "which tenant?" to "which topic?" to "which change landed in the last 7 days?"

Output.

mean_today	mean_last_week	delta	status
0.78	0.85	-0.07	PAGE

Rule of thumb. Production monitoring is the same harness as CI, just pointed at a different data source. Do not build a second scoring stack — build one, run it twice, alert on two schedules.

Worked example — using Ragas + DeepEval side by side

Detailed explanation. For teams that want a fast start without building the harness from scratch, Ragas and DeepEval are the two most-adopted 2026 frameworks. Ragas is RAG-first with pre-built metrics (faithfulness, answer_relevancy, context_precision, context_recall); DeepEval is pytest-native and general-purpose (any-metric, any-scorer). Use both: Ragas for the RAG-specific axes, DeepEval as the test-runner glue.

• Ragas metrics. faithfulness (does the answer contradict the context?), answer_relevancy (does it address the question?), context_precision (are retrieved chunks relevant?), context_recall (were the necessary chunks retrieved?).
• DeepEval role. Wraps Ragas metrics inside pytest test cases; runs on PR; posts pass/fail.
• When to reach for each. Ragas alone for a quick RAG dashboard; DeepEval alone for non-RAG tasks; both together for pytest-native RAG gating.

Question. Wire Ragas metrics into a DeepEval test suite that runs on PR. Show the exact test file, the config, and the failure output on a faithfulness drop.

Input.

Parameter	Value
Task	RAG chatbot
Ragas metrics	faithfulness, answer_relevancy
DeepEval integration	GEval wrapper
PR gate	faithfulness >= 0.85, answer_relevancy >= 0.80

Code.

# tests/eval/test_rag_ragas.py
import json
from pathlib import Path

import pytest
from deepeval import assert_test
from deepeval.metrics import RagasMetric
from deepeval.test_case import LLMTestCase


GOLDEN = json.loads(Path("golden/rag_support_v3.jsonl").read_text().splitlines()[0])


faithfulness_metric = RagasMetric(threshold=0.85, metric="faithfulness", model="gpt-4o")
relevancy_metric    = RagasMetric(threshold=0.80, metric="answer_relevancy", model="gpt-4o")


@pytest.mark.parametrize("row", [json.loads(l) for l in Path("golden/rag_support_v3.jsonl").read_text().splitlines()])
def test_ragas(row):
    tc = LLMTestCase(
        input=row["input"],
        actual_output=row["prediction"],
        expected_output=row["expected_output"],
        retrieval_context=row["retrieved_context"],
    )
    assert_test(tc, [faithfulness_metric, relevancy_metric])

# .github/workflows/rag-eval.yml
name: rag-eval
on: pull_request

jobs:
  ragas-deepeval:
    runs-on: ubuntu-latest
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install deepeval ragas
      - run: pytest tests/eval/test_rag_ragas.py -q --html=rag-eval.html --self-contained-html
      - uses: actions/upload-artifact@v4
        with:
          name: rag-eval-report
          path: rag-eval.html

Step-by-step explanation.

1. LLMTestCase from DeepEval carries the four fields Ragas needs: input (the user question), actual_output (the model's answer), expected_output (the golden), retrieval_context (the chunks the RAG retrieved). Ragas metrics compute against these fields.
2. RagasMetric wraps Ragas's faithfulness and answer_relevancy metrics with a DeepEval-compatible threshold and a scoring model (GPT-4o). assert_test runs the metrics and raises pytest failures if any threshold is missed.
3. The @pytest.mark.parametrize fans out one test per row. Pytest runs them in parallel with -n auto; a 500-row set completes in ~2 minutes at 20-way parallelism.
4. On a faithfulness regression (say, a retrieval index change that surfaces less-relevant chunks), the faithfulness metric drops on many rows; DeepEval fails the tests; the HTML report shows exactly which rows fell below 0.85. The PR is blocked.
5. Ragas + DeepEval is the fastest path from "we have a RAG chatbot" to "we have a gated CI harness" — typically a day of work vs a week for hand-rolled. The trade-off is less flexibility on the metric side; once you outgrow the built-in metrics, migrate to a hand-rolled harness with the same pytest wiring.

Output.

tests/eval/test_rag_ragas.py::test_ragas[row-0] PASSED
tests/eval/test_rag_ragas.py::test_ragas[row-1] PASSED
tests/eval/test_rag_ragas.py::test_ragas[row-2] FAILED

    faithfulness = 0.72 (threshold 0.85)
    answer_relevancy = 0.88 (threshold 0.80)

    Failure reason: the answer contained a URL not present in the retrieved
    context — hallucinated citation.

Rule of thumb. Ragas + DeepEval is the "buy" decision; hand-rolled is the "build" decision. Buy when time-to-first-metric matters more than metric flexibility. Migrate to build when the built-in metrics don't cover a task-specific axis.

Senior interview question on CI + production monitoring

A senior interviewer might ask: "You join a team with a green LLM eval dashboard but customers complaining about quality. Walk me through the first two weeks — how you'd diagnose the CI-vs-prod gap, what monitoring you'd add, and how you'd close the loop."

Solution Using a "same harness, two schedulers, three alerts" hardening plan

# hardening-plan.yaml — 2 weeks from "CI green, customers unhappy" to closed-loop eval
version: v1.0

week_1:
  day_1:
    - audit golden set: does it reflect real user turns?
    - if not: sample 200 rows from real live conversations, SME-relabel
  day_2:
    - stand up production sampler (1% stratified, hourly Airflow task)
    - deploy prod scoring: cosine + cheap judge, no-reference rubric
  day_3:
    - build drift alert query: 24h rolling vs same-day last week
    - integrate to PagerDuty; test with historical data
  day_4:
    - retrofit PR gate baseline: switch from single-run to 7-day moving avg
    - reason: single-run baseline is flake-prone; moving avg is stable
  day_5:
    - retrofit per-class alerts: any class drops 10% = page (finer signal)
  weekend:
    - soak: watch the metrics; expect ~2 alerts (real signal + calibration noise)

week_2:
  day_1:
    - reconcile: for the top-3 customer-reported issues, is the golden set covering them?
    - if not: add rows in the missing class + retrain the SME labelers
  day_2:
    - cross-judge calibration: run 100 rows through GPT-4o and Claude Opus
    - if r < 0.85: tighten rubric; publish updated hash
  day_3:
    - length-bias detector: run against last 30 days of judge scores
    - if bias > 0.02: append anti-bias language; re-baseline
  day_4:
    - Ragas / DeepEval evaluation: does the built-in RAG suite catch what our custom does not?
    - integrate the specific Ragas metric that catches a real failure
  day_5:
    - runbook + on-call rotation: eval-gate + prod-drift on the same paged rotation

Step-by-step trace.

Day	Activity	Signal added
1	Golden-set audit	Coverage of real turns
2	Prod sampler	Continuous prod metric
3	Drift alert	Weekly-seasonality-aware
4	Rolling baseline	Flake-resistant PR gate
5	Per-class alerts	Localized regression detection
Week 2	Judge calibration + bias + framework fill-ins	Judge trustworthiness

After two weeks, the team has: (1) a golden set that reflects real user turns, (2) a production sampler catching drift on a 24-hour horizon, (3) a PR gate with a stable rolling baseline, (4) per-class alerts for localized regressions, and (5) a calibrated judge with instrumented length-bias detection. The CI-vs-prod gap closes; customer complaints line up with metric drops.

Output:

Signal	Before	After
Golden set reflects real turns	no	yes
Production monitoring	none	1% hourly sample
PR baseline	last-run	7-day moving avg
Per-class alerts	none	10% drop = page
Cross-judge calibration	never	weekly
Length bias monitored	no	nightly
CI-prod gap	large	closed

Why this works — concept by concept:

• Same harness, two schedulers — the PR gate and the prod monitor share code paths. Debugging is easier because a bug in the harness surfaces in both places at once; the team is not maintaining two scoring stacks.
• Rolling baseline — 7-day moving average absorbs individual-run noise. A single unlucky run does not lock the team out of every future PR; a real regression persists through the moving average and still triggers the gate.
• Stratified prod sampler — 1 percent per (tenant, topic) prevents a single high-volume tenant from dominating. The metric is representative of the full traffic distribution, not just of the largest customer.
• Cross-judge calibration + length-bias check — the judge is a moving target too. Weekly cross-model calibration keeps the judge honest; nightly length-bias check catches rubric drift before it corrupts weeks of gating.
• Cost — 2 senior-engineer weeks + ~$200/month in judge + prod-sampler API cost. The alternative (unbounded customer complaints + eng escalation) is measured in engineering-days per week; the eval stack pays for itself the first time it catches a bad deploy.

ETL
Topic — etl
ETL problems on CI harnesses and drift monitoring

Practice →

Real-time analytics
Topic — real-time-analytics
Real-time analytics problems on rolling baselines and alerts

Practice →

---

Cheat sheet — LLM eval recipes

• Golden set YAML template. path: golden/support_v3.jsonl + version: v3.2-sha-e1f2c9 + rows: 300 + class_balance: {billing: 60, account: 60, product: 60, shipping: 60, other: 60} + growth: +20 rows/week + curation: SME-labelled + LLM-augmented. Every row is (input, expected_output, metadata) where metadata carries topic, difficulty, language, source_ticket_id, sme_labeler, reviewer, retrieval_snapshot_id, forbidden_patterns, style_tag. Store as JSONL in git; content-hash on canonicalized JSON; tag per release; verify hash on load.
• Cosine scorer 20-line skeleton. Pin embedding: text-embedding-3-large@2024-11-01. Batch 100 at a time; L2-normalize; dot product for cosine. Threshold per language (en=0.86, es/fr=0.83); calibrate against 200 correct + 200 contradiction pairs; sweep F1 or cost-weighted. Judge-on-miss to save cost. Every row emits (row_id, topic, cosine, cos_pass, judge_score, final_pass, embedding_version).
• LLM-as-judge prompt template. System: explicit 2-4 criterion rubric with (0.0-1.0) scale + failure modes named + "you are blind to length" + paired-example demonstration. User: INPUT: ... EXPECTED: ... PREDICTION: .... Response format: {"type": "json_object"}; temperature 0; pydantic-validate on receive. Weighted overall score, e.g. 0.5*correctness + 0.3*completeness + 0.2*style. Hash the prompt; bump on every edit.
• Judge model choice. Use a different-family model from production (GPT-4o judge for Claude Sonnet prod, Claude Opus judge for GPT-4o prod). Cheap sample judge (gpt-4o-mini) on PR + strong nightly judge (gpt-4o or claude-opus) for calibration. Cross-judge Pearson r ≥ 0.85 before trusting either.
• pytest regression harness. assert_test(TestCase(input=..., expected=..., metrics=[GEval(...)])) with DeepEval or a hand-rolled pytest.mark.parametrize. Gate: mean >= baseline - 0.05 against 7-day moving-average baseline. Post per-class delta table into PR via gh pr comment. Cache the baseline via actions/cache keyed on base branch.
• CI eval workflow. Trigger: pull_request on prompts/**, retrieval/**, model_config/**, golden/**. Sample 20 percent of the golden set (~400 rows) for the PR run; nightly full-set with the stronger judge. Timeout 5-10 minutes; parallelism 20; retry once on transient judge errors. Post pass/fail comment always.
• Production sampler. Hourly Airflow task; 1 percent stratified sample per (tenant, topic) via ROW_NUMBER() OVER (PARTITION BY tenant, topic ORDER BY random()). Score with cheap judge (gpt-4o-mini) + cosine on retrieved context. Write to prod_eval_scores(conversation_id, ts, tenant, topic, cosine, judge_score, run_id).
• Drift alert query. 24h_rolling_mean - same_day_last_week_mean <= -0.05 → PAGE, <= -0.03 → WARN. Use INTERVAL '8 days' to INTERVAL '7 days' for same-day-last-week baseline (handles weekly seasonality). Alert on per-class rolling means too — any class drops 10 percent = page.
• Sizing the eval budget. annual_cost = golden_set_size × judge_cost_per_row × runs_per_week × 52. Levers: sample 20 percent on PR + full nightly (cuts 5×); cheap judge on sample + expensive nightly (cuts another 5×); reduce sensitive runs to weekly (cuts another 5×). Target: total eval < 5 percent of the LLM-feature's production inference bill.
• Cross-judge calibration. Weekly: score 100 rows through two judges (different families), compute Pearson r + Cohen's κ at 0.7. r ≥ 0.85 → pass; r in [0.7, 0.85) → tighten rubric; r < 0.7 → rubric is broken. Log the r value; alert on deterioration week-over-week.
• Length-bias detector. Nightly regression of judge_score on standardized len(prediction) while controlling for correctness. Slope > 0.02 = biased. Fix: append "you are blind to length" plus a paired example (short + long same-content predictions scored identically). Re-run detector; slope should drop below 0.01.
• Ragas / DeepEval / promptfoo — when to reach for each. Ragas: RAG-specific metrics (faithfulness, answer_relevancy, context_precision, context_recall) with judge under the hood — best when you need a fast RAG dashboard. DeepEval: pytest-native, wraps any metric — best when your team already writes pytest. promptfoo: CLI matrix runner for prompt × model × input — best for lightweight prompt A/B, not production monitoring. Hand-rolled: once you outgrow built-in metrics, roll your own with the same pytest wiring.
• Prompt versioning discipline. Every prompt (production, judge, RAG-retrieval rewrite) is a hashed artefact in git. Every metric row carries the prompt_hash; comparing across hashes is disallowed. Prompt edits go through PR review; the eval-gate runs against the new hash before merge.
• Failure modes catalogue. Golden set drift (stagnant set diverges from prod distribution) → weekly production sampling into curation queue. Embedding version drift (silent OpenAI checkpoint change) → pin embedding to dated version. Judge bias creep (length preference returns after prompt edit) → nightly length-bias detector. Baseline flake (single unlucky run) → 7-day moving-average baseline. Prompt hash mismatch (rubric edit not propagated) → harness verifies hash on load and aborts on mismatch.

Frequently asked questions

What is LLM-as-judge?

llm as judge is the pattern where a strong LLM (typically GPT-4o, Claude Opus, or Gemini Ultra) scores your production model's prediction against a golden expected answer using an explicit rubric, returning a structured JSON with a numeric score plus a rationale. Concretely, the judge sees (input, expected_output, prediction) and a rubric like ["correctness", "completeness", "style"], and returns {"score": 0.87, "rationale": "…", "breakdown": {...}}. It is the dominant modern LLM-eval metric because it handles semantic equivalence (paraphrases score high) and semantic opposition (contradictions score low) far better than cosine similarity or BLEU. The engineering discipline that makes it work is: version-controlled rubric, structured JSON output (response_format={"type": "json_object"}), temperature=0 for reproducibility, and a stronger or different-family judge model than production to avoid correlated failure modes. Judge cost per row is the finance-visible constraint — typically $0.001-0.01 per row depending on model + rubric length, so sample-on-PR + full-nightly is the default cost-control pattern.

Cosine similarity vs LLM-as-judge — which do I use for what?

cosine similarity is the cheap, fast, deterministic first-pass metric: embed both the expected and the prediction with a pinned embedding model (text-embedding-3-large is the 2026 default), L2-normalize, dot-product, threshold. It costs a fraction of a cent per row, catches structural hallucinations (wrong entities, wrong URLs, missing named entities) beautifully, and is the right gate for "did we regress structurally by 10 percent aggregate." But it fails on paraphrase (semantically identical but different surface form) and contradiction (opposite polarity but similar surface form) — a "yes" and a "no" can embed similarly, and the metric silently misses the polarity flip. llm as judge is the truth arbiter for those cases: expensive but semantically correct. The mature pipeline runs cosine first (per-row, on every row), escalates to the judge only on cosine misses (a "judge-on-miss" pattern that cuts judge cost by ~7×), and uses both scores in the aggregate. Never use cosine alone as a truth metric; never use judge alone as a first-pass filter — the price-performance sweet spot is the cascade.

How big should my golden set be?

The floor is 200 rows for a mid-complexity task; the working default is 300-500; regression-forensics teams run at 1000-2000. The math: at N=20 rows with per-row score standard deviation of ~0.08, the two-sided 95 percent confidence half-width is 0.035 — meaning any regression under 3.5 percent is indistinguishable from noise. At N=200 the half-width drops to 0.011 — a 5 percent regression is caught with a 4× margin, and per-class slices (10-topic buckets × 20 rows each) have enough rows for statistical power. The bigger reason is class coverage: a 20-row demo set has no coverage at all, so localized regressions in unrepresented classes vanish silently; a 300-row balanced set with 5 topic classes × 60 rows each catches localized issues before they ship. Growth cadence: +20 rows/week from production sampling with SME review. Warning sign: if you cannot slice your metric by topic / difficulty / language and get a statistically meaningful number, your golden set is too small.

Do I need to re-evaluate on every prompt change?

Yes, on every PR that touches prompts, retrieval config, model config, or the golden set itself. Prompts are the highest-leverage lever on LLM behaviour; a two-word rewording of a system prompt can silently reshape entire answer classes. The mature pattern: PR gate runs a 20 percent sample (~400 rows) with the cheap judge in ~30 seconds and blocks the merge on a 5 percent aggregate drop; nightly full-set (2000 rows) with the stronger judge recalibrates the baseline. Skipping the PR gate means the first time you notice a bad prompt is when a customer opens a support ticket. Cost model: for a 400-row PR sample with a gpt-4o-mini judge at ~$0.006/row, each PR eval costs ~$2.40. At 40 PRs per week that is ~$5,000/year — a rounding error next to the cost of a single production incident. Exception: hotfix prompts that touch a well-understood, narrow behaviour can bypass the gate with an explicit override + a manual eval commitment within 24 hours. Never let the exception become the rule.

How do I detect LLM regression in production?

Stand up the same eval harness against a 1 percent stratified sample of live traffic, run hourly via Airflow / Dagster / Prefect, write per-conversation scores to a warehouse table, and alert on a 24-hour rolling mean drop vs same-day-last-week. Stratify by (tenant, topic) so a single high-volume tenant does not dominate. Score with the same cheap judge you use on the PR sample, so the production and CI metrics are directly comparable. Alert at two thresholds: -3 percent = WARN (Slack post); -5 percent = PAGE (PagerDuty). Add per-class alerts on any topic dropping more than 10 percent — localized regressions hide inside a healthy-looking aggregate. The reference alert query compares AVG(judge_score) WHERE ts >= NOW() - INTERVAL '24 hours' against AVG(judge_score) WHERE ts >= NOW() - INTERVAL '8 days' AND ts < NOW() - INTERVAL '7 days' — the last-week baseline handles weekly traffic seasonality. Cost: ~$50/week for a mid-size product; the cheapest reliable prod-quality sensor.

Should I use Ragas / DeepEval / promptfoo or roll my own harness?

Start with a framework — either Ragas (RAG-first with built-in faithfulness, answer_relevancy, context_precision, context_recall metrics) or DeepEval (pytest-native, general-purpose, wraps Ragas + any other metric). Ragas alone for a fast RAG dashboard when you have not yet decided on the judge / harness plumbing. DeepEval alone for non-RAG tasks (classification, extraction, summarization) where the built-in metrics fit. Both together for pytest-native RAG gating — DeepEval's LLMTestCase + Ragas's RagasMetric in a pytest file, gated by assert_test. promptfoo is CLI + YAML for prompt A/B — great for lightweight tuning, not production monitoring. Roll your own once you have (a) a mature golden set, (b) a task-specific rubric that no framework's built-in metric covers, and (c) a team comfortable maintaining Python infrastructure. Migration path from framework → hand-rolled is easy: keep the pytest wiring; swap the metric implementations from RagasMetric to your own scorer. The one thing not to do: build a bespoke non-pytest harness first — nobody trusts a system they cannot run locally with pytest.

Practice on PipeCode

• Drill the ETL practice library → for the golden-set curation, versioned dataset, and pipeline-scheduler problems senior interviewers love.
• Rehearse on the data-analysis practice library → for the class-balance, threshold-calibration, and per-slice-metric problems that motivate the LLM-eval harness in the first place.
• Sharpen the tuning axis with the optimization practice library → for the eval-cost, judge-sampling, and CI-throughput problems that decide whether the harness scales.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the golden-set + judge + CI intuition against real graded inputs.

Lock in LLM-evaluation muscle memory

Framework docs explain metrics. PipeCode drills explain the decision — when cosine is enough, when the judge must kick in, when the golden set is too small to trust, when the PR gate is the wrong place to catch drift. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice ETL problems →
Practice data-analysis problems →