LLM Evaluation for Data Pipelines: LangSmith, TruLens, Ragas & Snowflake Cortex Search Ops

llm evaluation for data pipelines is the load-bearing correctness discipline of the 2026 data stack — the difference between a RAG chatbot that quietly regresses for three weeks before a customer notices and a RAG chatbot whose retrieval quality is gated on every pull request. Every LLM call your pipeline makes — the summariser inside a nightly ingestion DAG, the classifier fronting a Kafka stream, the RAG chain answering support tickets, the SQL-generation copilot inside Snowsight — has failure modes that unit tests do not catch: hallucinations that look confident, retrieval misses that return the wrong policy document, prompt drift where a model swap flips the tone, cost explosions where a runaway agent burns a quarter's budget in one afternoon. Those failures do not raise exceptions; they degrade the output. The only way to catch them is to measure the output on every run and gate on the score.

This guide is the senior-DE / MLOps walkthrough you wished existed the first time an interviewer asked "walk me through your llm evaluation for data pipelines stack" or "how would you compare langsmith, trulens, and ragas?" or "how do you gate a RAG deployment on retrieval quality metrics?" It walks through the four axes every senior discussion converges on (groundedness, answer relevance, context precision / recall, latency + cost), the four production-hardened tools that ship those metrics in 2026 — LangSmith for trace-first eval and versioned datasets, TruLens for feedback functions and the rag evaluation triad, Ragas for reference-free metrics inside CI, and Snowflake Cortex Search Ops for eval at the semantic-search layer — and finishes each section with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the data-validation practice library →, and sharpen the systems axis with the design practice library →.

---

On this page

• Why LLM pipelines fail silently without eval — the four axes
• LangSmith — trace-first eval, dataset versioning, hosted judges
• TruLens — feedback functions and the RAG triad
• Ragas — reference-free RAG metrics for CI
• Snowflake Cortex Search Ops — eval at the semantic-search layer
• Cheat sheet — LLM eval recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why LLM pipelines fail silently without eval — the four axes

Four axes, four measurement tools — the choice binds every downstream release gate

The one-sentence invariant: llm evaluation for data pipelines is the discipline of measuring model + retrieval output on every run against four independent axes — groundedness (did the answer cite the retrieved context?), answer relevance (did the answer address the user's question?), context precision / recall (did retrieval surface the right chunks?), and latency + cost — because unit tests cannot catch hallucination, retrieval miss, prompt drift, or budget blowup, and shipping without a numeric score on each axis is the modern equivalent of shipping without regression tests. The four axes are orthogonal: a RAG chain can be perfectly grounded in the retrieved context and still fail answer relevance because the retrieval was wrong; retrieval can be perfect and grounding low because the model paraphrased poorly; both can be high while latency p99 blows past your SLO or per-request cost triples on a model swap. Measuring one axis and calling it "LLM eval" is the single most common mistake senior candidates make in interviews.

The four axes senior interviewers actually probe.

• Groundedness. Of every claim in the generated answer, what fraction is directly supported by the retrieved context? An answer that invents a fact the retrieved documents do not contain scores low on groundedness even if the fact is correct in the world. TruLens calls this "groundedness"; Ragas calls it "faithfulness"; both compute it as an LLM-judge score over (context, answer) pairs. This is the primary anti-hallucination metric and the first metric any RAG deployment should ship.
• Answer relevance. Does the answer actually address the user's question? A perfectly grounded answer that ignores the question ("here is the entire retrieved policy document") scores low on answer relevance. Ragas measures this by having an LLM generate hypothetical questions the answer could address, then computing cosine similarity between those questions and the original query. High answer relevance means the answer stays on topic.
• Context precision / recall. Of the top-k retrieved chunks, how many are relevant (precision), and of the relevant chunks in the corpus, how many appear in top-k (recall)? These are the classic IR metrics reborn for RAG. Ragas ships context_precision and context_recall; TruLens exposes "context relevance" as the top vertex of the RAG triad. Low precision means noisy retrieval; low recall means missing information.
• Latency + cost. The pipeline axis. p50 / p95 / p99 latency per LLM call, tokens per call, dollars per call, tokens per user query end-to-end. A model swap that improves groundedness by 3 points but triples cost is a regression on this axis. Every eval framework should ship latency + cost alongside quality scores; if it doesn't, wire it up separately with a @traceable wrapper.

The 2026 reality — four tools, four sweet spots, they coexist.

• LangSmith is the trace-first hosted platform: every LLM call is a Run inside a hierarchical Trace; datasets are versioned; experiments diff scores over the same dataset; hosted LLM-as-judge evaluators run in the LangSmith cloud. Sweet spot: teams already using LangChain / LangGraph who want a hosted dashboard and dataset store.
• TruLens is the local-first open-source eval library: TruChain / TruLlama wrap your app, Feedback functions score every Record, and the RAG triad (context relevance, groundedness, answer relevance) ships as a canonical pattern. Sweet spot: teams that want to keep eval inside their VPC, avoid a hosted vendor, and instrument any Python app (not just LangChain).
• Ragas is the reference-free metric library: EvaluationDataset + evaluate() returns a pandas DataFrame of faithfulness, answer_relevancy, context_precision, context_recall. Sweet spot: CI gates — a nightly Airflow job runs Ragas over a golden query set and fails the build if any metric regresses beyond a threshold.
• Snowflake Cortex Search Ops brings the eval into SQL: CORTEX SEARCH SERVICE is the semantic search index; AI_COMPLETE('claude-3-5-sonnet', ...) is the SQL judge; recall@k / MRR / nDCG are computed as plain aggregate SQL over (query, expected_doc_id, retrieved_doc_ids) tables. Sweet spot: teams whose entire data + eval + model stack lives inside one Snowflake account and one governance boundary.

What interviewers listen for.

• Do you name all four axes without prompting? — senior signal.
• Do you distinguish groundedness from answer relevance rather than treating "hallucination" as one blob? — senior signal.
• Do you name at least two eval tools and state their sweet spots? — required answer.
• Do you describe eval as a CI gate, not as "a dashboard we look at sometimes"? — senior signal.
• Do you name latency + cost as an eval axis, not as "just an SRE concern"? — required answer.

Worked example — the four-axis measurement plan for a support-ticket RAG

Detailed explanation. The single most useful artifact for an LLM eval interview is a memorised 4-axis measurement plan against a concrete scenario. Every senior LLM eval discussion converges on this plan within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the plan for a support-ticket RAG chatbot that answers customer questions from a Confluence corpus.

• Application. RAG chatbot; input = customer question; output = plain-language answer + citations to Confluence pages.
• Corpus. ~80,000 Confluence documents, ~1.2 million chunks.
• Retrieval. Hybrid (BM25 + vector) top-8 chunks.
• LLM. Claude Sonnet 4.7 in production; Claude Haiku for the judge.

Question. Build the four-axis measurement plan for the support-ticket RAG and name the tool that ships each metric.

Input.

Axis	Metric	Tool	Threshold (v1)
Groundedness	fraction of answer claims supported by context	Ragas faithfulness	≥ 0.85
Answer relevance	question-answer topical alignment	Ragas answer_relevancy	≥ 0.80
Context precision	fraction of top-8 chunks that are relevant	Ragas context_precision	≥ 0.70
Context recall	fraction of relevant chunks in top-8	Ragas context_recall	≥ 0.75
Latency p95	end-to-end wall-clock	LangSmith trace	≤ 4 s
Cost per query	tokens × price	LangSmith trace	≤ $0.015

Code.

# The four-axis eval plan, expressed as a dataclass
from dataclasses import dataclass
from typing import Callable

@dataclass
class EvalAxis:
    name: str
    metric: str
    tool: str
    threshold: float
    higher_is_better: bool

PLAN = [
    EvalAxis("Groundedness",       "faithfulness",       "Ragas",     0.85, True),
    EvalAxis("Answer relevance",   "answer_relevancy",   "Ragas",     0.80, True),
    EvalAxis("Context precision",  "context_precision",  "Ragas",     0.70, True),
    EvalAxis("Context recall",     "context_recall",     "Ragas",     0.75, True),
    EvalAxis("Latency p95",        "end_to_end_ms",      "LangSmith", 4000, False),
    EvalAxis("Cost per query",     "cost_usd",           "LangSmith", 0.015, False),
]

def gate(scores: dict[str, float]) -> tuple[bool, list[str]]:
    """Return (passed, list_of_failures). Runs in the CI job."""
    failures = []
    for axis in PLAN:
        score = scores.get(axis.metric)
        if score is None:
            failures.append(f"{axis.metric}: missing")
            continue
        ok = (score >= axis.threshold) if axis.higher_is_better else (score <= axis.threshold)
        if not ok:
            failures.append(f"{axis.metric}: {score:.3f} vs threshold {axis.threshold}")
    return (len(failures) == 0, failures)

Step-by-step explanation.

1. Each axis maps to one metric and one tool that ships that metric out of the box. Ragas is the workhorse for the four quality metrics because it computes all four from a single EvaluationDataset in one evaluate() call. LangSmith is the workhorse for latency + cost because every @traceable call emits a Run with wall-clock and token counts already attached.
2. Thresholds start conservative — 0.85 for faithfulness is aggressive but achievable with Sonnet-tier models and a well-tuned retriever. Set thresholds slightly below your first stable measurement so day-two changes have headroom; lift thresholds as the pipeline matures.
3. The higher_is_better flag flips the comparison for latency and cost — those are "lower is better" axes. Every gate library gets this wrong once; codifying the direction in the dataclass prevents it.
4. The gate() function is the CI contract. It runs in the GitHub Action after Ragas + LangSmith have populated the scores dict. Any failure fails the build; the failure list is posted as a PR comment. This is the pattern that turns eval from "a dashboard" into "a release gate."
5. In production, thresholds evolve. Version the PLAN list in git alongside your prompts and your dataset version; a threshold change is a code review just like a prompt change. Never bump a threshold to make a failing PR green without a written justification.

Output.

Scenario	Passed?	Failures
v1 baseline	yes	none
Model swap Sonnet → Haiku	no	faithfulness: 0.78 vs 0.85
Retriever change hybrid → vector-only	no	context_precision: 0.62 vs 0.70
Prompt tweak (add citation format)	yes	none
Cost-optimised (batch calls)	no	end_to_end_ms: 5200 vs 4000

Rule of thumb. Never pick an LLM eval tool without first writing down the four-axis plan and the numeric threshold on each axis. The tool is downstream of the plan; if the plan is missing, the tool choice is arbitrary and the eval will drift into "vibes."

Worked example — the senior LLM-eval interview grading rubric

Detailed explanation. The senior LLM-eval interview grades a small set of specific signals. Memorise the rubric; every answer should touch every row.

• Axes named in minute 1. All four (groundedness, answer relevance, context precision/recall, latency + cost) — senior signal.
• Tools per axis. Ragas for the four quality metrics, LangSmith for latency/cost, TruLens for VPC deployments, Cortex Search Ops for Snowflake-native — required.
• Gate mechanism. "GitHub Action fails the PR on regression" — required.
• Judge validation. Quarterly re-validation against human labels; correlation ≥ 0.75 — senior signal.

Question. Score a candidate's answer against the rubric and pick the one differentiator that separates senior from mid-level responses.

Input.

Row	Weak answer	Senior answer
Axes	"accuracy"	"four axes: groundedness, answer relevance, context precision/recall, latency + cost"
Tools	"we look at LangSmith"	"Ragas in CI, LangSmith for traces, TruLens for VPC"
Gate	"we watch a dashboard"	"GitHub Action fails PR on any axis regression"
Judge	"LLM judge is fine"	"re-validated quarterly against 100 human-labelled rows; correlation tracked"

Code.

# rubric.py — score a candidate answer against the LLM-eval rubric
RUBRIC = [
    ("axes",   ["groundedness", "answer relevance", "context precision", "latency", "cost"]),
    ("tools",  ["ragas", "langsmith", "trulens"]),
    ("gate",   ["github action", "fail", "regression"]),
    ("judge",  ["human", "correlation", "re-validate"]),
]

def score(answer: str) -> dict[str, float]:
    a = answer.lower()
    return {row: sum(1 for k in keys if k in a) / len(keys) for row, keys in RUBRIC}

candidate = ("I'd measure groundedness, answer relevance, context precision, and latency. "
             "Ragas + LangSmith. GitHub Action fails the PR on regression. "
             "The LLM judge is re-validated quarterly against human-labelled rows.")

print(score(candidate))
# → {'axes': 0.8, 'tools': 0.67, 'gate': 1.0, 'judge': 0.67}

Step-by-step explanation.

1. Each rubric row is a set of keyword indicators. Naming all axes hits row 1 fully; naming two tools scores partial credit on row 2.
2. The gate row is binary — either the candidate names an automated PR-blocking mechanism or they don't. This is the single biggest senior differentiator.
3. Judge validation is the row most candidates miss. Naming "human correlation" or "quarterly re-validation" separates the mid-level from senior.

Output.

Row	Weak candidate	Senior candidate
axes	0.2	0.8-1.0
tools	0.0-0.33	0.67-1.0
gate	0.0	1.0
judge	0.0	0.67-1.0

Rule of thumb. If you touch three of four rows at 0.6+, you pass the senior bar. The gate row is the mandatory one; missing it means "mid-level" regardless of the other three.

Senior interview question on LLM eval axes and CI

A senior interviewer often opens with: "You inherit a RAG chatbot that has no eval — just a LangChain chain wired to Postgres pgvector and Claude Sonnet. The product team keeps shipping prompt tweaks and quality is regressing. Walk me through the eval stack you'd stand up in the first sprint, the four axes you'd measure, and the CI gate you'd wire into GitHub Actions."

Solution Using Ragas as the CI gate with LangSmith traces for latency and cost

# eval_gate.py — the CI job that scores + gates every PR
import json, sys
from pathlib import Path
from datasets import Dataset
from langsmith import Client
from ragas import evaluate
from ragas.metrics import (faithfulness, answer_relevancy,
                            context_precision, context_recall)

GOLDEN = Path("eval/golden_v3.jsonl")
THRESHOLDS = {"faithfulness": 0.85, "answer_relevancy": 0.80,
              "context_precision": 0.70, "context_recall": 0.75}

def main() -> int:
    from myapp.rag import rag_chain
    predictions = []
    for row in [json.loads(l) for l in GOLDEN.read_text().splitlines()]:
        resp = rag_chain.invoke({"query": row["question"]})
        predictions.append({"question": row["question"],
                             "answer": resp["answer"],
                             "contexts": [d.page_content for d in resp["source_documents"]],
                             "ground_truth": row["ref_answer"]})

    result = evaluate(Dataset.from_list(predictions),
                      metrics=[faithfulness, answer_relevancy,
                                context_precision, context_recall])
    scores = result.to_pandas().mean(numeric_only=True).to_dict()

    # Attach latency + cost from LangSmith traces
    ls = Client()
    runs = list(ls.list_runs(project_name="rag-ci", limit=len(predictions)))
    latencies = sorted(r.latency for r in runs if r.latency is not None)
    scores["latency_p95_ms"] = latencies[int(0.95 * len(latencies))] * 1000
    scores["cost_usd_avg"]   = sum(r.total_cost or 0 for r in runs) / len(runs)

    failures = [f"{k}={scores[k]:.3f} < {v}" for k, v in THRESHOLDS.items()
                if scores.get(k, 0) < v]
    Path("eval/report.json").write_text(json.dumps(scores, indent=2))
    if failures:
        print("EVAL GATE FAILED:", failures); return 1
    print("EVAL GATE PASSED", scores); return 0

if __name__ == "__main__":
    sys.exit(main())

# .github/workflows/eval-gate.yml — the PR gate
name: LLM Eval Gate
on:
  pull_request:
    paths: ["prompts/**", "src/rag/**", "eval/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - name: Run gate
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
          LANGSMITH_PROJECT: "rag-ci"
        run: python eval_gate.py
      - name: Comment scores on PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require("fs");
            const report = fs.readFileSync("eval/report.json", "utf8");
            github.rest.issues.createComment({
              owner: context.repo.owner, repo: context.repo.repo,
              issue_number: context.issue.number,
              body: "## LLM eval scores\n```
{% endraw %}
json\n" + report + "\n
{% raw %}
```" });

Step-by-step trace.

Step	Before (no eval)	After (Ragas + LangSmith CI gate)
Regression detection	user complaint (weeks later)	PR fails within minutes
Metrics tracked	none	4 quality + 2 pipeline axes
Judge	none	Claude Haiku as Ragas judge
Dataset	ad-hoc curl commands	versioned golden_v3.jsonl
PR feedback	none	GitHub bot comments scores
Threshold governance	verbal	code-reviewed dict
Cost visibility	monthly bill shock	per-PR cost delta

After deployment, every PR that touches prompts/** or src/rag/** runs the full eval gate, the four Ragas metrics + latency p95 + cost/query land as a JSON on the PR, and any axis regression below its threshold fails the check. The product team's prompt tweaks either improve the numbers or fail the merge; there is no "ship first, measure later" path anymore.

Output:

Metric	Baseline (main)	PR #482 (Haiku swap)	PR #483 (retriever tweak)
faithfulness	0.89	0.78 (fail)	0.90
answer_relevancy	0.86	0.81	0.87
context_precision	0.74	0.72	0.79 (improved)
context_recall	0.78	0.77	0.82
latency_p95_ms	3800	2100	4300 (fail)
cost_usd_avg	0.011	0.003	0.013

Why this works — concept by concept:

• Ragas reference-free metrics — faithfulness, answer_relevancy, context_precision, and context_recall all compute from (question, answer, contexts, ground_truth?) alone. faithfulness and answer_relevancy don't even need ground_truth; context_precision and context_recall do. This is why Ragas is the CI workhorse — no reference-answer curation required for two of the four axes.
• LangSmith @traceable runs — every LLM call inside the RAG chain emits a Run with wall-clock latency, prompt tokens, completion tokens, and computed cost. The ls.list_runs() call at gate time pulls the last N runs for the rag-ci project. This is how latency and cost enter the gate without a separate instrumentation layer.
• Versioned golden dataset — golden_v3.jsonl in git is the single source of truth for what "the pipeline should do." Growing the dataset is monotonic (append new curated rows; never delete); a dataset bump is a PR just like a prompt bump. Two experiments over the same dataset version are the only fair comparison.
• Threshold dict in code — THRESHOLDS lives in eval_gate.py, code-reviewed with every change. This is the difference between "we lowered the threshold to make the PR pass" (a compliance failure) and "we lowered the threshold with a written justification approved by the on-call reviewer" (a governed change).
• Cost — one CI run costs roughly 120 questions × ($0.011 per RAG call + $0.002 per Ragas judge call) ≈ $1.60 per PR. Compared to the cost of a bad model swap reaching production undetected for a week (support tickets, refunds, reputation), this is the cheapest quality gate you will ever ship. O(dataset_size) per PR, O(1) latency to detect regressions.

ETL
Topic — etl
ETL problems on eval pipelines and gates

Practice →

Data validation Topic — data-validation Data validation problems on quality gates

Practice →

---

2. LangSmith — trace-first eval, dataset versioning, hosted judges

Every LLM call is a Run inside a Trace — datasets are versioned, experiments diff over them, judges score them

The mental model in one line: langsmith is the trace-first eval platform where every LLM call your code makes is wrapped in @traceable to emit a hierarchical Run (child of a parent Trace), your golden inputs live in a versioned Dataset, an Experiment runs your target function against every example in the dataset and pipes the outputs through a set of evaluators (LLM-as-judge or Python), and the LangSmith UI diffs experiments side by side so a prompt change or a model swap has an immediately visible before/after — it is the tool of choice when your app already lives inside LangChain or LangGraph and you want a hosted dashboard plus a dataset store without building either from scratch. Every senior DE / MLOps engineer working with LangChain has stood up LangSmith at some point; the pattern is the same across products.

The four axes for LangSmith.

• Groundedness. Ships as a hosted LLM-as-judge evaluator (langsmith.evaluation.LangChainStringEvaluator("labeled_criteria", criteria="correctness") or a custom judge prompt). You can also plug Ragas into a LangSmith experiment via the RunEvaluator interface; LangSmith records the Ragas scores as annotations on each Run.
• Answer relevance. Same story — a hosted criteria evaluator with criteria="relevance", or a custom Python judge. LangSmith's differentiator is that the score attaches to the exact trace, so you can drill from "answer relevance dropped 0.05" to the specific 6 runs that regressed.
• Context precision / recall. LangSmith itself doesn't ship retrieval metrics out of the box, but the trace captures the retrieved documents, and you can wire a custom RunEvaluator that computes precision/recall against a labelled dataset. The pattern: dataset row carries expected_context_ids; evaluator compares against the trace's retrieved IDs.
• Latency + cost. Free. Every @traceable decorator emits latency, prompt tokens, completion tokens, and cost per token computed against the model price sheet. The LangSmith dashboard groups by run name, model, project, and dataset. This is the killer feature that keeps teams on LangSmith even when they'd prefer a self-hosted stack.

The trace shape — nested runs.

• Trace. The top-level unit of work. One user query = one trace. Traces have a name, project_name, tags, and a wall-clock start/end.
• Run. A single operation inside the trace — an LLM call, a retriever call, a tool call, a chain step. Runs are nested (a chain run contains child runs for each component). Runs have inputs, outputs, run_type (llm / chain / tool / retriever), and cost/latency metadata.
• Feedback. A score attached to a run by an evaluator (LLM judge, Python function, or human). Multiple feedback entries per run are allowed (e.g. one for faithfulness, one for relevance, one for latency).
• Dataset. A versioned collection of Example rows (inputs, outputs optional). Datasets are immutable per version; adding rows creates a new version.
• Experiment. A run of a target function against every example in a dataset version, with a set of evaluators. Experiments have a name, a dataset version pin, and produce one Feedback per (Example, evaluator) pair.

Dataset versioning — the disciplined pattern.

• Immutable versions. Every dataset write (add / edit / delete row) creates a new version. Experiments pin to a version by ID; a rerun months later against the same version is byte-for-byte comparable.
• Naming. rag_eval_v3_2026_07_30 — semantic version + date. Never overwrite rag_eval without a version suffix; two experiments named the same but against different underlying rows is the debugging pain from hell.
• Tagging. Rows are tagged (golden, hard, edge_case, regression_from_pr_482). Tags let you slice eval scores by category without duplicating rows.
• Bootstrap. First dataset comes from a spreadsheet of "questions our support team gets" curated by a product manager. Second dataset comes from real production traces flagged as "interesting." Third dataset comes from regression cases (a specific question that broke, immortalised as a row).

The hosted judge — LLM-as-judge inside LangSmith.

• Prebuilt criteria. correctness, relevance, coherence, conciseness, harmfulness. Each is a prompt template that scores an output on a 0-1 scale.
• Custom criteria. Pass a natural-language criterion string; LangSmith wraps it in a scoring prompt and runs it against a model you pick.
• Pairwise evaluators. Given two candidate outputs (from two experiments), pick which is better. Useful for A/B comparisons of prompt versions.
• Cost. Every hosted-judge call is a paid LLM call. Budget: judge cost ≈ 1-2× the production call cost per dataset row.

Common interview probes on LangSmith.

• "What's a Run vs a Trace?" — required answer: Trace is the top-level unit of work; Runs are the nested operations inside it.
• "How do you version an eval dataset?" — required answer: immutable versions, pin experiments to a version ID.
• "What's the difference between a hosted evaluator and a custom evaluator?" — hosted = LangSmith cloud; custom = your Python callable receiving (run, example).
• "How do you gate a PR on LangSmith scores?" — required answer: run an experiment in CI, fetch scores via SDK, compare against thresholds.

Worked example — instrument a LangChain RAG with @traceable

Detailed explanation. The first thing any LangSmith deployment does is turn on tracing. The @traceable decorator (or the auto-instrumented langchain package) captures every LLM call, retriever call, and chain step and ships them to the LangSmith cloud. Once tracing is on, every subsequent query is a debugging goldmine; without it, LangSmith is empty. Walk through the setup for a Postgres-pgvector + Claude RAG chain.

• Chain. RetrievalQA from LangChain: pgvector retriever → prompt template → Claude Sonnet.
• Env vars. LANGSMITH_API_KEY, LANGSMITH_PROJECT=support-rag, LANGSMITH_TRACING=true.
• Traceable functions. The chain, the retriever, the LLM call — all auto-traced by the LangChain integration.

Question. Wire up @traceable around a LangChain RAG so every user query lands as a Trace in the LangSmith UI with per-step timings and token counts.

Input.

Component	Type	Traced?
rag_chain.invoke	Chain	yes (auto)
pgvector.similarity_search	Retriever	yes (auto)
ChatAnthropic.invoke	LLM	yes (auto)
format_answer (custom)	Utility	yes (via @traceable)
answer_endpoint (FastAPI)	Endpoint	yes (via @traceable)

Code.

# rag_app.py — LangChain RAG with LangSmith tracing
import os
os.environ.setdefault("LANGSMITH_TRACING", "true")
os.environ.setdefault("LANGSMITH_PROJECT", "support-rag")

from langchain.chains import RetrievalQA
from langchain_anthropic import ChatAnthropic
from langchain_community.vectorstores import PGVector
from langchain_openai import OpenAIEmbeddings
from langsmith import traceable
from fastapi import FastAPI
from pydantic import BaseModel

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
retriever  = PGVector(
    connection_string=os.environ["PG_URL"],
    embedding_function=embeddings,
    collection_name="support_docs",
).as_retriever(search_kwargs={"k": 8})

llm = ChatAnthropic(model="claude-sonnet-4-7", temperature=0.1)

rag_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True,
    chain_type_kwargs={"verbose": False},
)

@traceable(run_type="tool", name="format_answer")
def format_answer(question: str, resp: dict) -> dict:
    """Trim, cite, and structure the LLM output for the client."""
    return {
        "answer":     resp["result"].strip(),
        "citations":  [d.metadata.get("source_url") for d in resp["source_documents"]],
        "confidence": min(1.0, len(resp["source_documents"]) / 8),
    }

app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/answer")
@traceable(run_type="chain", name="answer_endpoint",
           tags=["prod", "v2.3"])
async def answer(q: Query):
    resp = rag_chain.invoke({"query": q.question})
    return format_answer(q.question, resp)

Step-by-step explanation.

1. Setting LANGSMITH_TRACING=true and LANGSMITH_PROJECT before importing langchain auto-instruments every LangChain primitive. No decorator needed on rag_chain.invoke — the LangChain integration wraps it for you. This is the fastest way to get useful traces.
2. The @traceable(run_type="tool", name="format_answer") on the custom formatter captures the non-LangChain utility inside the same trace tree. run_type="tool" is a hint to the UI (colours the run icon); name overrides the auto-detected function name.
3. The @traceable(run_type="chain", name="answer_endpoint", tags=["prod", "v2.3"]) on the FastAPI endpoint makes the endpoint the top-level trace. Tags like prod and v2.3 are filterable in the LangSmith UI — you can pull all traces for v2.3 in one query.
4. After deploy, every /answer POST creates one trace with three or four nested runs: the endpoint (chain), the RAG chain (chain), the retriever (retriever), the LLM call (llm), and the formatter (tool). Each run has latency, tokens, and cost.
5. Sampling in production: setting LANGSMITH_TRACING=true traces 100% of requests by default. For high-volume prod, use LANGSMITH_TRACING_SAMPLE_RATE=0.01 to trace 1%. Debugging incidents: flip to 1.0 for 15 minutes, capture the incident traces, flip back.

Output.

Trace field	Value
Trace name	answer_endpoint
Total latency	2,340 ms
Total tokens	4,120 prompt + 380 completion
Total cost	$0.0148
Nested runs	4 (endpoint → chain → retriever → llm → tool)
Tags	prod, v2.3
Project	support-rag

Rule of thumb. Turn on LangSmith tracing on day one — even before you have any evaluators wired up. Tracing alone gives you latency + cost per LLM call, per model, per prompt version, and per endpoint, which is more observability than most LLM apps ever get. Add evaluators once the trace tree feels rich enough.

Worked example — run a pairwise experiment over a versioned dataset

Detailed explanation. The killer LangSmith workflow is the pairwise experiment: two candidate configs (say, "prompt v1" vs "prompt v2") both run against the same dataset version, and a hosted pairwise judge picks the better output per row. The result is a win-rate ("prompt v2 wins on 68 of 100 rows"), which is directly consumable by a product review. Walk through the setup.

• Dataset. support_rag_v3 — 100 curated {question, expected_answer_gist} rows.
• Configs. prompt_v1 (current prod) vs prompt_v2 (proposed change).
• Judge. Hosted pairwise evaluator with a "which answer is more helpful?" criterion.
• Output. Win-rate table + drill-down to disagreements.

Question. Run a pairwise LangSmith experiment comparing prompt_v1 and prompt_v2 against support_rag_v3 and interpret the result.

Input.

Field	Value
Dataset	support_rag_v3 (100 examples)
Baseline function	run_with_prompt_v1
Candidate function	run_with_prompt_v2
Judge	Claude Sonnet, criterion "helpfulness"
Metric	pairwise win-rate

Code.

# pairwise_experiment.py
from langsmith import Client
from langsmith.evaluation import evaluate_comparative
from langchain_anthropic import ChatAnthropic

ls = Client()

# --- The two target functions ---
def run_with_prompt_v1(inputs: dict) -> dict:
    from myapp.rag import rag_chain_v1
    return {"answer": rag_chain_v1.invoke({"query": inputs["question"]})["result"]}

def run_with_prompt_v2(inputs: dict) -> dict:
    from myapp.rag import rag_chain_v2
    return {"answer": rag_chain_v2.invoke({"query": inputs["question"]})["result"]}

# --- The pairwise judge ---
JUDGE_PROMPT = """You are comparing two answers to a support question.
Question: {question}
Answer A: {answer_a}
Answer B: {answer_b}
Which answer is more helpful, accurate, and grounded?
Respond with exactly one of: "A", "B", "TIE"."""

judge_llm = ChatAnthropic(model="claude-sonnet-4-7", temperature=0)

def pairwise_helpfulness(runs: list, example: dict) -> dict:
    a, b = runs[0].outputs["answer"], runs[1].outputs["answer"]
    resp = judge_llm.invoke(JUDGE_PROMPT.format(
        question=example.inputs["question"], answer_a=a, answer_b=b,
    ))
    label = resp.content.strip().upper()
    return {
        "key": "helpfulness",
        "scores": [
            1 if label == "A" else 0,
            1 if label == "B" else 0,
        ],
    }

# --- Step 1: run each candidate as a normal experiment ---
r1 = ls.evaluate(
    run_with_prompt_v1,
    data="support_rag_v3",
    experiment_prefix="prompt_v1",
)
r2 = ls.evaluate(
    run_with_prompt_v2,
    data="support_rag_v3",
    experiment_prefix="prompt_v2",
)

# --- Step 2: run the pairwise judge over the two experiments ---
comparative = evaluate_comparative(
    experiments=[r1.experiment_name, r2.experiment_name],
    evaluators=[pairwise_helpfulness],
)
print(comparative)

Step-by-step explanation.

1. Two separate ls.evaluate calls run each candidate against the same dataset version. Each produces its own experiment with its own outputs stored per-example. Using the same dataset version is what makes the comparison fair — a change of dataset between runs would confuse the diff.
2. The pairwise judge is a custom evaluate_comparative evaluator. It receives both runs (one per experiment) plus the example, calls the judge LLM with a "which is better?" prompt, and returns per-experiment scores (1 for winner, 0 for loser, 0.5 for tie in some conventions).
3. Temperature 0 on the judge is mandatory — you want deterministic scoring so a re-run yields the same result. The judge model can be the same as the production model or a stronger one; the common pattern is "production uses Haiku for cost, judge uses Sonnet for quality."
4. evaluate_comparative returns a win-rate summary and stores per-row judge outputs in LangSmith. The UI shows a side-by-side diff for every disagreement, so you can eyeball whether the judge is agreeing with human intuition.
5. The gate: PR is merged if prompt_v2_wins / (prompt_v1_wins + prompt_v2_wins + ties) > 0.60. Anything less than 60% win-rate on a pairwise judge is not a meaningful improvement; ties count against both.

Output.

Metric	Value
Dataset version	support_rag_v3 (100 rows)
prompt_v1 wins	22
prompt_v2 wins	68
Ties	10
prompt_v2 win-rate (excl. ties)	75.6%
Judge cost	$0.42 total
Verdict	ship prompt_v2

Rule of thumb. Prefer pairwise judges over pointwise judges when the goal is "is this change better?" — pairwise is more sensitive to small quality deltas because the judge is asked to compare, not to score in an absolute frame. Save pointwise judges for absolute thresholds ("faithfulness ≥ 0.85").

Senior interview question on LangSmith trace + dataset ops

A senior interviewer might ask: "You've got a LangChain-based RAG in production, no eval, no dataset, complaints piling up. Design the first sprint: turn on tracing, extract a first dataset from production traces, set up a baseline experiment, and wire a pairwise comparison to gate the next prompt change."

Solution Using LangSmith tracing, production-trace-derived dataset, and a pairwise gate

# sprint1_langsmith_setup.py
import os
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_PROJECT"] = "support-rag-prod"

from datetime import datetime, timedelta, timezone
from langsmith import Client
from langsmith.evaluation import evaluate_comparative
from langchain_anthropic import ChatAnthropic

ls = Client()
DATASET = "support_rag_from_prod_v1"

# Step A: extract dataset from last week's low-confidence prod traces
def bootstrap_dataset():
    if any(d.name == DATASET for d in ls.list_datasets()):
        return
    since = datetime.now(timezone.utc) - timedelta(days=7)
    runs = list(ls.list_runs(project_name="support-rag-prod", start_time=since,
                              run_type="chain", limit=300))
    picks = sorted(runs, key=lambda r: (r.outputs or {}).get("confidence", 1.0))[:100]
    ls.create_dataset(dataset_name=DATASET, description="prod-derived seed")
    ls.create_examples(
        dataset_name=DATASET,
        inputs=[{"question": r.inputs["q"]["question"]} for r in picks],
        outputs=[None] * len(picks),
        metadata=[{"source_trace_id": str(r.id)} for r in picks],
    )

# Step B: pinned baseline + pairwise gate for the next prompt PR
def run_with_prompt(v):
    from myapp.rag import make_chain
    chain = make_chain(prompt_version=v)
    return lambda inputs: {"answer": chain.invoke({"query": inputs["question"]})["result"]}

JUDGE = ChatAnthropic(model="claude-sonnet-4-7", temperature=0)
JUDGE_PROMPT = ("Question: {q}\nAnswer A: {a}\nAnswer B: {b}\n"
                'Which is more helpful, accurate, grounded? Reply "A", "B", or "TIE".')

def pairwise_gate(runs, example):
    resp = JUDGE.invoke(JUDGE_PROMPT.format(
        q=example.inputs["question"],
        a=runs[0].outputs["answer"], b=runs[1].outputs["answer"]))
    label = resp.content.strip().upper()
    return {"key": "helpfulness",
            "scores": [1 if label == "A" else 0, 1 if label == "B" else 0]}

def gate_prompt_change(new_version: str, min_win_rate=0.60) -> bool:
    r_old = ls.evaluate(run_with_prompt("v1"), data=DATASET,
                        experiment_prefix="baseline_prompt_v1")
    r_new = ls.evaluate(run_with_prompt(new_version), data=DATASET,
                        experiment_prefix=f"candidate_{new_version}")
    comp = evaluate_comparative(experiments=[r_old.experiment_name, r_new.experiment_name],
                                 evaluators=[pairwise_gate])
    totals = comp["scores"]["helpfulness"]["totals"]
    win_rate = totals["B"] / max(1, totals["A"] + totals["B"])
    return win_rate >= min_win_rate

if __name__ == "__main__":
    bootstrap_dataset()
    print("APPROVED" if gate_prompt_change("v2") else "BLOCKED")

Step-by-step trace.

Step	Before (no eval)	After (LangSmith sprint 1)
Tracing	none	100% of prod traffic (later sampled)
Dataset	none	support_rag_from_prod_v1 — 100 real prod rows
Baseline	vibes	pinned experiment baseline_prompt_v1
Prompt change gate	verbal	pairwise judge, 60% win-rate threshold
Regression memory	Slack thread	permanent rows tagged regression
Judge	none	Claude Sonnet, deterministic (temp=0)
Dashboard	none	LangSmith UI with side-by-side diffs

After the sprint ends, the team has a running trace history, a 100-row prod-derived dataset, a pinned baseline experiment, and a gate_prompt_change() function anyone can call before proposing a prompt PR. The vibes-based prompt shipping cycle is over; a prompt change that doesn't win 60% of pairwise comparisons doesn't merge.

Output:

Sprint deliverable	Concrete artifact
Tracing on	LANGSMITH_TRACING=true in every service env
First dataset	support_rag_from_prod_v1 (100 rows)
Baseline experiment	baseline_prompt_v1 (immutable)
Pairwise gate function	gate_prompt_change(new_version)
CI hook	.github/workflows/prompt-gate.yml calls the function
Regression tag	metadata.tag = "regression" on 3 rows
Cost per gate run	~$0.85 (200 chain calls + 100 judge calls)

Why this works — concept by concept:

• @traceable + LangChain auto-instrumentation — one env var turns every LLM call in the app into a Run in a Trace, capturing latency, tokens, cost, inputs, outputs, and error stacks for free. This is the highest-leverage single change any LLM-app team can make.
• Production-derived dataset bootstrap — the 100 lowest-confidence prod traces from last week are the perfect seed dataset because they represent actual user friction, not synthetic questions. Ground-truth answers are filled in later by humans; the raw questions are already load-bearing.
• Pinned experiment as baseline — pinning baseline_prompt_v1 against dataset support_rag_from_prod_v1 makes every future comparison anchor-referenced. Anyone can rerun the baseline months later against the same dataset version and get the same score.
• Pairwise judge with temperature 0 — pairwise judges are more sensitive than pointwise judges to small quality deltas because the judge is asked to compare, not to score in an absolute frame. Temperature 0 makes the score deterministic so a re-run yields the same verdict.
• Cost — one gate run = 200 chain calls (100 rows × 2 candidates) + 100 judge calls ≈ $0.85. Compared to shipping a bad prompt to production for a week (support tickets, refunds, reputation), this is the cheapest quality gate you will ever ship. O(dataset_size) per gate; O(1) latency to detect regressions.

ETL
Topic — etl
ETL problems on trace-instrumented pipelines

Practice →

Python Topic — python Python problems on decorator patterns and observability

Practice →

---

3. TruLens — feedback functions and the RAG triad

Feedback functions score every Record on the three edges of the RAG triad — local-first, open, and framework-agnostic

The mental model in one line: trulens is the open-source, local-first eval library where you wrap your app in TruChain / TruLlama / TruCustomApp, attach Feedback functions that score every Record (one Record = one user query + its outputs), and lean on the canonical RAG triad — context relevance (query ↔ context), groundedness (context ↔ answer), and answer relevance (query ↔ answer) — as the three-edged completeness proof that no single-axis eval can match, and the entire eval history lives in a local SQLite (or your own Postgres) with no vendor round-trip required. TruLens's sweet spot is teams that need eval inside the VPC, want to instrument non-LangChain code, or prefer an unopinionated Feedback function primitive over a hosted platform's canned evaluators.

The four axes for TruLens.

• Groundedness. The classic Feedback(provider.groundedness_measure_with_cot_reasons) — split the answer into claims, ask an LLM judge whether each claim is supported by the context, aggregate. TruLens ships this as a named feedback function; you plug in the provider (OpenAI, Anthropic, Bedrock, local via LiteLLM) and it just works.
• Answer relevance. Feedback(provider.relevance) on (query, answer). Same LLM-judge pattern as Ragas's answer_relevancy but the prompt is TruLens's own and the API surface is Feedback object composition.
• Context precision / recall. Slightly indirect — TruLens exposes "context relevance" per-chunk (Feedback(provider.context_relevance_with_cot_reasons).on_input().on(context)) which is essentially per-chunk precision. Aggregate across chunks and you get context precision; recall requires labelled ground-truth context IDs.
• Latency + cost. TruLens records per-Record wall-clock and token counts natively. record.latency and record.cost come out of the box; you can pipe them into your dashboard of choice or query the local SQLite directly.

The Feedback function primitive.

• Anatomy. Feedback(provider.metric_fn, name="...").on_input().on_output() — a Feedback binds a callable (usually an LLM-judge call), a name, and selectors that pull the right fields off the Record.
• Selectors. .on_input() pulls the user query; .on_output() pulls the app's final answer; .on(Select.RecordCalls.retriever.rets) pulls the retriever's output. Selectors are the composable glue between the app trace and the feedback function.
• Aggregation. By default, feedback returns a scalar. For per-chunk metrics like context_relevance, aggregation strategies include np.mean, np.min, np.max, or a custom callable.
• Providers. LiteLLM, OpenAI, Anthropic, Bedrock, Huggingface — any provider with a chat completion API can be the judge. LiteLLM is the local-first choice because it wraps every provider under one API.

App vs Session vs Record scope.

• Record. One user query → one Record. Feedback runs per Record. Most metrics live here.
• Session. A sequence of Records tied to a user session (multi-turn chat). Session-scoped feedback measures things like coherence across turns.
• App. The app-level aggregate (all Records for one app version). App-scoped feedback is where you compute p95 latency, aggregate faithfulness, etc.
• App version. TruChain(chain, app_name="rag", app_version="v2.3") — every deploy bumps the version; the dashboard slices by version so a bad deploy shows up as a scored-lower app version.

Local-first — the offline story.

• Storage. Default backend is SQLite at default.sqlite. Every Record + every Feedback score lives there. Tru().get_records_and_feedback() returns a pandas DataFrame you can query, join, or export.
• Dashboard. Tru().run_dashboard() starts a Streamlit UI on localhost with per-Record drill-down, feedback distributions, and app-version diffs. No cloud round-trip.
• VPC deploy. For team use, swap SQLite for Postgres (Tru(database_url="postgresql://...")) — the schema is one-command migratable and the dashboard reads Postgres directly. Everything stays in your VPC.

Common interview probes on TruLens.

• "What's the RAG triad?" — required answer: context relevance, groundedness, answer relevance — the three edges of a triangle whose vertices are query, context, and answer.
• "Where does TruLens store its eval history?" — required answer: local SQLite by default, Postgres in production; no vendor round-trip.
• "How is Feedback different from LangSmith's evaluator?" — TruLens Feedback is a Python callable + selector composition, framework-agnostic; LangSmith evaluators are cloud-hosted primitives tied to Runs.
• "Why does groundedness split into claims?" — required answer: to score each atomic claim independently, otherwise a single unsupported sentence in a paragraph is invisible.

Worked example — instrument a RAG chain with TruChain and run the triad

Detailed explanation. The canonical TruLens setup: wrap an existing LangChain RAG in TruChain, define three Feedback functions (the triad), and let every subsequent chain.invoke produce a scored Record. Walk through the setup end-to-end.

• Chain. LangChain RetrievalQA with FAISS + Claude Sonnet.
• Provider. LiteLLM pointed at Claude Haiku (cheap judge).
• Feedbacks. Context relevance (per-chunk, mean-aggregated), groundedness, answer relevance.
• Storage. Default SQLite.

Question. Wire up TruChain around a LangChain RAG and produce the RAG triad scores for every query.

Input.

Component	Object
App	LangChain RetrievalQA
Wrapper	TruChain(app_name="support-rag", app_version="v2.3")
Provider	LiteLLM(model_engine="claude-3-5-haiku-20241022")
Feedbacks	f_context_relevance, f_groundedness, f_answer_relevance
Backend	SQLite (default.sqlite)

Code.

# trulens_rag.py — instrument a LangChain RAG with the RAG triad
import numpy as np
from trulens.core import TruSession, Feedback, Select
from trulens.apps.langchain import TruChain
from trulens.providers.litellm import LiteLLM
from langchain.chains import RetrievalQA
from langchain_anthropic import ChatAnthropic
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

# 1. Build the base RAG chain (unchanged from prod)
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vs         = FAISS.load_local("faiss_index", embeddings, allow_dangerous_deserialization=True)
retriever  = vs.as_retriever(search_kwargs={"k": 8})
llm        = ChatAnthropic(model="claude-sonnet-4-7", temperature=0.1)
rag_chain  = RetrievalQA.from_chain_type(llm=llm, retriever=retriever,
                                          return_source_documents=True)

# 2. Start TruLens session + provider
session  = TruSession()
provider = LiteLLM(model_engine="claude-3-5-haiku-20241022")

# 3. Define the RAG triad
context_selector = Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content

f_context_relevance = (
    Feedback(provider.context_relevance_with_cot_reasons,
             name="Context Relevance")
    .on_input()
    .on(context_selector)
    .aggregate(np.mean)
)

f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons,
             name="Groundedness")
    .on(context_selector.collect())
    .on_output()
)

f_answer_relevance = (
    Feedback(provider.relevance_with_cot_reasons,
             name="Answer Relevance")
    .on_input()
    .on_output()
)

# 4. Wrap the chain
tru_chain = TruChain(
    rag_chain,
    app_name="support-rag",
    app_version="v2.3",
    feedbacks=[f_context_relevance, f_groundedness, f_answer_relevance],
)

# 5. Every call is now scored
with tru_chain as recording:
    resp = rag_chain.invoke({"query": "How do I rotate my API key without downtime?"})

record = recording.records[-1]
print(f"latency: {record.latency:.2f}s, cost: ${record.cost:.4f}")
print("scores:", {k: v.result for k, v in record.feedback_and_future_results.items()})

# 6. Fetch the full history as a DataFrame
df = session.get_records_and_feedback(app_ids=[tru_chain.app_id])[0]
print(df[["input", "output", "Context Relevance", "Groundedness", "Answer Relevance"]].tail())

Step-by-step explanation.

1. The base LangChain RAG is unchanged from production — TruLens is a wrapper, not a replacement. This is TruLens's key adoption story: you don't rewrite your app to add eval.
2. The TruSession() singleton is the entry point; by default it uses SQLite (default.sqlite) in the current directory. Swap for Postgres by passing database_url= at construction. Every subsequent TruChain, TruLlama, or TruCustomApp registers with the session.
3. The context_selector is a TruLens selector expression — Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content navigates the recorded trace: pull the retriever call, get all returned documents, extract page_content from each. Selectors are the composable primitive that makes TruLens framework-agnostic.
4. The three Feedback functions are the RAG triad. Note how each uses different selectors: f_context_relevance scores per-chunk (mean-aggregated), f_groundedness collects all chunks as one context blob then scores the answer against it, f_answer_relevance compares query directly to answer.
5. The with tru_chain as recording context manager captures every rag_chain.invoke inside as a Record. recording.records is the list of Records made in the block; each has .latency, .cost, and .feedback_and_future_results. The dataframe view at the end is what you'd surface in a dashboard.

Output.

Query	Context Relevance	Groundedness	Answer Relevance	Latency	Cost
"How do I rotate my API key?"	0.86	0.92	0.94	2.1 s	$0.014
"What's the SLA at 3am ET?"	0.71	0.68	0.90	2.3 s	$0.015
"Why can't I export PDF?"	0.55	0.44	0.82	2.4 s	$0.016
Overall (mean)	0.71	0.68	0.89	2.27 s	$0.015

Rule of thumb. Wrap first, evaluate second. Get TruChain around the existing app before you fine-tune Feedback functions. The first useful signal comes from just seeing per-Record latencies + costs in the SQLite; the triad scores are the second layer.

Worked example — per-chunk context relevance and the drill-down that saves your retriever

Detailed explanation. The RAG triad scores at the Record level are useful, but the real diagnostic power of TruLens comes from per-chunk context relevance. When a query scores low on context relevance, the drill-down reveals which chunks are irrelevant, which points directly at retriever tuning: switch to hybrid retrieval, increase k, add a reranker, or fix the chunking strategy. Walk through the diagnosis flow for a low-scoring query.

• Low-scoring query. "Why can't I export PDF from a shared workspace?" — context relevance 0.55.
• Per-chunk drill-down. Select.RecordCalls.retriever.get_relevant_documents.rets[:] — inspect each chunk's individual score.
• Diagnosis. 6 of 8 chunks are billing-related; 2 are the actual "role-based export" chunks. Retriever is over-weighting a common bigram.
• Fix. Switch from pure-vector to hybrid (BM25 + vector) retrieval; add a reranker.

Question. Extract per-chunk context relevance for the failing query and use it to diagnose the retriever problem.

Input.

Chunk #	Source	Vector score	Context relevance score
1	billing-policy.md	0.88	0.12
2	billing-policy.md	0.86	0.10
3	shared-workspaces.md#export	0.84	0.94
4	payment-methods.md	0.83	0.08
5	export-formats.md#roles	0.82	0.91
6	billing-policy.md	0.81	0.11
7	account-limits.md	0.80	0.15
8	billing-history.md	0.79	0.09

Code.

# per_chunk_context_relevance.py
import numpy as np
from trulens.core import Feedback, Select
from trulens.providers.litellm import LiteLLM

provider = LiteLLM(model_engine="claude-3-5-haiku-20241022")

# Per-chunk context relevance — no aggregate; keep the vector of scores
f_context_relevance_per_chunk = (
    Feedback(provider.context_relevance_with_cot_reasons,
             name="Context Relevance (per chunk)")
    .on_input()
    .on(Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content)
    # NOTE: no .aggregate() — returns per-chunk scores as a list
)

# After running the query, drill down
record = recording.records[-1]
per_chunk = record.feedback_and_future_results["Context Relevance (per chunk)"].result

# Zip with retrieved sources for diagnosis
docs = record.calls[-1].rets  # the retriever's returned docs
diag = [
    {"source": d.metadata["source_url"], "vector": d.metadata["score"], "rel": s}
    for d, s in zip(docs, per_chunk)
]

# Print the diagnostic table
for d in sorted(diag, key=lambda x: -x["rel"]):
    print(f"{d['rel']:.2f}  {d['vector']:.2f}  {d['source']}")

# Diagnosis rule: if > 50% of top-k chunks score < 0.30 on context relevance,
# the retriever is over-weighting a common bigram
low = sum(1 for d in diag if d["rel"] < 0.30)
if low / len(diag) > 0.5:
    print(f"RETRIEVER PROBLEM: {low}/{len(diag)} chunks below 0.30 relevance")
    print("Recommend: switch to hybrid retrieval + reranker")

Step-by-step explanation.

1. Removing the .aggregate() call on the Feedback function is what turns a scalar score into a per-chunk vector. TruLens automatically iterates over the selector's returned list and applies the callable to each element.
2. The per-chunk table reveals the pathology: chunks 1, 2, 6 are all billing-policy.md, all score above 0.79 on the vector distance, but all score below 0.15 on context relevance. The vector retriever thinks these chunks match; the LLM judge thinks they don't. This is a classic "vector similarity captures surface bigrams" failure mode.
3. Chunks 3 and 5 (from shared-workspaces.md and export-formats.md) are the actually relevant chunks — high context relevance (0.94, 0.91) and moderate vector score (0.84, 0.82). They're in the top-8 but drowned out by the noise.
4. The rule of thumb — "if > 50% of top-k chunks score < 0.30 on context relevance, the retriever needs help" — is the operational signal that triggers a retriever change. Options: (a) hybrid BM25 + vector rerank; (b) add a cross-encoder reranker; (c) increase top-k and re-rank; (d) fix the chunking strategy (chunks 1/2/6 might be over-chunked from one long policy page).
5. After switching to hybrid retrieval + a bge-reranker-large reranker, this query's context relevance jumps from 0.55 to 0.87; the two relevant chunks now sit at ranks 1 and 2. The Record-level triad scores rise across the board.

Output.

Fix applied	Context Relevance (mean)	Faithfulness	Answer Relevance
baseline (pure vector, k=8)	0.55	0.44	0.82
hybrid BM25 + vector	0.72	0.68	0.86
hybrid + cross-encoder rerank	0.87	0.89	0.93

Rule of thumb. Any Record with context relevance below 0.60 should trigger a per-chunk drill-down. The per-chunk vector is the retriever's report card; scalar aggregates hide the pathology. Wire the drill-down as a Slack alert on the first N regressions.

Senior interview question on TruLens triad and production scoring

A senior interviewer might ask: "You've got a support RAG in production, no eval, running inside a bank's VPC — no data can leave the perimeter. Wire up TruLens with the RAG triad, run it locally against a golden set, then deploy a sampled production-time scoring plus a nightly batch back-fill without adding latency to 98% of traffic."

Solution Using TruLens RAG triad locally, sampled sync feedback + nightly batch back-fill

# trulens_vpc_prod.py — VPC-friendly TruLens deployment
import os, random, numpy as np
from trulens.core import TruSession, Feedback, Select
from trulens.apps.langchain import TruChain
from trulens.providers.litellm import LiteLLM

# Postgres inside VPC; Bedrock judge inside AWS — zero external egress
session = TruSession(database_url=os.environ["TRULENS_DATABASE_URL"])
provider = LiteLLM(model_engine="bedrock/anthropic.claude-3-5-haiku-20241022-v1:0")

from myapp.rag import make_chain
rag_chain = make_chain()

CTX = Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content

f_ctx = (Feedback(provider.context_relevance_with_cot_reasons, name="Context Relevance")
         .on_input().on(CTX).aggregate(np.mean))
f_grd = (Feedback(provider.groundedness_measure_with_cot_reasons, name="Groundedness")
         .on(CTX.collect()).on_output())
f_ans = (Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance")
         .on_input().on_output())

tru_chain = TruChain(rag_chain, app_name="support-rag",
                     app_version=os.environ.get("APP_VERSION", "v2.3"),
                     feedbacks=[f_ctx, f_grd, f_ans])

# 1. Local golden-set run (bootstrap eval)
def run_golden_set():
    import json; from pathlib import Path
    with tru_chain as _:
        for row in [json.loads(l) for l in Path("eval/golden.jsonl").read_text().splitlines()]:
            rag_chain.invoke({"query": row["question"]})
    df, _ = session.get_records_and_feedback(app_ids=[tru_chain.app_id])
    print(df[["Context Relevance", "Groundedness", "Answer Relevance"]].mean())

# 2. Production endpoint: 2% sync scoring, 98% record-only
SAMPLE_RATE = 0.02

def prod_answer(question: str) -> dict:
    ctx = tru_chain if random.random() < SAMPLE_RATE else tru_chain.record_only()
    with ctx as rec:
        resp = rag_chain.invoke({"query": question})
    mode = "sync" if ctx is tru_chain else "deferred"
    return {"answer": resp["result"], "feedback_mode": mode,
            "record_id": rec.records[-1].record_id}

# 3. Nightly back-fill for unscored Records
def nightly_backfill():
    df, _ = session.get_records_and_feedback(app_ids=[tru_chain.app_id])
    for _, row in df[df["Groundedness"].isna()].iterrows():
        record = session.get_record(row["record_id"])
        for f in [f_ctx, f_grd, f_ans]:
            session.add_feedback(feedback_result=f.run(record=record))

Step-by-step trace.

Layer	Component	Purpose
Storage	Postgres in VPC	zero-egress persistence
Provider	LiteLLM → Bedrock Claude Haiku	judge stays inside AWS
App wrapper	TruChain	captures Records for the LangChain RAG
Feedbacks	triad (context, groundedness, answer)	three-edge completeness
Sync path	2% of prod requests	real-time alerts
Async path	98% of prod requests	zero added latency
Nightly back-fill	batch judge calls	full coverage within 24h

After deployment, the RAG's every prod query is stored as a Record in Postgres, 2% of Records get scored instantly with the triad, and a nightly Airflow DAG back-fills the remaining 98% using batch LLM calls. The bank's compliance team sees no data leaving the VPC; the product team sees a real-time dashboard of triad scores per app version; a bad deploy shows up as a scored-lower app version within an hour.

Output:

Metric	Value
Records stored / day	600,000
Sync-scored Records / day	12,000
Batch-scored Records / day	588,000
Feedback cost / day (batch discount)	~$4,580
Data egress from VPC	0 bytes
Time from bad deploy to alert	≤ 15 min (on sync tail)
Storage in Postgres / day	~600 MB

Why this works — concept by concept:

• RAG triad completeness — the three edges (query↔context, context↔answer, query↔answer) form a triangle; a failure on any edge is a distinct failure mode. Context-relevance failure = bad retriever; groundedness failure = hallucination; answer-relevance failure = off-topic answer. Measuring only one edge lets the other two regress invisibly.
• Postgres backend inside VPC — TruLens's SQLAlchemy backend is swappable; pointing it at an in-VPC Postgres keeps every Record and every score inside the bank's perimeter. No data crosses a vendor boundary at any point.
• LiteLLM + Bedrock judge — the judge LLM also stays inside AWS (Bedrock is region-local). LiteLLM's unified provider API means switching between Bedrock, self-hosted vLLM, or Azure OpenAI is a one-line change with the rest of the eval code unchanged.
• 2% sync + 98% async — the sampled-sync tier gives real-time alerts (a bad deploy shows up in 12,000 scored Records within 15 minutes), while the async tier back-fills the aggregate view without adding latency to 98% of user requests. Coverage is 100% within 24 hours.
• Cost — 2% × $0.015 sync + 98% × $0.007 (batch discount) per Record ≈ $4,580/day for 600k queries, versus $9,000/day if all 100% went sync. Compared to shipping a bad model without any triad scoring, the alert-lead-time saving alone is worth 10× the eval cost. O(1) latency for 98% of prod requests; O(1) latency to detect regressions.

Python
Topic — python
Python problems on wrapper and observability patterns

Practice →

Design Topic — design Design problems on VPC-first eval systems

Practice →

---

4. Ragas — reference-free RAG metrics for CI

Reference-free metrics (faithfulness, answer_relevancy) plus reference-based (context_precision, context_recall) — one evaluate() call, one scored DataFrame

The mental model in one line: ragas is the metric library that turns rag evaluation into a one-line evaluate(dataset, metrics=[...]) call returning a pandas DataFrame of scored rows — with two metrics (faithfulness, answer_relevancy) that need no reference answer and two metrics (context_precision, context_recall) that use a labelled ground-truth, all judged by any LLM you configure via LiteLLM — and its sweet spot is the batch CI gate where a nightly Airflow job scores a fixed golden set, uploads the DataFrame, and a GitHub Action fails the PR when any metric regresses beyond a threshold. Ragas is the least opinionated of the four tools: it doesn't wrap your app, it doesn't own a dashboard, it just scores.

The four axes for Ragas.

• Groundedness. faithfulness — split the answer into atomic claims, ask the LLM judge whether each claim is entailed by the retrieved contexts, aggregate. Same intent as TruLens's groundedness; different prompt.
• Answer relevance. answer_relevancy — generate 3 hypothetical questions the answer could address, compute cosine similarity between each and the actual query, aggregate. Requires an embedding model in addition to the judge LLM.
• Context precision / recall. context_precision — for each retrieved chunk, is it relevant to the question given the ground-truth answer? Ranked precision. context_recall — decompose the ground-truth answer into statements; check which are covered by the retrieved contexts. Both require a labelled ground truth.
• Latency + cost. Not measured by Ragas itself. Wire LangSmith or TruLens for those axes, or compute them in your own wrapper. Ragas is metric-only.

Reference-free vs reference-based — the split that matters.

• Reference-free: faithfulness, answer_relevancy. Need only (question, answer, contexts). Cheap to curate the dataset — you don't need a golden answer written by a human. Perfect for bootstrapping eval when you have zero ground-truth data.
• Reference-based: context_precision, context_recall, answer_correctness, answer_similarity. Need (question, answer, contexts, ground_truth). Ground truth costs human time to curate; budget 5-10 minutes per row.
• Bootstrap pattern. Start with reference-free (day one). Add reference-based as you accumulate labelled rows (weeks in). Ragas's DataFrame skips reference-based metrics gracefully if ground_truth is missing.

The EvaluationDataset and evaluate() shape.

• Row shape. {question, answer, contexts: list[str], ground_truth?} per row. Contexts is a list of chunk strings; ground_truth is optional.
• Dataset. Dataset.from_list(rows) (HuggingFace datasets.Dataset); or Ragas's own EvaluationDataset in v0.2+. Both interoperate.
• evaluate. evaluate(dataset, metrics=[faithfulness, answer_relevancy, ...], llm=..., embeddings=...). Returns a Result with .to_pandas() giving per-row scores.
• LLM + embeddings. Any LiteLLM-compatible LLM as judge; any HF or provider embeddings for answer_relevancy. Common: Claude Haiku judge + OpenAI text-embedding-3-large.

Common interview probes on Ragas.

• "What's reference-free vs reference-based?" — required answer: reference-free needs no ground-truth answer; reference-based does.
• "Which Ragas metrics need ground truth?" — context_precision, context_recall, answer_correctness.
• "How is faithfulness computed?" — split answer into claims; judge each against contexts; aggregate.
• "How would you gate a PR on Ragas?" — nightly Airflow → evaluate() → JSON → GitHub Action compares to thresholds.

Worked example — score a golden set with the four Ragas metrics

Detailed explanation. The canonical Ragas workflow: a golden .jsonl file of (question, answer, contexts, ground_truth) rows, one evaluate() call returning a scored DataFrame, threshold comparison, JSON dump. Walk through the setup.

• Golden set. 100 rows curated by the product manager + support team.
• Metrics. All four.
• Judge. Claude Haiku via LiteLLM.
• Embeddings. OpenAI text-embedding-3-large.

Question. Score the golden set with the four Ragas metrics and emit a JSON report suitable for CI.

Input.

Column	Type	Required for
question	str	all metrics
answer	str	all metrics
contexts	list[str]	all metrics
ground_truth	str	context_precision, context_recall

Code.

# ragas_gate.py
import json
from pathlib import Path
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness, answer_relevancy, context_precision, context_recall,
)
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_anthropic import ChatAnthropic
from langchain_openai import OpenAIEmbeddings

# 1. Load the golden set
rows = [json.loads(l) for l in Path("eval/golden_v3.jsonl").read_text().splitlines()]

# 2. Run the pipeline on each question, populating `answer` and `contexts`
from myapp.rag import rag_chain
for r in rows:
    resp = rag_chain.invoke({"query": r["question"]})
    r["answer"]   = resp["result"]
    r["contexts"] = [d.page_content for d in resp["source_documents"]]

ds = Dataset.from_list(rows)

# 3. Score with Ragas
judge      = LangchainLLMWrapper(ChatAnthropic(model="claude-3-5-haiku-20241022", temperature=0))
embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-large"))

result = evaluate(
    ds,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    llm=judge,
    embeddings=embeddings,
)

# 4. Aggregate and emit
df    = result.to_pandas()
means = df[[m.name for m in [faithfulness, answer_relevancy, context_precision, context_recall]]].mean()
report = {"means": means.to_dict(), "rows": len(df)}
Path("eval/report.json").write_text(json.dumps(report, indent=2))

THRESHOLDS = {"faithfulness": 0.85, "answer_relevancy": 0.80,
              "context_precision": 0.70, "context_recall": 0.75}
failures = [f"{k}={v:.3f} < {THRESHOLDS[k]}" for k, v in means.items() if v < THRESHOLDS[k]]

if failures:
    print("RAGAS GATE FAILED:", failures)
    raise SystemExit(1)
print("RAGAS GATE PASSED:", means.to_dict())

Step-by-step explanation.

1. The golden .jsonl starts with just (question, ground_truth) per row; the pipeline call in step 2 populates answer and contexts on the fly. This means the eval is scoring the current pipeline, not a stale snapshot.
2. LangchainLLMWrapper and LangchainEmbeddingsWrapper adapt any LangChain LLM/embeddings object to Ragas's provider interface. Any LiteLLM-supported model works; picking Haiku for the judge keeps the per-row judge cost around $0.003.
3. evaluate() runs each metric against each row in parallel (Ragas uses async under the hood). For 100 rows × 4 metrics, expect ~2 minutes wall-clock and ~$1.60 in judge + embedding calls.
4. The to_pandas() DataFrame has one column per metric, one row per input example, plus the original inputs. Aggregating with .mean() gives you the scalar scores for the gate; retaining the per-row DataFrame lets you drill down to the low-scoring rows.
5. The gate raises SystemExit(1) on any threshold failure — that's how Airflow / GitHub Actions detect the failure. The report JSON is uploaded as a workflow artifact so the PR comment bot can render it inline.

Output.

Metric	Score	Threshold	Pass?
faithfulness	0.87	0.85	yes
answer_relevancy	0.83	0.80	yes
context_precision	0.71	0.70	yes
context_recall	0.79	0.75	yes

Rule of thumb. For a first Ragas gate, ship reference-free metrics only (faithfulness, answer_relevancy) so you don't need labelled ground truth on day one. Add context_precision / context_recall in week two once you've labelled 30-50 rows. Never gate on a metric whose baseline you haven't measured for two weeks.

Worked example — nightly Airflow DAG + GitHub PR gate

Detailed explanation. The production shape of a Ragas gate is a nightly Airflow DAG (for main-branch trend) plus a per-PR GitHub Action (for the blocking gate). Walk through both.

• Nightly DAG. Runs ragas_gate.py against main; stores the JSON in S3; emits Slack alert on regression.
• Per-PR gate. GitHub Action runs the same ragas_gate.py against the PR branch; posts scores as a PR comment; fails the check on regression.

Question. Wire the nightly Airflow DAG and the per-PR GitHub Action, sharing the same ragas_gate.py.

Input.

Environment	Trigger	Behaviour on failure
Nightly Airflow	cron 03:00 UTC	Slack alert + preserve trend
Per-PR GitHub Action	pull_request on prompts/, src/rag/	Fail check + comment scores

Code.

# dags/ragas_nightly.py — Airflow DAG
from datetime import datetime, timedelta
from airflow.decorators import dag, task

DEFAULTS = {"owner": "mlops", "retries": 0, "retry_delay": timedelta(minutes=5)}

@dag(dag_id="ragas_nightly",
     start_date=datetime(2026, 7, 1),
     schedule="0 3 * * *",
     catchup=False,
     default_args=DEFAULTS)
def ragas_nightly():
    @task
    def score():
        import subprocess, json
        rc = subprocess.call(["python", "/opt/eval/ragas_gate.py"])
        report = json.loads(open("/opt/eval/eval/report.json").read())
        return {"rc": rc, "report": report}

    @task
    def alert(payload):
        import requests, os
        if payload["rc"] == 0:
            return
        requests.post(os.environ["SLACK_WEBHOOK"], json={
            "text": f":rotating_light: Nightly Ragas gate failed\n```
{% endraw %}
{json.dumps(payload['report'], indent=2)}
{% raw %}
```"
        })

    @task
    def archive(payload):
        import boto3, json, datetime as dt
        s3 = boto3.client("s3")
        key = f"ragas/{dt.date.today():%Y/%m/%d}/report.json"
        s3.put_object(Bucket="mlops-eval",
                       Key=key,
                       Body=json.dumps(payload["report"]).encode())

    p = score()
    alert(p)
    archive(p)

ragas_nightly()

# .github/workflows/ragas-gate.yml
name: Ragas Gate
on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/rag/**"
      - "eval/**"

jobs:
  ragas:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - name: Run Ragas gate
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          OPENAI_API_KEY:    ${{ secrets.OPENAI_API_KEY }}
        run: python eval/ragas_gate.py
      - name: Comment scores on PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require("fs");
            let body = "## Ragas gate\n";
            try {
              const rpt = JSON.parse(fs.readFileSync("eval/report.json"));
              body += "| metric | score |\n|---|---|\n";
              for (const [k, v] of Object.entries(rpt.means))
                body += `| ${k} | ${v.toFixed(3)} |\n`;
            } catch (e) { body += "_no report produced_"; }
            github.rest.issues.createComment({
              owner: context.repo.owner, repo: context.repo.repo,
              issue_number: context.issue.number, body
            });
      - name: Upload report artifact
        if: always()
        uses: actions/upload-artifact@v4
        with: { name: ragas-report, path: eval/report.json }

Step-by-step explanation.

1. Both entry points call the same ragas_gate.py. Sharing the entry point means the nightly trend and the per-PR gate can never diverge in what they measure. Any threshold change is a code review against one file.
2. The Airflow DAG's score task shells out to the Python script and captures the return code plus the JSON report. alert fires the Slack webhook on failure; archive writes the daily report to S3 at ragas/YYYY/MM/DD/report.json — the historical trend lives in S3, queryable via Athena.
3. The GitHub Action gates on pull_request events touching prompts, RAG source, or eval configs. paths: filters keep the gate cheap — it doesn't run on markdown-only PRs. if: always() on the comment step ensures the PR gets scored even when the gate fails.
4. The comment step uses actions/github-script@v7 to render a markdown table of scores. On failure, the PR shows: red X check, comment table with scores, and a workflow artifact holding the raw report.
5. The 15-minute timeout-minutes guardrails against runaway judge calls (e.g. rate-limit backoffs). If a judge call storms happen, the job fails fast; the gate is treated as "unable to evaluate" and the human reviewer decides whether to re-run or override.

Output.

Scenario	Nightly DAG	PR gate
Baseline (all metrics pass)	archive JSON; no alert	green check; scores as comment
Faithfulness regression on main	Slack alert to #llm-eval	n/a
Faithfulness regression on PR	n/a	red X; PR blocked; scores as comment
Judge API rate-limited	retry once, then Slack alert	red X; artifact = partial report

Rule of thumb. Ship the nightly DAG before the per-PR gate. Two weeks of nightly baselines tell you where the thresholds should sit; setting thresholds on day one is guessing. Once the trend is stable, wire the per-PR gate at the observed p10 (the 10th percentile of the last 14 days).

Senior interview question on Ragas as the CI gate

A senior interviewer might ask: "Design a per-PR Ragas gate for a RAG chain. The gate must fail the PR on regression across four metrics, comment scores back on the PR, and archive results daily. Include a custom brand-voice metric. What thresholds do you set on day one?"

Solution Using Ragas evaluate + custom metric + GitHub Action + nightly Airflow

# ragas_full_gate.py — CI gate + custom metric + report emitter
import json, os, sys
from pathlib import Path
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (faithfulness, answer_relevancy,
                            context_precision, context_recall)
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_anthropic import ChatAnthropic
from langchain_openai import OpenAIEmbeddings
from custom_metric import BrandVoiceCompliance   # from previous example

GOLDEN = Path("eval/golden_v3.jsonl")
THRESHOLDS = {"faithfulness": 0.85, "answer_relevancy": 0.80,
              "context_precision": 0.70, "context_recall": 0.75,
              "brand_voice_compliance": 0.75}

def main():
    from myapp.rag import rag_chain
    rows = [json.loads(l) for l in GOLDEN.read_text().splitlines()]
    for r in rows:
        resp = rag_chain.invoke({"query": r["question"]})
        r["answer"]   = resp["result"]
        r["contexts"] = [d.page_content for d in resp["source_documents"]]

    judge = LangchainLLMWrapper(ChatAnthropic(model="claude-3-5-haiku-20241022", temperature=0))
    emb   = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-large"))

    result = evaluate(Dataset.from_list(rows),
                      metrics=[faithfulness, answer_relevancy, context_precision,
                                context_recall, BrandVoiceCompliance()],
                      llm=judge, embeddings=emb)
    means = result.to_pandas()[list(THRESHOLDS)].mean().to_dict()

    Path("eval/report.json").write_text(json.dumps(
        {"means": means, "rows": len(rows),
         "commit": os.environ.get("GITHUB_SHA", "local")}, indent=2))

    failures = [f"{k}={v:.3f}<{THRESHOLDS[k]}"
                for k, v in means.items() if v < THRESHOLDS[k]]
    if failures:
        print("GATE FAILED:", failures); sys.exit(1)
    print("GATE PASSED:", means)

if __name__ == "__main__":
    main()

Step-by-step trace.

Layer	Component	Purpose
Dataset	eval/golden_v3.jsonl	100 curated (question, ground_truth)
Pipeline runner	run_pipeline()	populate answer + contexts from prod code
Metrics	5 (4 Ragas + 1 custom)	faithfulness + answer_relevancy + context_precision + context_recall + brand_voice
Judge	Claude Haiku, temp=0	deterministic scoring
Embeddings	OpenAI text-embedding-3-large	for answer_relevancy
Gate	sys.exit(1) on any regression	CI-friendly failure
Comment	GitHub Actions bot	markdown table on PR

After deployment, the gate blocks any PR that touches prompts/** or src/rag/** if any of the five metrics drops below its threshold on the 100-row golden set. The nightly Airflow DAG runs the same script against main, alerts on regression, and archives the JSON to S3. Custom metrics live in myapp.eval.metrics and are reviewed like any other code.

Output:

Deliverable	Location
Gate script	eval/ragas_full_gate.py
Custom metrics module	myapp/eval/metrics/brand_voice.py
PR gate workflow	.github/workflows/ragas-gate.yml
Nightly DAG	dags/ragas_nightly.py
Archived reports	s3://mlops-eval/ragas/YYYY/MM/DD/report.json
Threshold config	eval/ragas_full_gate.py (code-reviewed)
Slack alert channel	#llm-eval

Why this works — concept by concept:

• Reference-free + reference-based split — faithfulness and answer_relevancy bootstrap the gate without needing labelled ground truth; context_precision, context_recall, and custom metrics harden it as labels arrive. This lets the gate ship on day one with two axes and grow to five as the dataset matures.
• PydanticPrompt custom metric — extending Ragas with a domain-specific metric is a first-class pattern; the framework handles async, batching, retries, and DataFrame integration. Rolling your own scorer means reimplementing all four; using MetricWithLLM means writing one method.
• One entry point for CI and nightly — sharing ragas_full_gate.py between the PR gate and the Airflow DAG guarantees the two can never diverge. Threshold changes are one file, one code review.
• JSON report + GitHub bot comment — the eval score becomes a first-class artifact on the PR discussion. Reviewers see the numbers inline; no dashboard round-trip; failures are self-explanatory.
• Cost — 100 rows × 5 metrics × ~$0.003 judge cost ≈ $1.50 per PR gate run. Compared to shipping a bad prompt without a gate, this is a rounding error. O(dataset_size × num_metrics) per run; O(1) latency to detect regressions.

ETL
Topic — etl
ETL problems on batch quality gates

Practice →

Data validation Topic — data-validation Data validation problems on threshold gating

Practice →

---

5. Snowflake Cortex Search Ops — eval at the semantic-search layer

CORTEX SEARCH SERVICE + AI_COMPLETE judges + recall@k in SQL — data, index, model, and eval inside one governance boundary

The mental model in one line: snowflake cortex search is Snowflake's managed hybrid (BM25 + vector) semantic-search primitive, CORTEX SEARCH SERVICE is the DDL that turns a table of text into a searchable index, AI_COMPLETE('claude-3-5-sonnet', ...) is the SQL-level LLM you can use both to generate answers and to judge them, and Cortex Search Ops is the pattern of running eval — recall@k, MRR, nDCG, groundedness — as plain SQL over a labelled (query, expected_doc_id) table so the entire data + index + model + eval stack sits inside one Snowflake account and one governance boundary. This is the eval-in-SQL story that fits banks, healthcare, and government where data-egress is a compliance loss.

The four axes for Cortex Search Ops.

• Groundedness. Compute in SQL via AI_COMPLETE('claude-3-5-sonnet', 'Rate 0-1 whether this answer is grounded in the retrieved context: ...'). The judge call is one function invocation; parse the numeric response.
• Answer relevance. Same pattern — AI_COMPLETE with a "does the answer address the question?" prompt. Cortex handles the model round-trip inside the account; no external API needed.
• Context precision / recall. Pure IR SQL. Label (query, expected_doc_id) pairs in a table; run the search; compute recall@k, MRR, nDCG@k with aggregate SQL. This is the classic search-eval pattern reborn.
• Latency + cost. Query history table (SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY) captures every CORTEX SEARCH_PREVIEW and every AI_COMPLETE call with wall-clock and credits burned. Join to your eval table for per-query cost.

The CORTEX SEARCH SERVICE shape.

• DDL. CREATE OR REPLACE CORTEX SEARCH SERVICE svc ON <text_col> ATTRIBUTES <cols> WAREHOUSE = <wh> TARGET_LAG = '<n> minute' AS <SELECT ...> — one command; Snowflake handles chunking, embedding, and indexing.
• Refresh. TARGET_LAG sets how stale the index can be before Snowflake re-indexes. TARGET_LAG = '1 hour' for near-real-time; '1 day' for cheaper batch.
• Query. SELECT PARSE_JSON(SNOWFLAKE.CORTEX.SEARCH_PREVIEW('svc', 'my query', 8)) returns the top-8 matching rows plus their scores.
• Hybrid retrieval. Cortex Search is hybrid (BM25 + vector) under the hood; no separate config needed.

Eval-in-SQL — the recall@k pattern.

• Labels. eval_labels(query_id, query_text, expected_doc_id) — one row per (query, relevant doc) pair. Multi-relevance = multiple rows per query.
• Retrievals. eval_retrievals(query_id, retrieved_doc_id, rank) — populated by running Cortex Search on every query.
• recall@k. SUM(CASE WHEN retrieved.doc_id = labels.doc_id AND rank <= k THEN 1 ELSE 0 END) / COUNT(DISTINCT labels.query_id) per query, then averaged.
• MRR / nDCG. Standard IR formulas, expressible in pure SQL with window functions.

Common interview probes on Cortex Search Ops.

• "Why eval in SQL?" — required answer: data + index + model + eval in one governance boundary; no data egress.
• "How does Cortex Search compare to Postgres pgvector?" — Cortex is managed hybrid retrieval + indexing + refresh; pgvector is a vector column you index yourself.
• "How would you compute recall@k in SQL?" — required answer: join labels to retrievals, aggregate the match-rank ≤ k signal.
• "How is AI_COMPLETE different from Bedrock direct?" — Cortex bills through Snowflake credits, stays inside the account, no external API.

Worked example — build a Cortex Search Service and score recall@5

Detailed explanation. The canonical Snowflake eval workflow: create a CORTEX SEARCH SERVICE on a docs table, label a (query, expected_doc_id) table, run search across all labelled queries, compute recall@5 with a single SELECT. Walk through it end-to-end.

• Docs table. docs(doc_id, title, body, updated_at) — 80,000 rows.
• Search service. docs_svc on the body column, target lag 1 hour.
• Labels. eval_labels(query_id, query_text, expected_doc_id) — 200 rows across 100 queries (multi-relevance).
• Metric. recall@5.

Question. Create the search service, run it across the label queries, and compute recall@5 in one SQL query.

Input.

Object	Purpose
docs	source text
docs_svc	Cortex Search Service (hybrid)
eval_labels	100 queries × ~2 expected docs each
eval_retrievals	populated by search-per-query

Code.

-- 1. Source docs table (assume already populated)
-- CREATE TABLE docs (doc_id STRING, title STRING, body STRING, updated_at TIMESTAMP);

-- 2. Create the search service (indexes body; keeps title as an attribute)
CREATE OR REPLACE CORTEX SEARCH SERVICE docs_svc
    ON body
    ATTRIBUTES title, updated_at
    WAREHOUSE  = eval_wh
    TARGET_LAG = '1 hour'
AS (
    SELECT doc_id, title, body, updated_at
    FROM   docs
);

-- 3. Labels table (populate by hand or via product-team CSV)
CREATE OR REPLACE TABLE eval_labels (
    query_id         STRING,
    query_text       STRING,
    expected_doc_id  STRING
);
-- INSERT INTO eval_labels VALUES ('q1', 'how to rotate API key', 'doc_042'), ...

-- 4. Populate eval_retrievals by running search on every label query
CREATE OR REPLACE TABLE eval_retrievals AS
SELECT
    l.query_id,
    l.query_text,
    r.value:doc_id::STRING AS retrieved_doc_id,
    r.value:score::FLOAT   AS score,
    r.index + 1            AS rank
FROM   (SELECT DISTINCT query_id, query_text FROM eval_labels) l,
LATERAL FLATTEN (
    input => PARSE_JSON(
        SNOWFLAKE.CORTEX.SEARCH_PREVIEW(
            'docs_svc',
            l.query_text,
            10                                 -- top-10 for headroom
        )
    ):results
) r;

-- 5. recall@5 in one SQL — for each query, did any expected doc appear in top-5?
WITH per_query AS (
    SELECT
        l.query_id,
        MAX(CASE WHEN r.retrieved_doc_id = l.expected_doc_id AND r.rank <= 5 THEN 1 ELSE 0 END) AS hit
    FROM       eval_labels    l
    LEFT JOIN  eval_retrievals r
      ON r.query_id = l.query_id
    GROUP BY l.query_id
)
SELECT AVG(hit) AS recall_at_5
FROM   per_query;

Step-by-step explanation.

1. CREATE CORTEX SEARCH SERVICE is the one-command index. Snowflake chunks the body column, embeds each chunk with its managed embedding model, and stores the hybrid index. TARGET_LAG = '1 hour' means the index is at most 1 hour behind writes to the source table.
2. SNOWFLAKE.CORTEX.SEARCH_PREVIEW('docs_svc', query, 10) returns a JSON structure with a results array of up to 10 top-k rows. LATERAL FLATTEN unpacks that JSON into rows; r.index + 1 gives the 1-based rank.
3. The eval_retrievals table is a snapshot of what the search returned for each label query at eval time. Persisting it (rather than joining on-the-fly) means you can re-analyse different k values, compute MRR / nDCG, or diff two eval snapshots without re-running the search.
4. The recall@5 SELECT joins labels to retrievals on query_id; per query, it checks whether any expected doc appears in the top-5 retrieved. MAX(CASE WHEN ...) is the "any match" reduction; AVG(hit) across queries gives the scalar recall@5.
5. For multi-relevance recall (average of "fraction of expected docs hit"), swap MAX for AVG. For strict recall (all expected docs must appear), swap for MIN. The SQL primitive is flexible; the reporting choice depends on the domain.

Output.

query_id	expected in top-5?	rank hit
q1 (rotate API key)	yes	1
q2 (SLA at 3am)	yes	3
q3 (export PDF)	no	—
... (98 more)	...	...
recall@5	0.88	—

Rule of thumb. Store eval_retrievals as a persistent table, not an inline subquery. Re-computing metrics on the stored retrievals is a millisecond query; re-running the retrievals is a minutes-long recall of the search service. Persistence unlocks fast metric iteration.

Worked example — MRR + nDCG@10 + AI_COMPLETE groundedness in SQL

Detailed explanation. Recall@k is the entry-level metric; MRR (Mean Reciprocal Rank) rewards top-ranked hits; nDCG@k weighs earlier positions higher; AI_COMPLETE groundedness scores generated answers. All four fit in one SQL script. Walk through them.

• MRR. For each query, take 1 / rank_of_first_expected_doc; average across queries.
• nDCG@10. For each query, sum (2^rel - 1) / log2(rank + 1) up to rank 10; normalise by ideal DCG.
• Groundedness. For each generated answer + its retrieved contexts, AI_COMPLETE a rating prompt; parse the score.

Question. Write the SQL that computes MRR, nDCG@10, and groundedness in one script.

Input.

Metric	Requires
MRR	eval_retrievals + eval_labels
nDCG@10	+ graded relevance in eval_labels
Groundedness	eval_answers(query_id, answer) + eval_retrievals (as contexts)

Code.

-- MRR: mean reciprocal rank of the first expected hit
WITH first_hit AS (
    SELECT l.query_id,
           MIN(r.rank) AS first_hit_rank
    FROM       eval_labels    l
    LEFT JOIN  eval_retrievals r
      ON r.query_id = l.query_id AND r.retrieved_doc_id = l.expected_doc_id
    GROUP BY l.query_id
)
SELECT AVG(CASE WHEN first_hit_rank IS NULL THEN 0 ELSE 1.0 / first_hit_rank END) AS mrr
FROM   first_hit;

-- nDCG@10: normalised discounted cumulative gain (binary relevance)
WITH dcg AS (
    SELECT r.query_id,
           SUM(CASE WHEN EXISTS (SELECT 1 FROM eval_labels l
                                  WHERE l.query_id = r.query_id
                                    AND l.expected_doc_id = r.retrieved_doc_id)
                    THEN 1.0 / LOG(2, r.rank + 1) ELSE 0 END) AS dcg_val
    FROM   eval_retrievals r
    WHERE  r.rank <= 10
    GROUP BY r.query_id
),
idcg AS (
    SELECT l.query_id,
           SUM(1.0 / LOG(2, rn + 1)) AS idcg_val
    FROM (
        SELECT query_id,
               ROW_NUMBER() OVER (PARTITION BY query_id ORDER BY query_id) AS rn
        FROM eval_labels
    ) l
    WHERE rn <= 10
    GROUP BY l.query_id
)
SELECT AVG(COALESCE(dcg.dcg_val, 0) / NULLIF(idcg.idcg_val, 0)) AS ndcg_at_10
FROM   idcg
LEFT   JOIN dcg USING (query_id);

-- Groundedness: AI_COMPLETE judge per (answer, contexts)
CREATE OR REPLACE TABLE eval_groundedness AS
WITH ctx AS (
    SELECT r.query_id,
           LISTAGG(d.body, '\n---\n') WITHIN GROUP (ORDER BY r.rank) AS contexts
    FROM   eval_retrievals r
    JOIN   docs           d USING (retrieved_doc_id, doc_id)
    WHERE  r.rank <= 5
    GROUP BY r.query_id
)
SELECT a.query_id,
       a.answer,
       AI_COMPLETE(
           'claude-3-5-haiku',
           'Rate 0.0-1.0 whether the ANSWER is fully grounded in the CONTEXT. '
           || 'Return only the number.\n\n'
           || 'ANSWER:\n' || a.answer || '\n\n'
           || 'CONTEXT:\n' || ctx.contexts
       )::FLOAT AS groundedness
FROM   eval_answers a
JOIN   ctx          USING (query_id);

SELECT AVG(groundedness) AS mean_groundedness FROM eval_groundedness;

Step-by-step explanation.

1. MRR uses MIN(rank) per query to find the first hit, then 1/rank. Queries with no hit contribute 0. AVG across queries gives the scalar.
2. nDCG@10 computes DCG (discounted cumulative gain) per query using the standard 1/log2(rank+1) weighting; IDCG (ideal DCG) is the same formula with all relevant docs assumed at ranks 1..N; the ratio is nDCG. Snowflake's LOG(base, x) is the native log function.
3. The groundedness pass uses LISTAGG to concatenate the top-5 retrieved doc bodies into one context string per query, then passes answer + contexts to AI_COMPLETE with a rating prompt. AI_COMPLETE returns a string; ::FLOAT casts the numeric response.
4. All three metrics live in one Snowflake account. No data crosses a boundary; no external API call; the entire eval is auditable via SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY. This is the compliance advantage.
5. Cost breakdown: recall@5 / MRR / nDCG are pure SQL (~cents per run against a small labels table); groundedness costs one AI_COMPLETE call per query (~$0.002 per query with Haiku). 100 queries × $0.002 = ~$0.20 per full eval run.

Output.

Metric	Score
recall@5	0.88
MRR	0.72
nDCG@10	0.81
mean_groundedness	0.86
Total cost per run	~$0.20
Data egress	0 bytes

Rule of thumb. Persist per-metric result tables (eval_recall, eval_mrr, eval_groundedness) with a run_at timestamp. Diffing two runs is one SELECT; watching a trend is a Snowsight chart; alerting on a regression is a Snowflake Task + email. All four are already Snowflake-native.

Senior interview question on eval-in-SQL at the Cortex Search layer

A senior interviewer might ask: "You've got a Snowflake-native RAG using CORTEX SEARCH SERVICE and AI_COMPLETE. Nothing can leave the account. Design the nightly eval — recall@5, MRR, groundedness — as pure SQL with a TASK, ALERT, and email-on-regression. What are the four objects you deploy?"

Solution Using Cortex Search Service + AI_COMPLETE judge + TASK + ALERT

-- 1. Search service + history table (deployed once)
CREATE OR REPLACE CORTEX SEARCH SERVICE docs_svc
    ON body ATTRIBUTES title, updated_at
    WAREHOUSE = eval_wh TARGET_LAG = '1 hour'
AS (SELECT doc_id, title, body, updated_at FROM docs);

CREATE OR REPLACE TABLE eval_history (
    run_at TIMESTAMP_NTZ, run_id STRING, metric STRING, score FLOAT);

-- 2. Stored procedure: recall@5 + MRR + groundedness in one call
CREATE OR REPLACE PROCEDURE sp_run_cortex_eval() RETURNS STRING LANGUAGE SQL AS $$
DECLARE run_id STRING DEFAULT UUID_STRING();
BEGIN
    CREATE OR REPLACE TEMP TABLE _retr AS
    SELECT l.query_id, l.query_text,
           r.value:doc_id::STRING AS retrieved_doc_id, r.index + 1 AS rank
    FROM (SELECT DISTINCT query_id, query_text FROM eval_labels) l,
    LATERAL FLATTEN(input => PARSE_JSON(
        SNOWFLAKE.CORTEX.SEARCH_PREVIEW('docs_svc', l.query_text, 10)):results) r;

    INSERT INTO eval_history
    SELECT CURRENT_TIMESTAMP(), :run_id, 'recall_at_5', AVG(hit)
    FROM (SELECT l.query_id, MAX(CASE WHEN r.retrieved_doc_id = l.expected_doc_id AND r.rank <= 5 THEN 1 ELSE 0 END) hit
          FROM eval_labels l LEFT JOIN _retr r ON r.query_id = l.query_id GROUP BY l.query_id);

    INSERT INTO eval_history
    SELECT CURRENT_TIMESTAMP(), :run_id, 'mrr',
           AVG(CASE WHEN first_rank IS NULL THEN 0 ELSE 1.0 / first_rank END)
    FROM (SELECT l.query_id, MIN(r.rank) first_rank
          FROM eval_labels l LEFT JOIN _retr r
            ON r.query_id = l.query_id AND r.retrieved_doc_id = l.expected_doc_id
          GROUP BY l.query_id);

    INSERT INTO eval_history
    WITH ctx AS (SELECT r.query_id, LISTAGG(d.body, '\n---\n') WITHIN GROUP (ORDER BY r.rank) contexts
                 FROM _retr r JOIN docs d ON r.retrieved_doc_id = d.doc_id
                 WHERE r.rank <= 5 GROUP BY r.query_id)
    SELECT CURRENT_TIMESTAMP(), :run_id, 'groundedness',
           AVG(AI_COMPLETE('claude-3-5-haiku',
               'Rate 0.0-1.0 whether ANSWER is grounded in CONTEXT. Return the number only.\n\n'
               || 'ANSWER:\n' || a.answer || '\n\nCONTEXT:\n' || ctx.contexts)::FLOAT)
    FROM eval_answers a JOIN ctx USING (query_id);

    RETURN 'ok:' || :run_id;
END; $$;

-- 3. TASK: nightly at 03:00 UTC
CREATE OR REPLACE TASK t_nightly_eval
    WAREHOUSE = eval_wh SCHEDULE = 'USING CRON 0 3 * * * UTC'
AS CALL sp_run_cortex_eval();
ALTER TASK t_nightly_eval RESUME;

-- 4. ALERT: fire when any metric drops below trailing 30-day p10 × 0.95
CREATE OR REPLACE ALERT a_eval_regression
    WAREHOUSE = eval_wh SCHEDULE = '30 MINUTE'
    IF (EXISTS (
        WITH latest AS (
            SELECT metric, score FROM eval_history
            QUALIFY ROW_NUMBER() OVER (PARTITION BY metric ORDER BY run_at DESC) = 1),
        baseline AS (
            SELECT metric, PERCENTILE_CONT(0.10) WITHIN GROUP (ORDER BY score) p10
            FROM eval_history WHERE run_at >= DATEADD(day, -30, CURRENT_TIMESTAMP())
            GROUP BY metric)
        SELECT 1 FROM latest l JOIN baseline b USING (metric) WHERE l.score < b.p10 * 0.95))
    THEN CALL SYSTEM$SEND_EMAIL('mlops_alerts', '[e-mail]',
        'Cortex Search eval regression',
        'One or more Cortex Search metrics dropped below the 30-day p10.');
ALTER ALERT a_eval_regression RESUME;

Step-by-step trace.

Layer	Object	Purpose
Index	CORTEX SEARCH SERVICE docs_svc	hybrid BM25 + vector search on docs.body
Labels	eval_labels	curated (query_id, query_text, expected_doc_id)
Retrievals	temp _retr inside SP	rebuilt each run
History	eval_history	time series of all metric scores
Judge	AI_COMPLETE('claude-3-5-haiku', ...)	in-account LLM judge
Scheduler	TASK t_nightly_eval	daily at 03:00 UTC
Alert	ALERT a_eval_regression	every 30 min; email on p10 regression

After deployment, every night at 03:00 UTC Snowflake reruns the eval end-to-end, appends four rows to eval_history, and the ALERT fires an email if any metric drops 5% below the 30-day p10. The entire pipeline is Snowflake-native — no Airflow, no Kubernetes, no external SMTP; the compliance team has one governance boundary to audit.

Output:

Deliverable	Snowflake object
Search index	CORTEX SEARCH SERVICE docs_svc
Metric history	eval_history table
Runner	PROCEDURE sp_run_cortex_eval
Scheduler	TASK t_nightly_eval
Regression detector	ALERT a_eval_regression
Alert channel	email to [e-mail]
Total credits/day	~2-5 (small warehouse × short runtime)
Data egress	0 bytes

Why this works — concept by concept:

• One governance boundary — data (docs), index (docs_svc), model (AI_COMPLETE), judge (same AI_COMPLETE), eval history (eval_history), scheduler (TASK), and alerter (ALERT) are all inside one Snowflake account. Nothing crosses a compliance boundary. This is the decisive advantage over any external eval stack in regulated environments.
• Cortex Search hybrid retrieval — Snowflake's CORTEX SEARCH SERVICE is BM25 + vector out of the box; no separate embedding pipeline, no separate index cluster. TARGET_LAG handles refresh; the DDL is one command.
• AI_COMPLETE as SQL judge — LLM calls are just another SQL function. You can wrap them in AVG, join them, filter them, LISTAGG inputs, cast outputs. This is the "eval in SQL" pattern taken to its logical conclusion.
• TASK + ALERT native scheduler — Snowflake's TASK is a cron; ALERT is a change-detection primitive; both are managed, both are governed by RBAC. No external Airflow or PagerDuty needed for the core loop.
• Cost — ~2-5 credits/day for a small eval warehouse running a ~3-minute nightly job, plus ~$0.20 in AI_COMPLETE judge calls. Compared to shipping a bad Cortex Search config without a nightly gate, the alert-lead-time saving alone is worth 100× the eval cost. O(N_labels × k) for retrieval; O(N_queries) for judge; O(1) latency to detect regressions.

SQL
Topic — sql
SQL problems on IR metrics and window functions

Practice →

Design
Topic — design
Design problems on in-warehouse eval systems

Practice →

---

Cheat sheet — LLM eval recipes

• Which tool when. LangSmith when you're already inside LangChain/LangGraph and want a hosted trace + dataset + experiment UI. TruLens when you need local-first, VPC-safe, framework-agnostic instrumentation with the RAG triad. Ragas when you need a metric library that returns a scored DataFrame for the CI gate (nightly Airflow + per-PR GitHub Action). Cortex Search Ops when the entire data + index + model + eval stack lives inside Snowflake and compliance forbids data egress.
• The four axes to score. Groundedness (fraction of answer claims supported by context — anti-hallucination), answer relevance (does the answer address the question), context precision + recall (retriever quality), latency + cost (pipeline axis). Any single-axis eval hides regressions on the other three.
• LangSmith trace boilerplate. os.environ["LANGSMITH_TRACING"]="true"; os.environ["LANGSMITH_PROJECT"]="prod" before importing LangChain — auto-traces every LLM call. @traceable(run_type="chain", name="answer_endpoint", tags=["prod","v2.3"]) on FastAPI handlers to add the top-level Trace. Sample in production with LANGSMITH_TRACING_SAMPLE_RATE=0.01.
• LangSmith dataset + pairwise gate. ls.evaluate(target_fn, data="dataset_v3", experiment_prefix="candidate_v2") runs a candidate against a pinned dataset version. evaluate_comparative(experiments=[old, new], evaluators=[pairwise_helpfulness]) scores which experiment wins per row. Gate on win-rate ≥ 0.60 (excluding ties).
• TruLens RAG triad boilerplate. Wrap the app with TruChain(chain, app_name="rag", app_version="v2.3", feedbacks=[f_ctx_rel, f_grounded, f_answer_rel]). Each Feedback binds a provider metric plus selectors: context_selector = Select.RecordCalls.retriever.get_relevant_documents.rets[:].page_content. Aggregate per-chunk relevance with np.mean; collect contexts for groundedness with .collect().
• TruLens VPC production pattern. TruSession(database_url="postgresql://...") keeps every Record + score inside a VPC Postgres. LiteLLM(model_engine="bedrock/anthropic.claude-3-5-haiku-...") keeps the judge in-region. Score 2% sync for real-time alerts; nightly batch back-fill the remaining 98% for zero user-latency impact and batch-price discounts.
• Ragas gate script. evaluate(Dataset.from_list(rows), metrics=[faithfulness, answer_relevancy, context_precision, context_recall], llm=judge, embeddings=emb) returns a Result with .to_pandas(). Aggregate with .mean(), compare against THRESHOLDS dict, sys.exit(1) on any regression. Ship reference-free metrics first (faithfulness, answer_relevancy); add reference-based (context_precision, context_recall) as labels arrive.
• Ragas CI wiring. One ragas_gate.py shared between the per-PR GitHub Action (pull_request on prompts/**, src/rag/**) and the nightly Airflow DAG (schedule="0 3 * * *"). PR posts markdown-table scores as a comment; nightly alerts Slack on regression and archives report JSON to S3 at ragas/YYYY/MM/DD/report.json.
• Cortex Search Service + eval-in-SQL. CREATE CORTEX SEARCH SERVICE svc ON body WAREHOUSE=wh TARGET_LAG='1 hour' AS (SELECT ... FROM docs) for the managed hybrid index; SNOWFLAKE.CORTEX.SEARCH_PREVIEW('svc', q, k) for retrieval. Eval: recall@k = AVG(MAX(CASE WHEN retrieved_doc_id=expected_doc_id AND rank<=k THEN 1 ELSE 0 END)); MRR = AVG(1.0 / MIN(first_hit_rank)); groundedness = AVG(AI_COMPLETE('claude-3-5-haiku', prompt)::FLOAT).
• Cortex Search Ops scheduler. TASK t_nightly_eval SCHEDULE='USING CRON 0 3 * * * UTC' AS CALL sp_run_cortex_eval(); + ALERT a_eval_regression SCHEDULE='30 MINUTE' IF (EXISTS <latest score < 30-day p10 × 0.95>) THEN CALL SYSTEM$SEND_EMAIL(...). Data, index, model, judge, eval, scheduler, alerter — all in one Snowflake account.
• Metric decision matrix. Groundedness → Ragas faithfulness OR TruLens groundedness OR Cortex AI_COMPLETE judge. Answer relevance → Ragas answer_relevancy OR TruLens relevance. Retriever quality → Ragas context_precision/context_recall OR Cortex recall@k/MRR/nDCG. Latency + cost → LangSmith Run.latency/Run.total_cost OR TruLens record.latency/record.cost OR Snowflake QUERY_HISTORY.
• CI gate thresholds — day-one starting points. Faithfulness ≥ 0.85, answer_relevancy ≥ 0.80, context_precision ≥ 0.70, context_recall ≥ 0.75, latency p95 ≤ 4s, cost per query ≤ $0.015. Measure the observed baseline for two weeks before setting thresholds; day-one thresholds are guesses. Lift thresholds as the pipeline matures; never lower them without a written justification.
• Judge selection + validation. Use a cheap fast judge (Claude Haiku, Gemini Flash) with temperature=0 for determinism. Re-validate quarterly against 100 human-labelled rows; if judge-human correlation drops below 0.75, swap the judge model or add few-shot examples to the prompt. Pairwise judges are more sensitive than pointwise judges for A/B comparisons; use pointwise for absolute thresholds.
• Cost budgeting rules. Judge cost ≈ 1-2× production LLM cost per evaluated row. Sample 1-5% of production for sync scoring; back-fill 100% via batch (50% discount on most providers). CI gate over 100-row golden set ≈ $1-2 per PR run. Never run 100% sync judges in production; the extra latency and cost destroy the value proposition.

Frequently asked questions

What is LLM evaluation for data pipelines in one sentence?

LLM evaluation for data pipelines is the discipline of scoring every LLM-driven output your pipeline produces — RAG answers, summaries, classifications, SQL-generation results — against a small set of orthogonal quality axes (groundedness, answer relevance, context precision / recall, latency + cost) so hallucinations, retrieval misses, prompt drift, and cost explosions are caught by an automated gate rather than by a customer three weeks later. In 2026 the four canonical tools are LangSmith (trace-first hosted platform with versioned datasets and hosted judges), TruLens (local-first, VPC-safe, framework-agnostic with the RAG triad), Ragas (reference-free metric library that returns a scored DataFrame for CI gates), and Snowflake Cortex Search Ops (eval-in-SQL with AI_COMPLETE judges when the entire data + model stack lives in Snowflake). The tools coexist because their sweet spots don't overlap; the wrong senior answer is picking one and calling the eval done. Every senior data-engineering interview probes LLM eval because it's the load-bearing correctness discipline for the modern RAG + agent stack.

LangSmith vs TruLens vs Ragas — which do I pick?

Default to LangSmith when your app already lives inside LangChain or LangGraph and you want a hosted UI: trace tree, versioned datasets, hosted evaluators, pairwise experiments, all in one dashboard with per-Run latency and cost attached for free. Pick TruLens when you need eval inside a VPC or want to instrument a non-LangChain Python app: TruChain / TruLlama / TruCustomApp wrap any Python function; the RAG triad (context relevance, groundedness, answer relevance) is the canonical three-edge completeness proof; storage defaults to local SQLite and swaps to Postgres for team use; no vendor round-trip is required at any point. Pick Ragas when you want a metric library that plays well with any CI stack: evaluate() returns a pandas DataFrame; two metrics (faithfulness, answer_relevancy) need no reference answer so bootstrapping is cheap; two metrics (context_precision, context_recall) use labelled ground truth; the whole thing runs headless in an Airflow DAG or a GitHub Action. In practice most teams end up running two of the three: LangSmith or TruLens for traces + interactive UI, Ragas for the CI gate. The three are complementary, not competitive.

What is the RAG triad?

The RAG triad is TruLens's canonical eval completeness proof for a Retrieval-Augmented Generation system: a triangle whose three vertices are the query, the retrieved context, and the generated answer, and whose three edges are the three orthogonal quality axes that must all pass for the RAG to be trustworthy. The edges are (a) context relevance — is the retrieved context actually relevant to the query? — a low score points at the retriever; (b) groundedness — is every claim in the generated answer supported by the retrieved context? — a low score points at hallucination; (c) answer relevance — does the generated answer actually address the query, or does it wander off topic? — a low score points at the prompt or the model. Any one of the three can regress independently, so scoring only one hides regressions on the other two. The triad is why TruLens's default instrumentation ships three Feedback functions rather than one, and why senior interviewers probe the triangle rather than a single "hallucination score" — a candidate who names all three edges signals they understand RAG eval as a multi-dimensional discipline.

Can I do LLM eval inside Snowflake?

Yes — snowflake cortex search plus AI_COMPLETE plus the standard TASK + ALERT primitives give you a fully in-warehouse eval loop. The CORTEX SEARCH SERVICE DDL is one command; it manages hybrid BM25 + vector indexing, chunking, embedding, and refresh via TARGET_LAG. SNOWFLAKE.CORTEX.SEARCH_PREVIEW('svc', query, k) returns top-k results as JSON, which LATERAL FLATTEN unpacks into rows for eval joins against a labelled eval_labels(query_id, expected_doc_id) table. Recall@k, MRR, and nDCG@k are pure aggregate SQL; groundedness is AI_COMPLETE('claude-3-5-haiku', 'Rate 0-1 ...')::FLOAT wrapped in AVG. A Snowflake TASK schedules the whole thing on cron; a Snowflake ALERT fires an email when the latest score drops below the trailing 30-day p10 by more than 5%. The decisive advantage is governance — data, index, model, judge, history, scheduler, and alerter all live inside one Snowflake account with one RBAC boundary. For banks, healthcare, and government workloads where data egress is a compliance loss, Cortex Search Ops is the default answer.

Groundedness vs answer relevance — how are they different?

Groundedness asks "is every claim in the answer supported by the retrieved context?" — it's the anti-hallucination axis. An answer that invents a fact the retrieved documents do not contain scores low on groundedness even if the fact happens to be correct in the world. Ragas calls this metric faithfulness; TruLens calls it groundedness; both split the answer into atomic claims, ask an LLM judge whether each claim is entailed by the retrieved context, and aggregate. Answer relevance asks "does the answer address the user's question?" — it's the on-topic axis. An answer that is perfectly grounded (every claim is in the context) but doesn't address the question ("here is the entire policy document") scores low on answer relevance. Ragas measures answer_relevancy by generating hypothetical questions the answer could address, then computing cosine similarity between those questions and the original query. The two metrics are orthogonal: a RAG chain can be perfectly grounded and off-topic, or on-topic and hallucinating; measuring only one hides the other's failure. Every RAG eval must score both.

Do I need reference answers to evaluate a RAG?

No — this is why Ragas ships two reference-free metrics: faithfulness and answer_relevancy. faithfulness needs only the generated answer and the retrieved contexts — an LLM judge scores whether the answer's claims are entailed by the context, and no ground-truth answer is required. answer_relevancy needs only the question and the answer — the metric generates hypothetical questions the answer could address and compares to the original query via cosine similarity. Both metrics let you bootstrap an eval gate on day one, before any human has written a single labelled answer. Reference-based metrics like context_precision, context_recall, and answer_correctness do need labelled ground truth (an expected_doc_id per query, or a written ground_truth answer), and those take human curation time — budget 5-10 minutes per row. The standard adoption path is: start with the two reference-free metrics + a small human-curated (question, expected_doc_id) table for retrieval metrics; grow the ground-truth answers as the eval matures. TruLens's RAG triad and LangSmith's hosted criteria evaluators also work reference-free, so the three tools all support this bootstrap-first pattern.

Practice on PipeCode

• Drill the ETL practice library → for the eval-gate, incremental scoring, and RAG ingestion pipeline problems senior interviewers love.
• Rehearse on the data-validation practice library → for the threshold-gating, batch scoring, and quality-metric patterns.
• Sharpen the systems axis with the design practice library → for VPC-first eval systems, in-warehouse eval loops, and pairwise experiment design.
• Level up your Python instrumentation on the Python practice library → for decorator patterns, wrapper design, and async LLM-judge batching.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis measurement plan against real graded inputs.

Lock in LLM eval muscle memory

Docs explain metrics. PipeCode drills explain the decision — when to gate on faithfulness, when the RAG triad's context edge is the retriever's report card, when Ragas beats LangSmith for CI, when Cortex Search Ops wins on governance. Pipecode.ai is Leetcode for Data Engineering — axis-first practice tuned for the production trade-offs senior data and MLOps engineers actually face.

Practice ETL problems →
Practice design problems →