RAGAS renamed ground_truth to reference
I lost an afternoon to this, so here is the short version.
A RAGAS evaluation dataset built a few months ago fails to load today. The JSON is
valid. The keys look right. Nothing about the file changed — the library did.
ground_truth became reference. And while the field was being renamed, the sample
schema also moved the question and answer: question became user_input, answer
became response. If your pipeline still emits the old names, you get a validation
error that points at the dataset rather than at the rename.
Why it breaks the way it does
RAGAS stores a dataset as columnar parallel arrays — one array per field, one entry
per sample. That shape hides renames badly. If a key is missing, the library does not
tell you "I expected reference". It tells you the dataset is invalid, and you go
looking through your data for a problem that is actually in your column names.
There are three schema generations in the wild, and they disagree:
| Format | Question | Contexts | Answer | Reference |
|---|---|---|---|---|
| Legacy | question |
contexts |
answer |
ground_truth |
| Current | question |
contexts |
answer |
reference |
Modern (SingleTurnSample) |
user_input |
retrieved_contexts |
response |
reference |
Every tutorial written before the rename uses the first row. Every dataset generated by
TestsetGenerator today uses the third. They are all "RAGAS datasets", and none of them
load in the same code.
Migrating by hand
If your data is already a DataFrame, the rename is three lines:
import pandas as pd
df = pd.read_json("eval_set.json")
df = df.rename(columns={
"question": "user_input",
"answer": "response",
"ground_truth": "reference",
})
df.to_json("eval_set_v2.json", orient="list")
orient="list" matters. That is the columnar shape RAGAS expects. orient="records"
produces one object per row, which is what TruLens and LangSmith use and what RAGAS
will reject.
Two things to check before you move on.
Contexts must be a list of strings, not a string. This is the most common silent
failure. A single string is valid JSON, loads fine, and then produces nonsense scores
because each character gets treated as a separate context.
import json
data = json.load(open("eval_set_v2.json"))
bad_contexts = [i for i, c in enumerate(data["retrieved_contexts"])
if not isinstance(c, list)]
if bad_contexts:
print("contexts not a list at rows:", bad_contexts)
Every array must be the same length. A short array shifts every later row onto the
wrong record — the dataset still loads, and every score you compute afterwards is
measuring the wrong pairing.
lengths = {k: len(v) for k, v in data.items() if isinstance(v, list)}
if len(set(lengths.values())) != 1:
print("ragged arrays:", lengths)
Neither of these throws. Both corrupt results quietly. Run both checks on every dataset
you convert.
contexts vs retrieved_contexts is not just a rename
This one cost me more than the ground_truth change.
TestsetGenerator produces reference_contexts — the gold chunks a question was built
from. Your pipeline produces retrieved_contexts — what your retriever actually
returned. They are different things, and Context Recall is defined as the overlap
between them.
If you rename contexts to retrieved_contexts blindly, you may be feeding gold chunks
in as retrieved chunks. Retrieval scores then describe the generator, not your retriever,
and they will look excellent for reasons that have nothing to do with your system.
Check which one you have before renaming.
Which metrics you lose without a reference
Not everything needs reference. If you have no reference answers, you can still run:
- Faithfulness — is the answer grounded in the retrieved context
- Response Relevancy — does the answer address the question
- Aspect Critic / Rubrics — does the answer satisfy a criterion you define
These need a reference and cannot run without one:
- Context Recall
- Context Precision
- Answer Correctness
- Factual Correctness
- Semantic Similarity
So a dataset with no references is still useful — it just measures generation quality and
not retrieval quality. That distinction is worth knowing before you spend an hour
generating references.
The shape mismatch across frameworks
The same four columns, five different names, depending on the framework:
| Framework | Question | Contexts | Answer | Reference |
|---|---|---|---|---|
| RAGAS modern | user_input |
retrieved_contexts |
response |
reference |
| RAGAS current | question |
contexts |
answer |
reference |
| RAGAS legacy | question |
contexts |
answer |
ground_truth |
| TruLens | query |
expected_chunks |
expected_response |
— |
| LangSmith | inputs.question |
outputs.contexts |
outputs.answer |
outputs.reference |
Note the structural differences too, not just the names. RAGAS is columnar. TruLens is
one record per line item. LangSmith nests everything under inputs / outputs. Moving
a dataset between them is a reshape as much as a rename — and reference has no TruLens
equivalent at all, so that column gets dropped on the way over.
If you'd rather not hand-edit JSON
I got tired of doing this by hand, so I built a converter that runs entirely in the
browser: ragasconverter.com.
Paste a dataset in any of the five shapes, pick a target, and it converts — then prints
every field it renamed, flattened, nested or dropped, so nothing disappears without you
seeing it. It also runs the two validation checks above (ragged arrays, string contexts)
and reports which RAGAS metrics your dataset can actually run right now, with the record
index of each problem.
No signup, no upload, no server. It is a client-side bundle, so eval data never leaves
the tab — which matters when the dataset contains internal documents.
The field mapping reference, if you just want the table without the tool:
ragasconverter.com/ragas-dataset-format.
And the metric-by-metric breakdown:
ragasconverter.com/ragas-without-ground-truth.
Field names above follow the
RAGAS docs and
ragas/dataset_schema.py.
Schema details are version-dependent — pin your RAGAS version before trusting any of
this, including mine.
Top comments (0)