DEV Community

Cover image for RAGAS has two from_dict methods with opposite shapes — and zip() eats your rows
kai wang
kai wang

Posted on AI-assisted

RAGAS has two from_dict methods with opposite shapes — and zip() eats your rows

RAGAS has two from_dict methods with opposite shapes

I lost a morning to KeyError: 0.

The message says nothing useful. It doesn't say "you passed columns where records were
expected". It doesn't name a field. It just says the key 0 is missing — and if you're
staring at a dict whose keys are user_input, retrieved_contexts, response, reference,
the number 0 looks like a bug in somebody else's library.

It isn't. It's a shape mismatch, and RAGAS has two entry points that want opposite shapes.

The two shapes

RAGAS stores an evaluation dataset as columnar parallel arrays: one list per field, one
entry per sample.

{
  "user_input": ["What is RAG?", "Who wrote it?"],
  "retrieved_contexts": [["RAG combines retrieval with generation."], ["Nobody."]],
  "response": ["RAG retrieves documents.", "Unknown."],
  "reference": ["Retrieval-augmented generation retrieves and generates.", "Unknown."]
}
Enter fullscreen mode Exit fullscreen mode

One list per column, two samples. That's the shape you get from a pandas
df.to_dict(orient="list"), and the shape HuggingFace wants:

from datasets import Dataset

hf = Dataset.from_dict(columnar)     # columnar  -> ok
result = evaluate(hf, metrics=[...])
Enter fullscreen mode Exit fullscreen mode

Now the other one:

from ragas import EvaluationDataset

ds = EvaluationDataset.from_dict(columnar)   # columnar -> KeyError: 0
Enter fullscreen mode Exit fullscreen mode

Same method name. Opposite requirement. EvaluationDataset.from_dict wants a list of
record dicts
, one dict per sample:

records = [
    {
        "user_input": "What is RAG?",
        "retrieved_contexts": ["RAG combines retrieval with generation."],
        "response": "RAG retrieves documents.",
        "reference": "Retrieval-augmented generation retrieves and generates.",
    },
]

ds = EvaluationDataset.from_dict(records)    # records -> ok
Enter fullscreen mode Exit fullscreen mode

The source explains the error. from_dict reads mapping[0]["user_input"] — it indexes
the argument as a list. A columnar dict has no key 0, hence the KeyError.

If you have columns and want records, EvaluationDataset.from_list exists, but it also
wants records. The columnar form belongs to HuggingFace, not to RAGAS's own container.

The trap in converting columns to records

So you convert. And this is where rows go missing.

cols = {
    "user_input": ["q1", "q2", "q3"],
    "reference":  ["r1", "r2"],
}

records = [dict(zip(cols, vals)) for vals in zip(*cols.values())]
Enter fullscreen mode Exit fullscreen mode

Three questions. Two references. How many records?

Two. The third question is gone, silently, with no error and no warning. zip stops at
the shortest iterable — that's its documented behaviour, and here it deletes data.

I ran it:

列长度: {'user_input': 3, 'reference': 2}
zip 转 records 后: 2 条
Enter fullscreen mode Exit fullscreen mode

If reference is the column you added last, from a partial manual labelling pass, you have
just thrown away every sample beyond the labelled ones. Every metric then reports on a
dataset that is smaller than you think, and the numbers look fine.

Check before you zip:

lengths = {k: len(v) for k, v in cols.items()}
if len(set(lengths.values())) != 1:
    raise ValueError(f"ragged columns: {lengths}")
Enter fullscreen mode Exit fullscreen mode

The contexts shape is not what the columns suggest

This one is subtle enough that I got it backwards at first.

In the columnar file, retrieved_contexts looks like a list of lists:

"retrieved_contexts": [["chunk one"], ["chunk two"]]
Enter fullscreen mode Exit fullscreen mode

That's two samples, one chunk each. But a SingleTurnSample represents one sample, so
its retrieved_contexts is a flat list of that sample's chunks:

SingleTurnSample(retrieved_contexts=["chunk one", "chunk two"])   # one sample, two chunks
Enter fullscreen mode Exit fullscreen mode

Which means if you take a column straight out of the columnar file and hand it to a sample,
you're passing the last level one too deep. I tested what RAGAS actually does, since the
docs don't spell it out:

Passed as retrieved_contexts Result
"chunk one" (a string) ValidationError — Input should be a valid list
["chunk one", "chunk two"] OK — stored as ['chunk one', 'chunk two']
[["chunk one"], ["chunk two"]] ValidationError — Input should be a valid string

Note the middle row. A flat list of strings is accepted — and it is correct for a single
sample. The same literal, taken from a two-sample columnar file, is wrong. The type system
can't tell the difference because there isn't one: the two cases are byte-identical JSON
with different intent.

That's the general lesson here. Pydantic validates types, not meaning.

What actually fails silently

Because pydantic is stricter than I expected, most shape mistakes throw. That's good news,
and it contradicts the usual advice to "beware of silent failures". Specifically, these
throw:

  • retrieved_contexts as a bare string
  • retrieved_contexts with one extra nesting level
  • from_dict with a columnar dict (KeyError: 0)
  • unequal columns passed to datasets.Dataset.from_dict (arrow length mismatch)

These do not throw, and they're the ones worth memorising:

1. Ragged columns through a manual zip. Covered above. Rows vanish.

2. reference_contexts used where retrieved_contexts belongs. Both are List[str].
Both validate. The difference is semantic: reference_contexts are the gold chunks a
question was built from; retrieved_contexts are what your retriever returned. Feed gold
chunks in as retrieved chunks and Context Recall measures your test-set generator instead
of your retriever. The scores will be excellent and meaningless.

3. Empty strings. user_input: "" and response: "" are valid Optional[str]. They
load, they evaluate, and the metrics quietly score nothing.

A checklist that catches all three

import json

cols = json.load(open("eval_set.json"))

# 1. columns must be equal length before any zip
lengths = {k: len(v) for k, v in cols.items() if isinstance(v, list)}
assert len(set(lengths.values())) == 1, f"ragged columns: {lengths}"

# 2. each contexts entry must be a list, not a string
bad = [i for i, c in enumerate(cols.get("retrieved_contexts", []))
       if not isinstance(c, list)]
assert not bad, f"contexts not a list at rows: {bad}"

# 3. empty questions/answers
for field in ("user_input", "response"):
    empty = [i for i, v in enumerate(cols.get(field, [])) if not str(v).strip()]
    if empty:
        print(f"{field} empty at rows: {empty}")

# 4. did you mean reference_contexts? they are not interchangeable
if "reference_contexts" in cols and "retrieved_contexts" not in cols:
    print("only gold chunks present — retrieval metrics will describe the generator")
Enter fullscreen mode Exit fullscreen mode

None of these four checks is in the library, because none of them is a type error.

Doing it without writing the checks

I built a converter for exactly this loop, since I kept re-deriving it. It reads RAGAS
modern / current / legacy, TruLens and LangSmith JSON, converts to whichever target you
pick, and prints every field it renamed, flattened, nested or dropped instead of quietly
reshaping your data.

It also runs the four checks above and reports which RAGAS metrics the dataset can run
right now.

ragasconverter.com — client-side,
no upload, no signup.


Behaviour above is from RAGAS dataset_schema.py (SingleTurnSample, EvaluationDataset.from_dict)
and a local pydantic v2 test of the three retrieved_contexts shapes. from_dict's shape
requirement has been stable across recent releases, but SingleTurnSample fields change —
pin your version.

Top comments (0)