<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: kai wang</title>
    <description>The latest articles on DEV Community by kai wang (@kai_wang_52aff506bc3184b8).</description>
    <link>https://dev.to/kai_wang_52aff506bc3184b8</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4253407%2F5b1e2b98-a875-4781-ba6e-133cee8ba9a8.jpg</url>
      <title>DEV Community: kai wang</title>
      <link>https://dev.to/kai_wang_52aff506bc3184b8</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/kai_wang_52aff506bc3184b8"/>
    <language>en</language>
    <item>
      <title>RAGAS has two from_dict methods with opposite shapes — and zip() eats your rows</title>
      <dc:creator>kai wang</dc:creator>
      <pubDate>Thu, 19 Nov 2026 09:39:43 +0000</pubDate>
      <link>https://dev.to/kai_wang_52aff506bc3184b8/ragas-has-two-fromdict-methods-with-opposite-shapes-and-zip-eats-your-rows-n8i</link>
      <guid>https://dev.to/kai_wang_52aff506bc3184b8/ragas-has-two-fromdict-methods-with-opposite-shapes-and-zip-eats-your-rows-n8i</guid>
      <description>&lt;h1&gt;
  
  
  RAGAS has two &lt;code&gt;from_dict&lt;/code&gt; methods with opposite shapes
&lt;/h1&gt;

&lt;p&gt;I lost a morning to &lt;code&gt;KeyError: 0&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The message says nothing useful. It doesn't say "you passed columns where records were&lt;br&gt;
expected". It doesn't name a field. It just says the key &lt;code&gt;0&lt;/code&gt; is missing — and if you're&lt;br&gt;
staring at a dict whose keys are &lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;retrieved_contexts&lt;/code&gt;, &lt;code&gt;response&lt;/code&gt;, &lt;code&gt;reference&lt;/code&gt;,&lt;br&gt;
the number &lt;code&gt;0&lt;/code&gt; looks like a bug in somebody else's library.&lt;/p&gt;

&lt;p&gt;It isn't. It's a shape mismatch, and RAGAS has two entry points that want opposite shapes.&lt;/p&gt;
&lt;h2&gt;
  
  
  The two shapes
&lt;/h2&gt;

&lt;p&gt;RAGAS stores an evaluation dataset as &lt;strong&gt;columnar parallel arrays&lt;/strong&gt;: one list per field, one&lt;br&gt;
entry per sample.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"user_input"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"What is RAG?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Who wrote it?"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"retrieved_contexts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"RAG combines retrieval with generation."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Nobody."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"response"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"RAG retrieves documents."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Unknown."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"reference"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Retrieval-augmented generation retrieves and generates."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Unknown."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One list per column, two samples. That's the shape you get from a pandas&lt;br&gt;
&lt;code&gt;df.to_dict(orient="list")&lt;/code&gt;, and the shape HuggingFace wants:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datasets&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Dataset&lt;/span&gt;

&lt;span class="n"&gt;hf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Dataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;columnar&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;     &lt;span class="c1"&gt;# columnar  -&amp;gt; ok&lt;/span&gt;
&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the other one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;ragas&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;EvaluationDataset&lt;/span&gt;

&lt;span class="n"&gt;ds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;EvaluationDataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;columnar&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# columnar -&amp;gt; KeyError: 0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same method name. Opposite requirement. &lt;code&gt;EvaluationDataset.from_dict&lt;/code&gt; wants &lt;strong&gt;a list of&lt;br&gt;
record dicts&lt;/strong&gt;, one dict per sample:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;records&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"user_input"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"What is RAG?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"retrieved_contexts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"RAG combines retrieval with generation."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"response"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"RAG retrieves documents."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"reference"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Retrieval-augmented generation retrieves and generates."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;ds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;EvaluationDataset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;    &lt;span class="c1"&gt;# records -&amp;gt; ok&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The source explains the error. &lt;code&gt;from_dict&lt;/code&gt; reads &lt;code&gt;mapping[0]["user_input"]&lt;/code&gt; — it indexes&lt;br&gt;
the argument as a list. A columnar dict has no key &lt;code&gt;0&lt;/code&gt;, hence the KeyError.&lt;/p&gt;

&lt;p&gt;If you have columns and want records, &lt;code&gt;EvaluationDataset.from_list&lt;/code&gt; exists, but it also&lt;br&gt;
wants records. The columnar form belongs to HuggingFace, not to RAGAS's own container.&lt;/p&gt;
&lt;h2&gt;
  
  
  The trap in converting columns to records
&lt;/h2&gt;

&lt;p&gt;So you convert. And this is where rows go missing.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;cols&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"user_input"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"q1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"q2"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"q3"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"reference"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"r1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"r2"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;records&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;vals&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three questions. Two references. How many records?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Two.&lt;/strong&gt; The third question is gone, silently, with no error and no warning. &lt;code&gt;zip&lt;/code&gt; stops at&lt;br&gt;
the shortest iterable — that's its documented behaviour, and here it deletes data.&lt;/p&gt;

&lt;p&gt;I ran it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;列长度: {'user_input': 3, 'reference': 2}
zip 转 records 后: 2 条
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;reference&lt;/code&gt; is the column you added last, from a partial manual labelling pass, you have&lt;br&gt;
just thrown away every sample beyond the labelled ones. Every metric then reports on a&lt;br&gt;
dataset that is smaller than you think, and the numbers look fine.&lt;/p&gt;

&lt;p&gt;Check before you zip:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;lengths&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lengths&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"ragged columns: {lengths}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The contexts shape is not what the columns suggest
&lt;/h2&gt;

&lt;p&gt;This one is subtle enough that I got it backwards at first.&lt;/p&gt;

&lt;p&gt;In the columnar file, &lt;code&gt;retrieved_contexts&lt;/code&gt; looks like a list of lists:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"retrieved_contexts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"chunk one"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"chunk two"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's two samples, one chunk each. But a &lt;code&gt;SingleTurnSample&lt;/code&gt; represents &lt;strong&gt;one&lt;/strong&gt; sample, so&lt;br&gt;
its &lt;code&gt;retrieved_contexts&lt;/code&gt; is a flat list of that sample's chunks:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nc"&gt;SingleTurnSample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;retrieved_contexts&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"chunk one"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"chunk two"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# one sample, two chunks&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Which means if you take a column straight out of the columnar file and hand it to a sample,&lt;br&gt;
you're passing the last level one too deep. I tested what RAGAS actually does, since the&lt;br&gt;
docs don't spell it out:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Passed as &lt;code&gt;retrieved_contexts&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;"chunk one"&lt;/code&gt; (a string)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ValidationError&lt;/code&gt; — Input should be a valid list&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;["chunk one", "chunk two"]&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;OK&lt;/strong&gt; — stored as &lt;code&gt;['chunk one', 'chunk two']&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;[["chunk one"], ["chunk two"]]&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ValidationError&lt;/code&gt; — Input should be a valid string&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Note the middle row. A flat list of strings is accepted — and it is &lt;em&gt;correct&lt;/em&gt; for a single&lt;br&gt;
sample. The same literal, taken from a two-sample columnar file, is wrong. The type system&lt;br&gt;
can't tell the difference because there isn't one: the two cases are byte-identical JSON&lt;br&gt;
with different intent.&lt;/p&gt;

&lt;p&gt;That's the general lesson here. Pydantic validates &lt;strong&gt;types&lt;/strong&gt;, not &lt;strong&gt;meaning&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually fails silently
&lt;/h2&gt;

&lt;p&gt;Because pydantic is stricter than I expected, most shape mistakes throw. That's good news,&lt;br&gt;
and it contradicts the usual advice to "beware of silent failures". Specifically, these&lt;br&gt;
throw:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;retrieved_contexts&lt;/code&gt; as a bare string&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;retrieved_contexts&lt;/code&gt; with one extra nesting level&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;from_dict&lt;/code&gt; with a columnar dict (&lt;code&gt;KeyError: 0&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;unequal columns passed to &lt;code&gt;datasets.Dataset.from_dict&lt;/code&gt; (arrow length mismatch)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These do &lt;strong&gt;not&lt;/strong&gt; throw, and they're the ones worth memorising:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Ragged columns through a manual &lt;code&gt;zip&lt;/code&gt;.&lt;/strong&gt; Covered above. Rows vanish.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. &lt;code&gt;reference_contexts&lt;/code&gt; used where &lt;code&gt;retrieved_contexts&lt;/code&gt; belongs.&lt;/strong&gt; Both are &lt;code&gt;List[str]&lt;/code&gt;.&lt;br&gt;
Both validate. The difference is semantic: &lt;code&gt;reference_contexts&lt;/code&gt; are the gold chunks a&lt;br&gt;
question was built from; &lt;code&gt;retrieved_contexts&lt;/code&gt; are what your retriever returned. Feed gold&lt;br&gt;
chunks in as retrieved chunks and Context Recall measures your test-set generator instead&lt;br&gt;
of your retriever. The scores will be excellent and meaningless.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Empty strings.&lt;/strong&gt; &lt;code&gt;user_input: ""&lt;/code&gt; and &lt;code&gt;response: ""&lt;/code&gt; are valid &lt;code&gt;Optional[str]&lt;/code&gt;. They&lt;br&gt;
load, they evaluate, and the metrics quietly score nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  A checklist that catches all three
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;

&lt;span class="n"&gt;cols&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"eval_set.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 1. columns must be equal length before any zip&lt;/span&gt;
&lt;span class="n"&gt;lengths&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lengths&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"ragged columns: {lengths}"&lt;/span&gt;

&lt;span class="c1"&gt;# 2. each contexts entry must be a list, not a string&lt;/span&gt;
&lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"retrieved_contexts"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
       &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"contexts not a list at rows: {bad}"&lt;/span&gt;

&lt;span class="c1"&gt;# 3. empty questions/answers&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"user_input"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"response"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;empty&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;empty&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{field} empty at rows: {empty}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 4. did you mean reference_contexts? they are not interchangeable&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;"reference_contexts"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cols&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="s2"&gt;"retrieved_contexts"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"only gold chunks present — retrieval metrics will describe the generator"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;None of these four checks is in the library, because none of them is a type error.&lt;/p&gt;

&lt;h2&gt;
  
  
  Doing it without writing the checks
&lt;/h2&gt;

&lt;p&gt;I built a converter for exactly this loop, since I kept re-deriving it. It reads RAGAS&lt;br&gt;
modern / current / legacy, TruLens and LangSmith JSON, converts to whichever target you&lt;br&gt;
pick, and prints every field it renamed, flattened, nested or dropped instead of quietly&lt;br&gt;
reshaping your data.&lt;/p&gt;

&lt;p&gt;It also runs the four checks above and reports which RAGAS metrics the dataset can run&lt;br&gt;
right now.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://ragasconverter.com/ragas-dataset-format/" rel="noopener noreferrer"&gt;ragasconverter.com&lt;/a&gt;&lt;/strong&gt; — client-side,&lt;br&gt;
no upload, no signup.&lt;/p&gt;




&lt;p&gt;Behaviour above is from RAGAS &lt;code&gt;dataset_schema.py&lt;/code&gt; (&lt;code&gt;SingleTurnSample&lt;/code&gt;, &lt;code&gt;EvaluationDataset.from_dict&lt;/code&gt;)&lt;br&gt;
and a local pydantic v2 test of the three &lt;code&gt;retrieved_contexts&lt;/code&gt; shapes. &lt;code&gt;from_dict&lt;/code&gt;'s shape&lt;br&gt;
requirement has been stable across recent releases, but &lt;code&gt;SingleTurnSample&lt;/code&gt; fields change —&lt;br&gt;
pin your version.&lt;/p&gt;

</description>
      <category>rag</category>
      <category>llm</category>
      <category>python</category>
      <category>data</category>
    </item>
    <item>
      <title>LangSmith to RAGAS — converting an exported dataset without losing fields silently</title>
      <dc:creator>kai wang</dc:creator>
      <pubDate>Thu, 19 Nov 2026 09:33:43 +0000</pubDate>
      <link>https://dev.to/kai_wang_52aff506bc3184b8/langsmith-to-ragas-converting-an-exported-dataset-without-losing-fields-silently-5ap8</link>
      <guid>https://dev.to/kai_wang_52aff506bc3184b8/langsmith-to-ragas-converting-an-exported-dataset-without-losing-fields-silently-5ap8</guid>
      <description>&lt;h1&gt;
  
  
  LangSmith to RAGAS: converting an exported dataset
&lt;/h1&gt;

&lt;p&gt;I moved an eval set from LangSmith into RAGAS last month and lost half a day to two&lt;br&gt;
things that never raised an error: the question column came out empty, and the reference&lt;br&gt;
column silently didn't exist.&lt;/p&gt;

&lt;p&gt;Both had the same root cause. LangSmith and RAGAS disagree about what a dataset &lt;em&gt;is&lt;/em&gt;.&lt;/p&gt;
&lt;h2&gt;
  
  
  The shape problem
&lt;/h2&gt;

&lt;p&gt;RAGAS stores a dataset as &lt;strong&gt;columnar parallel arrays&lt;/strong&gt; — one array per field, one entry per&lt;br&gt;
sample, every array the same length. LangSmith stores &lt;strong&gt;one example per record&lt;/strong&gt;, with the&lt;br&gt;
actual content nested inside two free-form objects.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="n"&gt;LangSmith&lt;/span&gt;                          &lt;span class="n"&gt;RAGAS&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;                                 &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"inputs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"question"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;     &lt;span class="nl"&gt;"user_input"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"outputs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;                       &lt;span class="nl"&gt;"retrieved_contexts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"contexts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;               &lt;span class="nl"&gt;"response"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"answer"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"..."&lt;/span&gt;                  &lt;span class="nl"&gt;"reference"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;                                &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So the conversion is a reshape, not a find-and-replace. You are unstacking rows into&lt;br&gt;
columns &lt;em&gt;and&lt;/em&gt; renaming fields, and the second half is where things go quiet.&lt;/p&gt;
&lt;h2&gt;
  
  
  What a LangSmith example actually contains
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;Example&lt;/code&gt; model in the &lt;code&gt;langsmith&lt;/code&gt; SDK has these fields:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;id&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;UUID&lt;/td&gt;
&lt;td&gt;example identity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dataset_id&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;UUID&lt;/td&gt;
&lt;td&gt;which dataset it belongs to&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;inputs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;free-form dict&lt;/td&gt;
&lt;td&gt;whatever your pipeline logged&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;outputs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;free-form dict&lt;/td&gt;
&lt;td&gt;whatever your pipeline logged&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;metadata&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;free-form dict&lt;/td&gt;
&lt;td&gt;arbitrary, e.g. &lt;code&gt;{"source": "Wikipedia"}&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;created_at&lt;/code&gt; / &lt;code&gt;modified_at&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;datetime&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;source_run_id&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;UUID or null&lt;/td&gt;
&lt;td&gt;if built from a trace&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Note what's missing: &lt;strong&gt;&lt;code&gt;inputs&lt;/code&gt; and &lt;code&gt;outputs&lt;/code&gt; have no schema.&lt;/strong&gt; LangSmith does not require&lt;br&gt;
&lt;code&gt;inputs.question&lt;/code&gt; or &lt;code&gt;outputs.answer&lt;/code&gt;. The docs' own example uses exactly those names, so&lt;br&gt;
everyone copies them — but a pipeline that logged &lt;code&gt;{"query": ...}&lt;/code&gt; produces a perfectly&lt;br&gt;
valid export that no RAGAS converter can read.&lt;/p&gt;
&lt;h2&gt;
  
  
  The field mapping
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;LangSmith&lt;/th&gt;
&lt;th&gt;RAGAS (&lt;code&gt;SingleTurnSample&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Required&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;inputs.question&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;user_input&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;outputs.contexts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;retrieved_contexts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;For retrieval metrics&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;outputs.answer&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;response&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;For answer metrics&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;outputs.reference&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;reference&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;For reference-based metrics&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Four fields in, four fields out. Everything else has nowhere to go.&lt;/p&gt;
&lt;h2&gt;
  
  
  The six fields that get dropped
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;SingleTurnSample&lt;/code&gt; accepts exactly these: &lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;retrieved_contexts&lt;/code&gt;,&lt;br&gt;
&lt;code&gt;reference_contexts&lt;/code&gt;, &lt;code&gt;retrieved_context_ids&lt;/code&gt;, &lt;code&gt;reference_context_ids&lt;/code&gt;, &lt;code&gt;response&lt;/code&gt;,&lt;br&gt;
&lt;code&gt;multi_responses&lt;/code&gt;, &lt;code&gt;reference&lt;/code&gt;, &lt;code&gt;rubrics&lt;/code&gt;, &lt;code&gt;persona_name&lt;/code&gt;, &lt;code&gt;query_style&lt;/code&gt;, &lt;code&gt;query_length&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;There is &lt;strong&gt;no generic &lt;code&gt;metadata&lt;/code&gt; field&lt;/strong&gt;. So on the way over you lose:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;id&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;dataset_id&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;created_at&lt;/code&gt; and &lt;code&gt;modified_at&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;&lt;code&gt;source_run_id&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;metadata&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last one stings. If your LangSmith examples carry &lt;code&gt;{"source": "Wikipedia"}&lt;/code&gt; or a&lt;br&gt;
document ID, RAGAS cannot store it. Your options are to keep the original export and join&lt;br&gt;
back on array index (which is preserved in order), or fold the values into one of the&lt;br&gt;
named fields where it makes sense.&lt;/p&gt;

&lt;p&gt;Anything in &lt;code&gt;outputs&lt;/code&gt; that isn't &lt;code&gt;contexts&lt;/code&gt;, &lt;code&gt;answer&lt;/code&gt; or &lt;code&gt;reference&lt;/code&gt; is dropped too —&lt;br&gt;
including &lt;code&gt;outputs.ground_truth&lt;/code&gt;, if that's what your pipeline called the reference answer.&lt;/p&gt;
&lt;h2&gt;
  
  
  Why the question column comes out empty
&lt;/h2&gt;

&lt;p&gt;This is the failure that costs the most time, because nothing warns you.&lt;/p&gt;

&lt;p&gt;A converter reads &lt;code&gt;inputs.question&lt;/code&gt;. If your export says &lt;code&gt;inputs.query&lt;/code&gt; or &lt;code&gt;inputs.input&lt;/code&gt;,&lt;br&gt;
the structure is still detected as LangSmith — &lt;code&gt;inputs&lt;/code&gt; and &lt;code&gt;outputs&lt;/code&gt; are both present —&lt;br&gt;
but every &lt;code&gt;user_input&lt;/code&gt; entry is empty. RAGAS will load it. The metrics will run. The&lt;br&gt;
scores will be meaningless.&lt;/p&gt;

&lt;p&gt;Two fixes: rename the key in the export before converting, or rename the column after.&lt;br&gt;
Always check that &lt;code&gt;user_input&lt;/code&gt; is non-empty before you trust any number downstream:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;

&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"converted.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;empty&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"user_input"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;empty&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"empty questions at rows:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;empty&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  A detail about &lt;code&gt;split&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;LangSmith's &lt;code&gt;ExampleCreate&lt;/code&gt; and &lt;code&gt;ExampleUpdate&lt;/code&gt; accept a &lt;code&gt;split&lt;/code&gt; field, so you can tag&lt;br&gt;
examples as &lt;code&gt;train&lt;/code&gt; / &lt;code&gt;test&lt;/code&gt; when you write them. The &lt;code&gt;Example&lt;/code&gt; read model does not&lt;br&gt;
expose it.&lt;/p&gt;

&lt;p&gt;Practically: don't count on splits surviving an export-and-convert round trip. If you&lt;br&gt;
need them in RAGAS, carry them yourself and slice the dataset before evaluation.&lt;/p&gt;
&lt;h2&gt;
  
  
  Converting by hand
&lt;/h2&gt;

&lt;p&gt;If every example has the four expected keys, this is enough:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;ragas&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;EvaluationDataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;SingleTurnSample&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;langsmith&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Client&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;examples&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;list_examples&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataset_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"my-eval-set"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;samples&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;ex&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;examples&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;inputs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;outputs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ex&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inputs&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ex&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;outputs&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nc"&gt;SingleTurnSample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;user_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;inputs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"question"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;retrieved_contexts&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;outputs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"contexts"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;outputs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"answer"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;reference&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;outputs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"reference"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;dataset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;EvaluationDataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;samples&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This breaks the moment one example is missing a key — which is the usual reason to run&lt;br&gt;
the JSON through a converter first and read the report instead of the traceback.&lt;/p&gt;

&lt;p&gt;If you export from the UI you get JSONL, so read it line by line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;records&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"dataset.jsonl"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The UI also offers CSV and OpenAI's fine-tuning format. RAGAS wants neither.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you can evaluate afterwards
&lt;/h2&gt;

&lt;p&gt;If your LangSmith examples never logged &lt;code&gt;outputs.reference&lt;/code&gt;, the converted dataset has no&lt;br&gt;
&lt;code&gt;reference&lt;/code&gt; column, and these cannot run: Context Recall, Context Entities Recall, Answer&lt;br&gt;
Correctness, Factual Correctness, Semantic Similarity, Noise Sensitivity.&lt;/p&gt;

&lt;p&gt;These still work: Faithfulness, Response Relevancy, Aspect Critic, and&lt;br&gt;
&lt;code&gt;LLMContextPrecisionWithoutReference&lt;/code&gt; — which scores chunks against your own generated&lt;br&gt;
answer rather than a gold one.&lt;/p&gt;

&lt;p&gt;Worth knowing before you spend an hour generating references you may not need for a first&lt;br&gt;
pass.&lt;/p&gt;

&lt;h2&gt;
  
  
  Doing it without the hand-editing
&lt;/h2&gt;

&lt;p&gt;I built a converter for this. It takes RAGAS modern / current / legacy, TruLens or&lt;br&gt;
LangSmith JSON, converts to whichever target you pick, and &lt;strong&gt;prints every field it&lt;br&gt;
renamed, flattened, nested or dropped&lt;/strong&gt; — so the six lost fields show up in a list&lt;br&gt;
instead of disappearing.&lt;/p&gt;

&lt;p&gt;It also flags the two failures that corrupt metrics silently rather than throwing:&lt;br&gt;
unequal parallel arrays, and &lt;code&gt;contexts&lt;/code&gt; stored as a string instead of a list of strings.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://ragasconverter.com/langsmith-to-ragas/" rel="noopener noreferrer"&gt;ragasconverter.com&lt;/a&gt;&lt;/strong&gt; — runs entirely&lt;br&gt;
in the browser, no upload, no signup.&lt;/p&gt;




&lt;p&gt;Field names above are read from &lt;code&gt;langsmith/schemas.py&lt;/code&gt; (&lt;code&gt;ExampleBase&lt;/code&gt;, &lt;code&gt;Example&lt;/code&gt;,&lt;br&gt;
&lt;code&gt;ExampleCreate&lt;/code&gt;) and&lt;br&gt;
&lt;a href="https://github.com/vibrantlabsai/ragas/blob/main/src/ragas/dataset_schema.py" rel="noopener noreferrer"&gt;&lt;code&gt;src/ragas/dataset_schema.py&lt;/code&gt;&lt;/a&gt;&lt;br&gt;
(&lt;code&gt;SingleTurnSample&lt;/code&gt;). Both projects change these between releases — I checked against&lt;br&gt;
&lt;code&gt;langsmith&lt;/code&gt; 0.15 and current RAGAS &lt;code&gt;main&lt;/code&gt;, and you should pin your own versions before&lt;br&gt;
trusting the table.&lt;/p&gt;

</description>
      <category>rag</category>
      <category>llm</category>
      <category>langsmith</category>
      <category>python</category>
    </item>
    <item>
      <title>Which RAGAS metrics can you run without ground truth?</title>
      <dc:creator>kai wang</dc:creator>
      <pubDate>Thu, 19 Nov 2026 09:28:48 +0000</pubDate>
      <link>https://dev.to/kai_wang_52aff506bc3184b8/which-ragas-metrics-can-you-run-without-ground-truth-4fc7</link>
      <guid>https://dev.to/kai_wang_52aff506bc3184b8/which-ragas-metrics-can-you-run-without-ground-truth-4fc7</guid>
      <description>&lt;h1&gt;
  
  
  Which RAGAS metrics can you run without ground truth?
&lt;/h1&gt;

&lt;p&gt;Short answer: &lt;strong&gt;three of ten run with no reference at all, one more can if you pick a&lt;br&gt;
different class, and one is commonly miscategorised.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;But the useful answer is the table, because "do I need ground truth" is really "which&lt;br&gt;
columns do I have". Every RAGAS metric declares its required columns, and if one is&lt;br&gt;
missing the metric does not degrade — it refuses to run.&lt;/p&gt;
&lt;h2&gt;
  
  
  The table
&lt;/h2&gt;

&lt;p&gt;Ten metrics, checked against the source in&lt;br&gt;
&lt;a href="https://github.com/vibrantlabsai/ragas/tree/main/src/ragas/metrics" rel="noopener noreferrer"&gt;&lt;code&gt;src/ragas/metrics/&lt;/code&gt;&lt;/a&gt;&lt;br&gt;
rather than against tutorials:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Required columns&lt;/th&gt;
&lt;th&gt;Reference?&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Faithfulness&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;response&lt;/code&gt;, &lt;code&gt;retrieved_contexts&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Response Relevancy&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;response&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Aspect Critic / Rubrics&lt;/td&gt;
&lt;td&gt;all optional — you define the rubric&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context Precision &lt;em&gt;(default)&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;retrieved_contexts&lt;/code&gt;, &lt;code&gt;reference&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context Precision &lt;em&gt;(&lt;code&gt;WithoutReference&lt;/code&gt; variant)&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;response&lt;/code&gt;, &lt;code&gt;retrieved_contexts&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;No&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context Recall&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;retrieved_contexts&lt;/code&gt;, &lt;code&gt;reference&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context Entities Recall&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;reference&lt;/code&gt;, &lt;code&gt;retrieved_contexts&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Answer Correctness&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;response&lt;/code&gt;, &lt;code&gt;reference&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Factual Correctness&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;response&lt;/code&gt;, &lt;code&gt;reference&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Semantic Similarity&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;response&lt;/code&gt;, &lt;code&gt;reference&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Noise Sensitivity&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;response&lt;/code&gt;, &lt;code&gt;reference&lt;/code&gt;, &lt;code&gt;retrieved_contexts&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three things in that table cost me real time.&lt;/p&gt;
&lt;h2&gt;
  
  
  1. Context Entities Recall needs &lt;code&gt;reference&lt;/code&gt;, not &lt;code&gt;reference_contexts&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;The name sounds like it compares entity sets between gold chunks and retrieved chunks.&lt;br&gt;
It does not. &lt;code&gt;ContextEntityRecall&lt;/code&gt; takes the &lt;strong&gt;reference answer&lt;/strong&gt;, extracts entities from&lt;br&gt;
it, and measures what fraction of those entities appear in the retrieved chunks:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Context Entity Recall = |entities(retrieved) ∩ entities(reference)| / |entities(reference)|
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Its declared columns are &lt;code&gt;{"reference", "retrieved_contexts"}&lt;/code&gt;. So if you have gold&lt;br&gt;
chunks (&lt;code&gt;reference_contexts&lt;/code&gt;) but no reference answer, this metric will not run —&lt;br&gt;
and plenty of write-ups, including an earlier version of my own field table, say the&lt;br&gt;
opposite.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;NonLLMContextPrecisionWithReference&lt;/code&gt; is the metric that actually uses&lt;br&gt;
&lt;code&gt;reference_contexts&lt;/code&gt;, if that is what you were reaching for.&lt;/p&gt;
&lt;h2&gt;
  
  
  2. Context Precision has a reference-free variant
&lt;/h2&gt;

&lt;p&gt;The default &lt;code&gt;ContextPrecision&lt;/code&gt; requires a &lt;code&gt;reference&lt;/code&gt;. But RAGAS also ships:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;ragas&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;LLMContextPrecisionWithoutReference&lt;/span&gt;

&lt;span class="n"&gt;scorer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LLMContextPrecisionWithoutReference&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;scorer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ascore&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;user_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"What is RAG?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"RAG retrieves documents, then generates an answer."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;retrieved_contexts&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"RAG combines retrieval with generation."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It judges whether each retrieved chunk was useful in arriving at &lt;em&gt;your own&lt;/em&gt; generated&lt;br&gt;
answer, instead of at a gold answer. That makes it usable before you have any labels —&lt;br&gt;
which is exactly the situation most teams are in when they first try to evaluate&lt;br&gt;
retrieval.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;ContextUtilization&lt;/code&gt; subclasses the same class if you want the score named for what it&lt;br&gt;
measures.&lt;/p&gt;
&lt;h2&gt;
  
  
  3. Aspect Critic's columns are all optional
&lt;/h2&gt;

&lt;p&gt;Aspect Critic declares every column as &lt;code&gt;:optional&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"user_input:optional"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"response:optional"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"retrieved_contexts:optional"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="s2"&gt;"reference:optional"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"reference_contexts:optional"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because the metric is whatever criterion you write. A "is the answer polite" rubric needs&lt;br&gt;
only &lt;code&gt;response&lt;/code&gt;. A "does the answer cite the retrieved policy section" rubric needs&lt;br&gt;
&lt;code&gt;response&lt;/code&gt; and &lt;code&gt;retrieved_contexts&lt;/code&gt;. The dependency is in your prompt, not in the schema,&lt;br&gt;
so no static table can tell you whether your rubric will work — read your own criteria.&lt;/p&gt;
&lt;h2&gt;
  
  
  What "no reference" actually costs you
&lt;/h2&gt;

&lt;p&gt;Being able to run a metric is not the same as being able to answer your question.&lt;/p&gt;

&lt;p&gt;Without a reference you can measure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Groundedness&lt;/strong&gt; — Faithfulness: is every claim in the answer supported by the chunks&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Relevance&lt;/strong&gt; — Response Relevancy: does the answer address the question&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chunk usefulness&lt;/strong&gt; — &lt;code&gt;LLMContextPrecisionWithoutReference&lt;/code&gt;: were the retrieved chunks
actually needed to produce this answer&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What you cannot measure without a reference:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Recall&lt;/strong&gt; — did the retriever find everything it should have? Context Recall and
Context Entities Recall both need a target to compare against.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Correctness&lt;/strong&gt; — Answer Correctness, Factual Correctness, Semantic Similarity all
compare against a gold answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Noise robustness&lt;/strong&gt; — Noise Sensitivity needs to know which claims are &lt;em&gt;wrong&lt;/em&gt;, which
requires the truth.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Notice the shape of that split: &lt;strong&gt;precision can be judged against your own output, recall&lt;br&gt;
cannot.&lt;/strong&gt; If your retrieval returned three good chunks out of ten that existed, nothing in&lt;br&gt;
the answer tells you about the seven you missed. That is not a RAGAS limitation, it is&lt;br&gt;
information you do not have.&lt;/p&gt;

&lt;p&gt;So the honest framing is not "how do I avoid ground truth" but "what can I learn before I&lt;br&gt;
have it". The answer: whether your answer is faithful and on-topic, and whether your&lt;br&gt;
chunks were useful. That is enough to catch a broken prompt or a badly-tuned retriever&lt;br&gt;
top-k, and it is not enough to claim your system is accurate.&lt;/p&gt;
&lt;h2&gt;
  
  
  Checking your own dataset
&lt;/h2&gt;

&lt;p&gt;Before running anything, verify the two things that fail silently rather than throwing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;

&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"eval_set.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 1. every parallel array must be the same length&lt;/span&gt;
&lt;span class="n"&gt;lengths&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lengths&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ragged arrays — rows will shift:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lengths&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 2. contexts must be a list of strings, not a string&lt;/span&gt;
&lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"retrieved_contexts"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
       &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"contexts not a list at rows:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Neither of these raises. Both produce scores that look plausible and mean nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you want this without the code
&lt;/h2&gt;

&lt;p&gt;I built a converter that reads a dataset in any of the five common shapes (RAGAS modern,&lt;br&gt;
current and legacy, TruLens, LangSmith), converts it, and reports &lt;strong&gt;which of these ten&lt;br&gt;
metrics your dataset can run right now&lt;/strong&gt; — naming the exact fields each missing metric&lt;br&gt;
still needs.&lt;/p&gt;

&lt;p&gt;It runs entirely in the browser: &lt;strong&gt;&lt;a href="https://ragasconverter.com/" rel="noopener noreferrer"&gt;ragasconverter.com&lt;/a&gt;&lt;/strong&gt;.&lt;br&gt;
No upload, no signup, no server.&lt;/p&gt;

&lt;p&gt;The standalone page for this topic, if you only want the table:&lt;br&gt;
&lt;a href="https://ragasconverter.com/ragas-without-ground-truth/" rel="noopener noreferrer"&gt;ragasconverter.com/ragas-without-ground-truth&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Column requirements above are read from&lt;br&gt;
&lt;a href="https://github.com/vibrantlabsai/ragas/tree/main/src/ragas/metrics" rel="noopener noreferrer"&gt;&lt;code&gt;src/ragas/metrics/&lt;/code&gt;&lt;/a&gt;&lt;br&gt;
and cross-checked against the&lt;br&gt;
&lt;a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/" rel="noopener noreferrer"&gt;available metrics docs&lt;/a&gt;.&lt;br&gt;
These declarations have changed between releases — pin your RAGAS version before trusting&lt;br&gt;
any table, including this one.&lt;/p&gt;

</description>
      <category>rag</category>
      <category>llm</category>
      <category>ai</category>
      <category>python</category>
    </item>
    <item>
      <title>RAGAS renamed ground_truth to reference — how to migrate your eval dataset</title>
      <dc:creator>kai wang</dc:creator>
      <pubDate>Thu, 19 Nov 2026 09:27:57 +0000</pubDate>
      <link>https://dev.to/kai_wang_52aff506bc3184b8/ragas-renamed-groundtruth-to-reference-how-to-migrate-your-eval-dataset-4nm7</link>
      <guid>https://dev.to/kai_wang_52aff506bc3184b8/ragas-renamed-groundtruth-to-reference-how-to-migrate-your-eval-dataset-4nm7</guid>
      <description>&lt;h1&gt;
  
  
  RAGAS renamed &lt;code&gt;ground_truth&lt;/code&gt; to &lt;code&gt;reference&lt;/code&gt;
&lt;/h1&gt;

&lt;p&gt;I lost an afternoon to this, so here is the short version.&lt;/p&gt;

&lt;p&gt;A RAGAS evaluation dataset built a few months ago fails to load today. The JSON is&lt;br&gt;
valid. The keys look right. Nothing about the file changed — the library did.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;ground_truth&lt;/code&gt; became &lt;code&gt;reference&lt;/code&gt;. And while the field was being renamed, the sample&lt;br&gt;
schema also moved the question and answer: &lt;code&gt;question&lt;/code&gt; became &lt;code&gt;user_input&lt;/code&gt;, &lt;code&gt;answer&lt;/code&gt;&lt;br&gt;
became &lt;code&gt;response&lt;/code&gt;. If your pipeline still emits the old names, you get a validation&lt;br&gt;
error that points at the dataset rather than at the rename.&lt;/p&gt;
&lt;h2&gt;
  
  
  Why it breaks the way it does
&lt;/h2&gt;

&lt;p&gt;RAGAS stores a dataset as &lt;strong&gt;columnar parallel arrays&lt;/strong&gt; — one array per field, one entry&lt;br&gt;
per sample. That shape hides renames badly. If a key is missing, the library does not&lt;br&gt;
tell you "I expected &lt;code&gt;reference&lt;/code&gt;". It tells you the dataset is invalid, and you go&lt;br&gt;
looking through your data for a problem that is actually in your column names.&lt;/p&gt;

&lt;p&gt;There are three schema generations in the wild, and they disagree:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Format&lt;/th&gt;
&lt;th&gt;Question&lt;/th&gt;
&lt;th&gt;Contexts&lt;/th&gt;
&lt;th&gt;Answer&lt;/th&gt;
&lt;th&gt;Reference&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Legacy&lt;/td&gt;
&lt;td&gt;&lt;code&gt;question&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;contexts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;answer&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ground_truth&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Current&lt;/td&gt;
&lt;td&gt;&lt;code&gt;question&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;contexts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;answer&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;reference&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Modern (&lt;code&gt;SingleTurnSample&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;user_input&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;retrieved_contexts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;response&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;reference&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every tutorial written before the rename uses the first row. Every dataset generated by&lt;br&gt;
&lt;code&gt;TestsetGenerator&lt;/code&gt; today uses the third. They are all "RAGAS datasets", and none of them&lt;br&gt;
load in the same code.&lt;/p&gt;
&lt;h2&gt;
  
  
  Migrating by hand
&lt;/h2&gt;

&lt;p&gt;If your data is already a DataFrame, the rename is three lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pandas&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;

&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"eval_set.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"question"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user_input"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"answer"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"response"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"ground_truth"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"reference"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;to_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"eval_set_v2.json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orient&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"list"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;orient="list"&lt;/code&gt; matters. That is the columnar shape RAGAS expects. &lt;code&gt;orient="records"&lt;/code&gt;&lt;br&gt;
produces one object per row, which is what TruLens and LangSmith use and what RAGAS&lt;br&gt;
will reject.&lt;/p&gt;

&lt;p&gt;Two things to check before you move on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Contexts must be a list of strings, not a string.&lt;/strong&gt; This is the most common silent&lt;br&gt;
failure. A single string is valid JSON, loads fine, and then produces nonsense scores&lt;br&gt;
because each character gets treated as a separate context.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;

&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"eval_set_v2.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;bad_contexts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"retrieved_contexts"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;bad_contexts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"contexts not a list at rows:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bad_contexts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Every array must be the same length.&lt;/strong&gt; A short array shifts every later row onto the&lt;br&gt;
wrong record — the dataset still loads, and every score you compute afterwards is&lt;br&gt;
measuring the wrong pairing.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;lengths&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lengths&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ragged arrays:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lengths&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Neither of these throws. Both corrupt results quietly. Run both checks on every dataset&lt;br&gt;
you convert.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;contexts&lt;/code&gt; vs &lt;code&gt;retrieved_contexts&lt;/code&gt; is not just a rename
&lt;/h2&gt;

&lt;p&gt;This one cost me more than the &lt;code&gt;ground_truth&lt;/code&gt; change.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;TestsetGenerator&lt;/code&gt; produces &lt;code&gt;reference_contexts&lt;/code&gt; — the gold chunks a question was built&lt;br&gt;
from. Your pipeline produces &lt;code&gt;retrieved_contexts&lt;/code&gt; — what your retriever actually&lt;br&gt;
returned. They are different things, and Context Recall is defined as the overlap&lt;br&gt;
between them.&lt;/p&gt;

&lt;p&gt;If you rename &lt;code&gt;contexts&lt;/code&gt; to &lt;code&gt;retrieved_contexts&lt;/code&gt; blindly, you may be feeding gold chunks&lt;br&gt;
in as retrieved chunks. Retrieval scores then describe the generator, not your retriever,&lt;br&gt;
and they will look excellent for reasons that have nothing to do with your system.&lt;/p&gt;

&lt;p&gt;Check which one you have before renaming.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which metrics you lose without a reference
&lt;/h2&gt;

&lt;p&gt;Not everything needs &lt;code&gt;reference&lt;/code&gt;. If you have no reference answers, you can still run:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Faithfulness&lt;/strong&gt; — is the answer grounded in the retrieved context&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Response Relevancy&lt;/strong&gt; — does the answer address the question&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Aspect Critic / Rubrics&lt;/strong&gt; — does the answer satisfy a criterion you define&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These need a &lt;code&gt;reference&lt;/code&gt; and cannot run without one:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Context Recall&lt;/li&gt;
&lt;li&gt;Context Precision&lt;/li&gt;
&lt;li&gt;Answer Correctness&lt;/li&gt;
&lt;li&gt;Factual Correctness&lt;/li&gt;
&lt;li&gt;Semantic Similarity&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So a dataset with no references is still useful — it just measures generation quality and&lt;br&gt;
not retrieval quality. That distinction is worth knowing before you spend an hour&lt;br&gt;
generating references.&lt;/p&gt;

&lt;h2&gt;
  
  
  The shape mismatch across frameworks
&lt;/h2&gt;

&lt;p&gt;The same four columns, five different names, depending on the framework:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Framework&lt;/th&gt;
&lt;th&gt;Question&lt;/th&gt;
&lt;th&gt;Contexts&lt;/th&gt;
&lt;th&gt;Answer&lt;/th&gt;
&lt;th&gt;Reference&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;RAGAS modern&lt;/td&gt;
&lt;td&gt;&lt;code&gt;user_input&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;retrieved_contexts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;response&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;reference&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RAGAS current&lt;/td&gt;
&lt;td&gt;&lt;code&gt;question&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;contexts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;answer&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;reference&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RAGAS legacy&lt;/td&gt;
&lt;td&gt;&lt;code&gt;question&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;contexts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;answer&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ground_truth&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TruLens&lt;/td&gt;
&lt;td&gt;&lt;code&gt;query&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;expected_chunks&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;expected_response&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LangSmith&lt;/td&gt;
&lt;td&gt;&lt;code&gt;inputs.question&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;outputs.contexts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;outputs.answer&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;outputs.reference&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Note the structural differences too, not just the names. RAGAS is columnar. TruLens is&lt;br&gt;
one record per line item. LangSmith nests everything under &lt;code&gt;inputs&lt;/code&gt; / &lt;code&gt;outputs&lt;/code&gt;. Moving&lt;br&gt;
a dataset between them is a reshape as much as a rename — and &lt;code&gt;reference&lt;/code&gt; has no TruLens&lt;br&gt;
equivalent at all, so that column gets dropped on the way over.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you'd rather not hand-edit JSON
&lt;/h2&gt;

&lt;p&gt;I got tired of doing this by hand, so I built a converter that runs entirely in the&lt;br&gt;
browser: &lt;strong&gt;&lt;a href="https://ragasconverter.com/" rel="noopener noreferrer"&gt;ragasconverter.com&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Paste a dataset in any of the five shapes, pick a target, and it converts — then prints&lt;br&gt;
every field it renamed, flattened, nested or dropped, so nothing disappears without you&lt;br&gt;
seeing it. It also runs the two validation checks above (ragged arrays, string contexts)&lt;br&gt;
and reports which RAGAS metrics your dataset can actually run right now, with the record&lt;br&gt;
index of each problem.&lt;/p&gt;

&lt;p&gt;No signup, no upload, no server. It is a client-side bundle, so eval data never leaves&lt;br&gt;
the tab — which matters when the dataset contains internal documents.&lt;/p&gt;

&lt;p&gt;The field mapping reference, if you just want the table without the tool:&lt;br&gt;
&lt;a href="https://ragasconverter.com/ragas-dataset-format/" rel="noopener noreferrer"&gt;ragasconverter.com/ragas-dataset-format&lt;/a&gt;.&lt;br&gt;
And the metric-by-metric breakdown:&lt;br&gt;
&lt;a href="https://ragasconverter.com/ragas-without-ground-truth/" rel="noopener noreferrer"&gt;ragasconverter.com/ragas-without-ground-truth&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Field names above follow the&lt;br&gt;
&lt;a href="https://docs.ragas.io/en/stable/concepts/datasets/" rel="noopener noreferrer"&gt;RAGAS docs&lt;/a&gt; and&lt;br&gt;
&lt;a href="https://github.com/vibrantlabsai/ragas/blob/main/src/ragas/dataset_schema.py" rel="noopener noreferrer"&gt;&lt;code&gt;ragas/dataset_schema.py&lt;/code&gt;&lt;/a&gt;.&lt;br&gt;
Schema details are version-dependent — pin your RAGAS version before trusting any of&lt;br&gt;
this, including mine.&lt;/p&gt;

</description>
      <category>rag</category>
      <category>python</category>
      <category>llm</category>
      <category>ai</category>
    </item>
  </channel>
</rss>
