<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: jacob baruch</title>
    <description>The latest articles on DEV Community by jacob baruch (@jacob_baruch_2a2aec3f6e3c).</description>
    <link>https://dev.to/jacob_baruch_2a2aec3f6e3c</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3691918%2F179ed7f8-63b7-4ff0-9571-12d15e1ca430.png</url>
      <title>DEV Community: jacob baruch</title>
      <link>https://dev.to/jacob_baruch_2a2aec3f6e3c</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jacob_baruch_2a2aec3f6e3c"/>
    <language>en</language>
    <item>
      <title>Do you need bronze, silver and gold layers? Keep three things instead</title>
      <dc:creator>jacob baruch</dc:creator>
      <pubDate>Fri, 20 Nov 2026 09:30:12 +0000</pubDate>
      <link>https://dev.to/jacob_baruch_2a2aec3f6e3c/do-you-need-bronze-silver-and-gold-layers-keep-three-things-instead-5432</link>
      <guid>https://dev.to/jacob_baruch_2a2aec3f6e3c/do-you-need-bronze-silver-and-gold-layers-keep-three-things-instead-5432</guid>
      <description>&lt;p&gt;&lt;strong&gt;TL;DR.&lt;/strong&gt; Probably not, if you are a startup with no data person. The three layers, called the medallion architecture, are a recommended practice, not a requirement, and the real work of the middle one, making two systems agree, can start as one mapping table. What you should not skip are three things that are often impossible to bring back later: an untouched copy of data as it arrived, one model that keeps history instead of overwriting it, and written definitions your coding agent can read. Add a layer when mappings multiply or a second consumer with different needs shows up, not before.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5tolxbh9wmxz0yo46pea.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5tolxbh9wmxz0yo46pea.png" alt="Layers are optional, three things are not: what arrived, what was true when, what it means" width="800" height="392"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Illustration, not measured data.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  What is the layer cake, and what is it for?
&lt;/h2&gt;

&lt;p&gt;Data platforms often sort data into three layers. Bronze holds data as it arrived. Silver holds it cleaned and made to agree across sources. Gold holds tables shaped for reports. Databricks, whose platform made the pattern popular, calls it &lt;a href="https://docs.databricks.com/aws/lakehouse/medallion" rel="noopener noreferrer"&gt;a recommended best practice but not a requirement&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The silver layer does real work. Picture a startup that moved billing from PayPal to Stripe in March. Two systems now hold its revenue, with two customer IDs and two spellings of each plan. A yearly revenue number needs both, matched to one customer. The same happens when the app and the CRM each keep their own customer list. That work does not go away when someone calls layers unnecessary.&lt;/p&gt;

&lt;p&gt;The question for a startup is narrower: does that work need a whole layer of copied tables, or one mapping table and a view? Before your first data person, I think it is usually the second.&lt;/p&gt;
&lt;h2&gt;
  
  
  What can you not skip?
&lt;/h2&gt;

&lt;p&gt;I think the layers are optional and three things are not. Each one protects something you cannot rebuild later.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. A copy of what arrived, never edited.&lt;/strong&gt; Your billing tool sends an event. Your code reads it, converts it and saves the result. Picture a startup whose code read amounts in cents as dollars for two months. If only the converted numbers were kept, the history of revenue is wrong and stays wrong. If the original events were kept as they arrived, you fix the code and rebuild.&lt;br&gt;
Guardrail: store every incoming event or file in one table, as it arrived, with the time it arrived and where it came from. Engineers call this an append-only landing table. Create it as the admin user, so your app cannot update or delete rows there.&lt;br&gt;
Check: try to change one row with the login your app uses. It must fail.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. One model that keeps history.&lt;/strong&gt; Your app stores a customer's plan in one field and overwrites it when the plan changes. A month later someone asks how many customers paid on the first of October. The app cannot answer, because the old value is gone. We ran this on a small test database on 20 November 2026: today's table showed no paying customers, while the history table showed two on the date asked. No agent can recover a value that was never kept.&lt;br&gt;
Guardrail: keep a history table for the few things that change and matter, such as plan, price and status, with the date each value started and ended. Data engineers call this a type 2 slowly changing dimension.&lt;br&gt;
Check: ask for last month's number for one customer. If nobody can answer, history is not kept.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Written definitions.&lt;/strong&gt; An agent that reads your tables sees column names, not meaning. It does not know that a row is one paying account, not one person, or that test accounts should be left out.&lt;br&gt;
Guardrail: one short file per important table, in your repository, that says what one row is and how the main numbers are counted.&lt;br&gt;
Check: ask your agent "how many active customers do we have", then ask how it counted. Compare the answer with the file.&lt;/p&gt;
&lt;h2&gt;
  
  
  When does a layer earn its place?
&lt;/h2&gt;

&lt;p&gt;A layer is worth its cost when it removes work that repeats. I think four situations justify one. Two systems describe the same thing in many places, so mapping tables multiply and every report repeats the same joins. A second consumer appears with different needs, such as finance wanting closed months that never change while product wants today's numbers. A heavy reshaping runs every hour and is cheaper to store than to repeat. Or someone outside the team, such as an auditor, must follow each step from source to report. A &lt;a href="https://wherescape.com/blog/medallion-architecture-ai-readiness/" rel="noopener noreferrer"&gt;WhereScape recap&lt;/a&gt; of a practitioner panel, in August 2026, calls medallion a shared vocabulary, not a specification. WhereScape sells data warehouse automation. Its line is still a fair way to use the names: labels for stages, added when a stage is needed.&lt;br&gt;
Check: count customers in billing with no match in the app. If the count is above zero and growing, start a mapping table. If no second team reads a table and nothing rebuilds it on a schedule, skip the layer.&lt;/p&gt;

&lt;p&gt;What changes the answer is who consumes the data and what it costs to rebuild, more than company size.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Before launch.&lt;/strong&gt; Keep the copy of what arrived, history for plan and status, and one line saying what a row in each table is. Skip everything else.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First paying customers.&lt;/strong&gt; Add the definitions file for the tables your numbers come from. When a second system holds the same customers, after a billing migration or when sales starts using a CRM, add a mapping table. An agent can build the report tables on request from the history model.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Growing, with a first data person.&lt;/strong&gt; That person decides which reshaped tables to store, because by now some questions repeat every week. They inherit raw data and history they could never have recreated.&lt;/p&gt;
&lt;h2&gt;
  
  
  For your data team
&lt;/h2&gt;

&lt;p&gt;This part gets technical. The SQL below ran on PostgreSQL 16.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- 1. What arrived: one table, rows only added.&lt;/span&gt;
&lt;span class="c1"&gt;-- Run this block as the admin user, not the app login.&lt;/span&gt;
&lt;span class="k"&gt;create&lt;/span&gt; &lt;span class="k"&gt;table&lt;/span&gt; &lt;span class="n"&gt;raw_arrivals&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt;          &lt;span class="kt"&gt;bigserial&lt;/span&gt; &lt;span class="k"&gt;primary&lt;/span&gt; &lt;span class="k"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;received_at&lt;/span&gt; &lt;span class="kt"&gt;timestamptz&lt;/span&gt; &lt;span class="k"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;source&lt;/span&gt;      &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;          &lt;span class="c1"&gt;-- 'billing', 'crm', 'webhook:signup'&lt;/span&gt;
  &lt;span class="n"&gt;payload&lt;/span&gt;     &lt;span class="kt"&gt;jsonb&lt;/span&gt; &lt;span class="k"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- The table belongs to the admin user, so the app login can add and read, nothing else.&lt;/span&gt;
&lt;span class="k"&gt;grant&lt;/span&gt; &lt;span class="k"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="k"&gt;on&lt;/span&gt; &lt;span class="n"&gt;raw_arrivals&lt;/span&gt; &lt;span class="k"&gt;to&lt;/span&gt; &lt;span class="n"&gt;app_writer&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;grant&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt; &lt;span class="k"&gt;on&lt;/span&gt; &lt;span class="n"&gt;sequence&lt;/span&gt; &lt;span class="n"&gt;raw_arrivals_id_seq&lt;/span&gt; &lt;span class="k"&gt;to&lt;/span&gt; &lt;span class="n"&gt;app_writer&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- 2. What was true when: history kept by the database, not by app code.&lt;/span&gt;
&lt;span class="k"&gt;create&lt;/span&gt; &lt;span class="k"&gt;table&lt;/span&gt; &lt;span class="n"&gt;customers_history&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;email&lt;/span&gt; &lt;span class="kt"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt; &lt;span class="kt"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;valid_from&lt;/span&gt; &lt;span class="kt"&gt;timestamptz&lt;/span&gt; &lt;span class="k"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;valid_to&lt;/span&gt;   &lt;span class="kt"&gt;timestamptz&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;               &lt;span class="c1"&gt;-- null means current&lt;/span&gt;
  &lt;span class="n"&gt;deleted&lt;/span&gt;    &lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="k"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- security definer: the trigger writes history with the owner's rights, so the app&lt;/span&gt;
&lt;span class="c1"&gt;-- does not need, and does not get, permission to edit the history table.&lt;/span&gt;
&lt;span class="k"&gt;create&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;keep_customer_history&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;returns&lt;/span&gt; &lt;span class="k"&gt;trigger&lt;/span&gt; &lt;span class="n"&gt;language&lt;/span&gt; &lt;span class="n"&gt;plpgsql&lt;/span&gt;
&lt;span class="n"&gt;security&lt;/span&gt; &lt;span class="n"&gt;definer&lt;/span&gt; &lt;span class="k"&gt;set&lt;/span&gt; &lt;span class="n"&gt;search_path&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;public&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; $$
&lt;span class="k"&gt;begin&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tg_op&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'UPDATE'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s1"&gt;'DELETE'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
    &lt;span class="k"&gt;update&lt;/span&gt; &lt;span class="n"&gt;customers_history&lt;/span&gt; &lt;span class="k"&gt;set&lt;/span&gt; &lt;span class="n"&gt;valid_to&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
     &lt;span class="k"&gt;where&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;old&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;and&lt;/span&gt; &lt;span class="n"&gt;valid_to&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tg_op&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'INSERT'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s1"&gt;'UPDATE'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
    &lt;span class="k"&gt;insert&lt;/span&gt; &lt;span class="k"&gt;into&lt;/span&gt; &lt;span class="n"&gt;customers_history&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;valid_from&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;values&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;else&lt;/span&gt;
    &lt;span class="k"&gt;insert&lt;/span&gt; &lt;span class="k"&gt;into&lt;/span&gt; &lt;span class="n"&gt;customers_history&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;valid_from&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;deleted&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;values&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;old&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;old&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;old&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt; $$&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;create&lt;/span&gt; &lt;span class="k"&gt;trigger&lt;/span&gt; &lt;span class="n"&gt;customers_history_trg&lt;/span&gt;
&lt;span class="n"&gt;after&lt;/span&gt; &lt;span class="k"&gt;insert&lt;/span&gt; &lt;span class="k"&gt;or&lt;/span&gt; &lt;span class="k"&gt;update&lt;/span&gt; &lt;span class="k"&gt;or&lt;/span&gt; &lt;span class="k"&gt;delete&lt;/span&gt; &lt;span class="k"&gt;on&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;each&lt;/span&gt; &lt;span class="k"&gt;row&lt;/span&gt; &lt;span class="n"&gt;execute&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;keep_customer_history&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- Run once, so today's values are the starting point.&lt;/span&gt;
&lt;span class="k"&gt;insert&lt;/span&gt; &lt;span class="k"&gt;into&lt;/span&gt; &lt;span class="n"&gt;customers_history&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;valid_from&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- Paying customers on a past date.&lt;/span&gt;
&lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="nb"&gt;count&lt;/span&gt;&lt;span class="cm"&gt;(*) from customers_history
where valid_from &amp;lt;= '2026-10-01' and (valid_to is null or valid_to &amp;gt; '2026-10-01')
  and not deleted and plan &amp;lt;&amp;gt; 'cancelled';
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A trigger keeps history even when a script or a teammate updates the table outside the app. It adds one insert per change, which is small for tables like customers and plans. Do not put it on a table that changes thousands of times a second. A deleted customer leaves a row marked &lt;code&gt;deleted&lt;/code&gt;, so last month's counts stay right.&lt;/p&gt;

&lt;p&gt;When two systems hold the same customers, one mapping table and one check keep them in line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;create&lt;/span&gt; &lt;span class="k"&gt;table&lt;/span&gt; &lt;span class="n"&gt;customer_map&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;source&lt;/span&gt;      &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;          &lt;span class="c1"&gt;-- 'paypal', 'stripe', 'crm'&lt;/span&gt;
  &lt;span class="n"&gt;source_id&lt;/span&gt;   &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;customer_id&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;  &lt;span class="k"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;          &lt;span class="c1"&gt;-- your app's id&lt;/span&gt;
  &lt;span class="k"&gt;primary&lt;/span&gt; &lt;span class="k"&gt;key&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;source_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- Billing customers no one has matched yet. Should be zero, or shrinking.&lt;/span&gt;
&lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'customer'&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;stripe_customer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;count&lt;/span&gt;&lt;span class="cm"&gt;(*)
from raw_arrivals r
where source = 'billing'
  and not exists (select 1 from customer_map m
                  where m.source = 'stripe' and m.source_id = r.payload-&amp;gt;&amp;gt;'customer')
group by 1;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For the third item, a file such as &lt;code&gt;data/definitions/customers.md&lt;/code&gt; with three lines is enough to start: what one row is, which accounts are excluded, and how "active" and "paying" are counted. Point your agent's instructions file at that folder.&lt;/p&gt;

&lt;p&gt;In our test, a revenue table built with the cents bug summed to 6,800 dollars. Rebuilt from the raw copy with the fix, it summed to 68. An update to the raw table from the app login failed with "permission denied". One run on made-up data, to show the mechanism.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check it in 10 minutes
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Ask where incoming billing and signup events are stored before your code changes them. If the answer is nowhere, add the arrivals table.&lt;/li&gt;
&lt;li&gt;Pick one field that changes and matters, usually plan or status, and ask whether its old values are kept. If not, add the history table and trigger above.&lt;/li&gt;
&lt;li&gt;Ask your agent how it counts active customers, and write its answer into a definitions file once a person has corrected it.&lt;/li&gt;
&lt;li&gt;No coding agent and no repository? This step is for you. Open your database host's SQL editor as the admin user and paste the second block, changing the column names to match your customers table, and run the last insert once. From then on, every plan change is kept, and the app can still save changes without being able to edit the history.&lt;/li&gt;
&lt;li&gt;The engineer who owns the app database owns these three items.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;See where your setup stands with the &lt;a href="https://goagentdata.com/#self-check" rel="noopener noreferrer"&gt;two minute self-check&lt;/a&gt;, then read &lt;a href="https://goagentdata.com/posts/data-contracts-test-meaning" rel="noopener noreferrer"&gt;the data contracts post&lt;/a&gt; for checks on what each field means.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://goagentdata.com/posts/skip-the-layer-cake" rel="noopener noreferrer"&gt;goagentdata.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>postgres</category>
      <category>startup</category>
      <category>ai</category>
    </item>
    <item>
      <title>How to check your AI's data answers: evals for each stage</title>
      <dc:creator>jacob baruch</dc:creator>
      <pubDate>Mon, 26 Oct 2026 18:32:53 +0000</pubDate>
      <link>https://dev.to/jacob_baruch_2a2aec3f6e3c/how-to-check-your-ais-data-answers-evals-for-each-stage-4pfn</link>
      <guid>https://dev.to/jacob_baruch_2a2aec3f6e3c/how-to-check-your-ais-data-answers-evals-for-each-stage-4pfn</guid>
      <description>&lt;p&gt;&lt;strong&gt;TL;DR.&lt;/strong&gt; Your assistant gives a number, and you cannot tell if it is right. An eval is a repeatable test of its answers against something you trust, and no single eval is enough. Trusted queries show that something broke, checks without an answer key catch errors on every answer, and a weekly sample finds what you never tested. An AI judge scales the review once people have checked it.&lt;/p&gt;

&lt;p&gt;Start with trusted queries before launch, and add layers as money and askers grow.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl104m1dflm8fj16yd786.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl104m1dflm8fj16yd786.png" alt="Four layers of evals added by stage: a fixed set with trusted queries, checks without an answer key, a weekly sample of real questions, and an AI judge checked against people" width="799" height="433"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is one test file not enough?
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://goagentdata.com/posts/ai-analytics-more-data-work" rel="noopener noreferrer"&gt;previous post&lt;/a&gt; argued that judging answers is the new data work once AI makes answers cheap. This post shows how to do that judging without reading every query.&lt;/p&gt;

&lt;p&gt;Here is how I think about it. Each kind of eval catches one failure and misses others, so you add them in layers as the stakes grow.&lt;/p&gt;

&lt;p&gt;The problem is not new, and it is not solved. The &lt;a href="https://arxiv.org/abs/2305.03111" rel="noopener noreferrer"&gt;BIRD benchmark&lt;/a&gt; (2023) tests text-to-SQL, turning a question into a database query, on large and messy real databases. It scores execution accuracy: run both queries and compare the results. In 2023, people scored 92.96 percent and the best model, ChatGPT, scored 40.08 percent. In June 2026, Google Research &lt;a href="https://www.marktechpost.com/2026/06/12/google-releases-gemini-sql2-gemini-3-1-pro-text-to-sql-scores-80-04-on-bird-single-model-leaderboard/" rel="noopener noreferrer"&gt;announced 80.04 percent&lt;/a&gt; for its Gemini-SQL2 on the BIRD single-model track, a figure Google reported itself. Models improved a lot, and are still below people on messy data.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does a fixed set of trusted queries catch?
&lt;/h2&gt;

&lt;p&gt;A fixed set is a short list of real questions, each paired with a query your team approved. The eval runs that query on today's data, runs the assistant's query, and compares the two results.&lt;/p&gt;

&lt;p&gt;A frozen number goes wrong the day new invoices arrive. A trusted query stays right as data changes, and needs an update only when a definition changes.&lt;/p&gt;

&lt;p&gt;The limit: it covers only the questions you thought of, and the key itself can be wrong. &lt;a href="https://www.vldb.org/cidrdb/papers/2026/p5-jin.pdf" rel="noopener noreferrer"&gt;Researchers at the University of Illinois Urbana-Champaign&lt;/a&gt; (CIDR 2026) audited public text-to-SQL benchmarks. They found annotation errors in 52.8 percent of BIRD Mini-Dev problems. Among the 121 Spider 2.0-Snow problems with released reference queries, 66.1 percent had errors. After fixing 100 BIRD Dev problems, five agents' scores changed by between 3 percent lower and 31 percent higher, and rankings moved by up to three places. If expert keys go wrong that often, your trusted queries need an owner and a review date.&lt;/p&gt;

&lt;h2&gt;
  
  
  What can you check without an answer key?
&lt;/h2&gt;

&lt;p&gt;Some rules must hold whatever the right answer is, so they can run on every live answer.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Same question, different words.&lt;/strong&gt; Both phrasings give the same number.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Parts add up.&lt;/strong&gt; Revenue by region sums to total revenue. Use this for amounts, not for customers who can sit in two groups.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sources agree.&lt;/strong&gt; Billing and your CRM agree within a margin you set in advance.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The definition exists.&lt;/strong&gt; The cited definition is in your metric files.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It says when it cannot answer.&lt;/strong&gt; Ask something your data cannot answer. It should say so.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Jumps get flagged.&lt;/strong&gt; A change from last period beyond a threshold you set per metric goes to a person.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is metamorphic testing: checking how answers relate, not whether each matches a key. A March 2026 study, &lt;a href="https://arxiv.org/abs/2603.13173" rel="noopener noreferrer"&gt;"Semantic Invariance in Agentic AI"&lt;/a&gt;, rephrased multi-step scientific reasoning problems for seven models. Bigger models were not reliably more consistent. The most stable one kept its result nearly unchanged in 79.6 percent of cases. The test used only 19 science problems, not data questions.&lt;/p&gt;

&lt;p&gt;The limit: if the assistant always uses the wrong definition, every check passes.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does a weekly sample of real questions show?
&lt;/h2&gt;

&lt;p&gt;In my view, this is the layer small teams skip and need most. Each week, take a random handful of questions people actually asked. A person who knows the business marks each answer right or wrong, and gives each miss one cause:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Missing definition.&lt;/strong&gt; No written meaning existed for the term.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unclear instruction.&lt;/strong&gt; A definition exists, but nothing says to use it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Wrong data.&lt;/strong&gt; It used the wrong or an incomplete table.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stale trusted answer.&lt;/strong&gt; The assistant was right and the key was out of date.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each miss becomes a new case in the fixed set. This is where new question types show up first.&lt;/p&gt;

&lt;h2&gt;
  
  
  When can an AI judge do the review?
&lt;/h2&gt;

&lt;p&gt;An AI judge is a second model that grades each answer. For data answers, I think it should answer narrow questions: did the answer use the definition it cites, and does the query fit the question?&lt;/p&gt;

&lt;p&gt;In the &lt;a href="https://arxiv.org/abs/2306.05685" rel="noopener noreferrer"&gt;MT-Bench paper&lt;/a&gt; (2023), strong judges agreed with human preferences over 80 percent of the time, but favored the first answer shown, longer answers and their own. A June 2026 study, &lt;a href="https://arxiv.org/abs/2606.19544" rel="noopener noreferrer"&gt;"Reliability without Validity"&lt;/a&gt;, tested 21 judges from nine providers across about 541,000 judgments. Plain match rate overstated judge quality: on MT-Bench, agreement beyond chance (kappa) was 33 to 41 points lower. Judges repeated their verdicts reliably on reruns, yet two judges used in production favored answers by position.&lt;/p&gt;

&lt;p&gt;So calibrate it on your own answers: people and the judge mark the same set, and you measure agreement beyond chance. The limit: a stable judge is not a correct one. Never let it decide alone on money questions.&lt;/p&gt;

&lt;h2&gt;
  
  
  What should you add at each stage?
&lt;/h2&gt;

&lt;p&gt;I think the layers should arrive in this order.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Before launch.&lt;/strong&gt; Use the fixed set only: a handful of real questions, each with a trusted query and the definition behind it. Rephrase the key questions by hand and compare. If your people ask in Hebrew, rephrase in Hebrew too.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First paying customers.&lt;/strong&gt; Rerun the set automatically on every change to the model, the instructions or a definition, and once a week. Add the checks without an answer key to every money number. Start the weekly sample and the cause list.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Growing with a first data person.&lt;/strong&gt; Add an AI judge calibrated on human marks. Track the pass rate over time; only a fixed set makes improvement visible. Keep a set of cases you never use while tuning instructions, and fix (pin) the model version, recording it with each run.&lt;/p&gt;

&lt;p&gt;In my estimate, not a measured figure, the cost looks like this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Who owns it&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fixed set&lt;/td&gt;
&lt;td&gt;The owner of each definition&lt;/td&gt;
&lt;td&gt;About an hour to write each case&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Checks&lt;/td&gt;
&lt;td&gt;Whoever runs the assistant&lt;/td&gt;
&lt;td&gt;A one-off setup&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Weekly sample&lt;/td&gt;
&lt;td&gt;Someone who knows the business&lt;/td&gt;
&lt;td&gt;About 30 minutes a week&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AI judge&lt;/td&gt;
&lt;td&gt;The first data person&lt;/td&gt;
&lt;td&gt;A set of human-marked answers to calibrate&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What goes wrong, and how do you catch it?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The trusted answers are wrong.&lt;/strong&gt; The key is wrong, and the team changes the assistant to match it.&lt;br&gt;
Guardrail: every trusted query has a named owner, a definition and a review date. Write it without the assistant's help, because two queries built on the same wrong definition will agree.&lt;br&gt;
Check: when a case fails, have its owner rerun the trusted query by hand before anyone touches the assistant. A failing money case blocks the change until the owner decides which side is wrong.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The AI judge looks reliable but is not valid.&lt;/strong&gt; It gives the same verdict every time, yet marks wrong answers as right.&lt;br&gt;
Guardrail: calibrate it against human-marked answers using agreement beyond chance, and keep people on money questions.&lt;br&gt;
Check: give the judge a few answers a person marked wrong. If it passes them, it is not ready.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Answers shift silently after a model change.&lt;/strong&gt; The provider updates the model and nothing shows an error.&lt;br&gt;
Guardrail: pin the model version, record it with each run, and rerun the full set before switching.&lt;br&gt;
Check: compare this week's run with last week's. A case that changed result while its definition did not needs a person.&lt;/p&gt;
&lt;h2&gt;
  
  
  For your data team
&lt;/h2&gt;

&lt;p&gt;This part gets technical. One case file, with made-up values:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# evals/paying_customers.yaml&lt;/span&gt;
&lt;span class="na"&gt;case&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;q-001&lt;/span&gt;
&lt;span class="na"&gt;question&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;How&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;many&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;paying&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;customers&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;did&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;we&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;have&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;on&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;30&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;September&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;2026?"&lt;/span&gt;
&lt;span class="na"&gt;trusted_query&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;queries/paying_customers_at_date.sql&lt;/span&gt;   &lt;span class="c1"&gt;# run on today's data, not a stored number&lt;/span&gt;
&lt;span class="na"&gt;params&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{&lt;/span&gt;&lt;span class="nv"&gt;as_of&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;2026-09-30&lt;/span&gt;&lt;span class="pi"&gt;}&lt;/span&gt;
&lt;span class="na"&gt;definition&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;metrics/paying_customer.md@v2&lt;/span&gt;
&lt;span class="na"&gt;owner&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;finance_lead&lt;/span&gt;
&lt;span class="na"&gt;last_reviewed&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2026-10-01&lt;/span&gt;
&lt;span class="na"&gt;compare&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;result_set&lt;/span&gt;          &lt;span class="c1"&gt;# execution accuracy&lt;/span&gt;
&lt;span class="na"&gt;holdout&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;                &lt;span class="c1"&gt;# true = never used while tuning&lt;/span&gt;
&lt;span class="na"&gt;invariants&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;rephrase&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;How&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;many&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;customers&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;were&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;paying&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;at&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;the&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;of&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;September?"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;parts_sum&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{&lt;/span&gt;&lt;span class="nv"&gt;by&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;region&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;tolerance_pct&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;0.1&lt;/span&gt;&lt;span class="pi"&gt;}&lt;/span&gt;   &lt;span class="c1"&gt;# amounts only&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;cross_source&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{&lt;/span&gt;&lt;span class="nv"&gt;other&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;crm.accounts&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;max_gap_pct&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;2&lt;/span&gt;&lt;span class="pi"&gt;}&lt;/span&gt;   &lt;span class="c1"&gt;# set per metric&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;cites_definition&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;jump_vs_prior&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{&lt;/span&gt;&lt;span class="nv"&gt;max_change_pct&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;25&lt;/span&gt;&lt;span class="pi"&gt;}&lt;/span&gt;   &lt;span class="c1"&gt;# set per metric&lt;/span&gt;
&lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pinned-model-2026-09&lt;/span&gt;     &lt;span class="c1"&gt;# recorded with every run&lt;/span&gt;
  &lt;span class="na"&gt;repeats&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt;                      &lt;span class="c1"&gt;# a case that passes only sometimes is a finding&lt;/span&gt;
  &lt;span class="na"&gt;triggers&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;model_change&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;instruction_change&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;definition_change&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;weekly&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;Ask the question in a fresh session with read-only credentials and the real instructions.&lt;/li&gt;
&lt;li&gt;Run the trusted query and the assistant's query, and compare result sets.&lt;/li&gt;
&lt;li&gt;Run each invariant, including every rephrasing.&lt;/li&gt;
&lt;li&gt;Diff against the previous run, so new failures stand out.&lt;/li&gt;
&lt;li&gt;Send misses to the owner with a cause tag, and add or fix a case in the same pull request.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Check it in 10 minutes
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Pick one money question your team asks often, such as paying customers last month.&lt;/li&gt;
&lt;li&gt;Write the query you trust, run it, and note its definition.&lt;/li&gt;
&lt;li&gt;Ask the assistant the question in two phrasings and compare all three numbers.&lt;/li&gt;
&lt;li&gt;Ask for the same number by region or plan, and check the parts add up.&lt;/li&gt;
&lt;li&gt;Put an owner's name and a review date on the trusted query.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;See where your setup stands with the &lt;a href="https://goagentdata.com/#self-check" rel="noopener noreferrer"&gt;two minute self-check&lt;/a&gt;. Then read the &lt;a href="https://goagentdata.com/guide#checks" rel="noopener noreferrer"&gt;draft checks&lt;/a&gt; your coding agent can run.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://goagentdata.com/posts/evals-for-data-answers" rel="noopener noreferrer"&gt;goagentdata.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>data</category>
      <category>sql</category>
      <category>testing</category>
    </item>
    <item>
      <title>The real work in a data agent is the harness around it</title>
      <dc:creator>jacob baruch</dc:creator>
      <pubDate>Tue, 20 Oct 2026 20:04:48 +0000</pubDate>
      <link>https://dev.to/jacob_baruch_2a2aec3f6e3c/the-real-work-in-a-data-agent-is-the-harness-around-it-1hln</link>
      <guid>https://dev.to/jacob_baruch_2a2aec3f6e3c/the-real-work-in-a-data-agent-is-the-harness-around-it-1hln</guid>
      <description>&lt;p&gt;&lt;strong&gt;TL;DR.&lt;/strong&gt; An AI agent is a model plus a harness: everything around the model that lets it act, such as its instructions, tools, permissions, memory, checks and approval steps. The agent tool you use ships with a general harness. For data work, you add five parts of your own: a rule file it must follow, metric definitions it must cite, read-only access to a copy of the data, a short list of checks before an answer counts as done, and a rule to stop and ask when no definition covers the question. Add them in that order as your company grows. The harness, more than the model, decides whether you can trust the number.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is a harness?
&lt;/h2&gt;

&lt;p&gt;Think of a car engine. On its own, on a stand in a workshop, it is powerful and useless. It becomes a car only once you add steering, brakes, a fuel line, gauges and a seat for the driver. Nobody buys an engine to drive to work.&lt;/p&gt;

&lt;p&gt;An AI model is the engine. It reasons and writes queries fast. The harness is the rest of the car: the instructions it reads, the tools it may use, what it is allowed to touch, what it remembers between steps, the checks its work must pass, and the points where a person approves before it continues. A common way to put it, as &lt;a href="https://addyosmani.com/blog/agent-harness-engineering/" rel="noopener noreferrer"&gt;Addy Osmani explains in his overview of harness engineering&lt;/a&gt;: agent equals model plus harness.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where does the term come from?
&lt;/h2&gt;

&lt;p&gt;Software has used "harness" for a long time, for the code that runs a program under test. The current sense comes from AI coding. There, the model writes the code, and engineers spend more and more of their effort on the harness around it: instructions split into small files instead of one giant prompt, context kept between sessions, checks such as tests that must pass, a clear definition of "done", and a gate where the agent stops and asks. People now call this harness engineering.&lt;/p&gt;

&lt;p&gt;I think data work needs the same shift, and needs it more, because a wrong data answer looks exactly like a right one. A failing test is loud. A wrong number is quiet.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why does a data agent need one?
&lt;/h2&gt;

&lt;p&gt;Picture a small team that ran three ad campaigns last month and asks its AI assistant, "Which campaign brought us the most customers?" The assistant joins sign-ups to campaign tags and names a winner in one confident sentence.&lt;/p&gt;

&lt;p&gt;Behind that sentence sit four choices nobody made on purpose. It counted sign-ups, not paying customers. It gave each sign-up to the last ad clicked, although the team usually credits the first. It included the sign-ups the team created while testing the landing page. And it read "last month" in a different time zone from the ad platform. Any one of these alone can change the winner. The team moves budget to it anyway, because the answer was fluent.&lt;/p&gt;

&lt;p&gt;The model was not broken. Nothing around it said what "customer" means here, which attribution rule the team uses, what to leave out, or when to stop and ask. A better model alone does not close that gap.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is a minimal harness for a data agent?
&lt;/h2&gt;

&lt;p&gt;The agent tool already gives you the loop, the tools and a way to read files. What it cannot know is your business. Here are the five data parts you add, in the order I would build them.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;A rule file.&lt;/strong&gt; One short document the agent must follow on every task. It says which sources the agent may use and how it must answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Metric definition files.&lt;/strong&gt; One small file per key number, covering what it means, who is counted, which period, and who owns it. The agent must cite the file behind every number. The &lt;a href="https://goagentdata.com/posts/one-definition-per-metric" rel="noopener noreferrer"&gt;one definition per metric&lt;/a&gt; post covers these in detail.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read-only access to a copy of the data.&lt;/strong&gt; Never the live database. The agent can look and calculate, but its credentials cannot change anything, and its queries run on the copy instead of the system your customers use. The &lt;a href="https://goagentdata.com/guide#check-no-analytics-on-primary" rel="noopener noreferrer"&gt;no-analytics-on-primary check&lt;/a&gt; covers this.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A short list of checks before an answer counts as done.&lt;/strong&gt; For example: test accounts excluded, period stated, definition cited. An answer that fails a check goes back to the agent. It does not reach the reader.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A stop rule.&lt;/strong&gt; When no definition covers the question, the agent says "I don't know" or asks which definition you mean. It does not invent one.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Four parts are text files. The fifth is a copy of the data and a login that can only read it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does it look like at each stage?
&lt;/h2&gt;

&lt;p&gt;I think the harness should grow in the same order as the company, and no faster.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Before launch.&lt;/strong&gt; You have one or two questions and almost no data. Write the rule file, plus a one page glossary for the one or two numbers you need. Give the agent a copy of the data to read, even a spreadsheet export. Add one check: every answer names its definition.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First paying customers.&lt;/strong&gt; Money now flows through billing, so wrong numbers cost more. Turn each glossary entry into its own metric file. Make the read-only copy a scheduled copy of your production database and billing data. Write the checklist down, with the three example checks as the minimum. One named person owns the files.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Growing with a first data person.&lt;/strong&gt; Questions arrive faster than one person can review. The data person turns the checks into code that runs before an answer is shown, logs every answer with the files it cited, and reviews any change to a metric file. Questions the agent declines go to them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What goes wrong, and how do you catch it?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The agent invents a definition.&lt;/strong&gt; It answers with a meaning of "active" that exists nowhere in your company.&lt;br&gt;
Guardrail: the stop rule, plus a rule that every number cites a metric file.&lt;br&gt;
Check: ask a question your files do not cover. A good harness answers "I don't know" or asks you to choose a definition.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The agent claims it is done, but it is not.&lt;/strong&gt; It reports a number without noting that test accounts were included.&lt;br&gt;
Guardrail: the checklist, with a clear definition of done.&lt;br&gt;
Check: open the log for the agent's last answer and look for each check's result. An agent saying it ran a check is not proof; a recorded result is. If there is no log, start one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The agent touches live data.&lt;/strong&gt; A badly formed query slows the product, or a write changes a real record.&lt;br&gt;
Guardrail: read-only access to a copy, never the live database.&lt;br&gt;
Check: ask your engineer to try a write with the agent's credentials, and to confirm those credentials cannot reach the live database. The write must be refused.&lt;/p&gt;
&lt;h2&gt;
  
  
  For your data team
&lt;/h2&gt;

&lt;p&gt;This part gets technical. The harness is a folder in your repository, changed by reviewed pull request, and read by the agent at the start of each task. A starting rule file might look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# agent-rules.yaml  (stage: first paying customers)&lt;/span&gt;
&lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="na"&gt;sources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;allowed&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;analytics_replica&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;billing_export&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;    &lt;span class="c1"&gt;# read-only copies&lt;/span&gt;
  &lt;span class="na"&gt;forbidden&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;production_primary&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="na"&gt;access&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;mode&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read_only&lt;/span&gt;
  &lt;span class="na"&gt;credentials&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;replica_reader&lt;/span&gt;                      &lt;span class="c1"&gt;# no write grants&lt;/span&gt;
&lt;span class="na"&gt;metrics&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;folder&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;metrics/&lt;/span&gt;
  &lt;span class="na"&gt;rule&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="s"&gt;Compute a reported number only from a metric file.&lt;/span&gt;
    &lt;span class="s"&gt;Cite the file name and version in the answer.&lt;/span&gt;
&lt;span class="na"&gt;before_done&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;                                       &lt;span class="c1"&gt;# every item must pass&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;test_and_internal_accounts_excluded&lt;/span&gt;            &lt;span class="c1"&gt;# is_test, is_internal&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;period_stated&lt;/span&gt;                                  &lt;span class="c1"&gt;# dates and time zone&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;definition_cited&lt;/span&gt;                               &lt;span class="c1"&gt;# file name and version&lt;/span&gt;
&lt;span class="na"&gt;stop_rule&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;when&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;no_metric_file_covers_the_question&lt;/span&gt;
  &lt;span class="na"&gt;do&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ask which definition to use, or answer "I don't know"&lt;/span&gt;
&lt;span class="na"&gt;changes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;metric_files&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pull_request_with_named_reviewer&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notes on how the pieces fit:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Enforcement versus guidance.&lt;/strong&gt; The rule file guides the agent. Permissions enforce it. Credentials with no write grants, scoped to the replica and the export only, hold even if the agent ignores a line of text. A read replica also keeps heavy queries off the primary, though very long queries can still delay replication on some setups, so set a statement timeout.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checks as code.&lt;/strong&gt; At first a person runs the checklist by hand. Later, write each check as a small test, for example a count of &lt;code&gt;is_test = true&lt;/code&gt; rows behind the answer that must be zero, and run it before the answer is shown.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Logging.&lt;/strong&gt; Store each question, the files cited, the checks passed and the answer, so you can replay how a challenged number was made. A question the agent keeps declining is a signal to write a new metric file.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Check it in 10 minutes
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Ask your agent the question you ask most often, for example which campaign brought the most customers last month. Write down the answer.&lt;/li&gt;
&lt;li&gt;Ask which definition it used and where that is written. If it cannot name a file, you have no harness for that number yet.&lt;/li&gt;
&lt;li&gt;Ask a question your files do not cover. It should say "I don't know" or ask you to choose.&lt;/li&gt;
&lt;li&gt;Look at the record of which checks passed for that answer: test accounts, period and definition. If nothing was recorded, nothing was checked.&lt;/li&gt;
&lt;li&gt;Ask your engineer whether the agent can write to the live database. The answer should be no.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;See where your setup stands with the &lt;a href="https://goagentdata.com/#self-check" rel="noopener noreferrer"&gt;two minute self-check&lt;/a&gt;, or read the &lt;a href="https://goagentdata.com/guide#checks" rel="noopener noreferrer"&gt;draft checks&lt;/a&gt; your coding agent can run.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://goagentdata.com/posts/data-agent-harness" rel="noopener noreferrer"&gt;goagentdata.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>dataengineering</category>
      <category>analytics</category>
      <category>data</category>
    </item>
  </channel>
</rss>
