<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: AI Explore</title>
    <description>The latest articles on DEV Community by AI Explore (@aiexplore369zoho).</description>
    <link>https://dev.to/aiexplore369zoho</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4006822%2Ff413777a-0ac2-47e6-a213-9bb7bf701085.png</url>
      <title>DEV Community: AI Explore</title>
      <link>https://dev.to/aiexplore369zoho</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/aiexplore369zoho"/>
    <language>en</language>
    <item>
      <title>Why Small LLMs Memorize: I Trained the Same Model at Four Sizes and Watched It Start Reciting</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Tue, 17 Nov 2026 12:30:01 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/why-small-llms-memorize-i-trained-the-same-model-at-four-sizes-and-watched-it-start-reciting-2507</link>
      <guid>https://dev.to/aiexplore369zoho/why-small-llms-memorize-i-trained-the-same-model-at-four-sizes-and-watched-it-start-reciting-2507</guid>
      <description>&lt;p&gt;&lt;em&gt;Part 3 of ten. Each post is a short, plain-words tour of one lesson from &lt;a href="https://theaivibe.org/emc2" rel="noopener noreferrer"&gt;e=mc²&lt;/a&gt;, a free build-an-LLM-from-scratch course; the full lesson, notebook and numbers are on the site.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The same little language model, trained on the same stories for the same 1,500 steps, either makes things up or recites, and the only thing that decides which is how many dials it has.&lt;/strong&gt; In Lesson 01 I built a 5.9-million-parameter storyteller, Momo, and caught him reciting 61 words of a training story word for word. That left a question one model cannot answer. How small does Momo have to be before he stops reciting and starts composing? So in Lesson 03 I built him four times.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four Momos, one Ella
&lt;/h2&gt;

&lt;p&gt;A quick refresher. &lt;strong&gt;Momo&lt;/strong&gt; is a young monkey learning to tell stories; inside his head are dials, and the dials are the model's parameters. &lt;strong&gt;Ella&lt;/strong&gt; is an elephant who never forgets. She reads Momo stories one word at a time, lets him guess the next word, tells him how wrong he was, and nudges his dials. That nudge is training.&lt;/p&gt;

&lt;p&gt;Today's twist is that there are four Momos. One has 953,856 dials; call him the 1M model. The next has 3,060,288 (3M), then 5,853,184 (6M, which is exactly Lesson 01's Momo), then 12,318,720 (12M). Everything else is held fixed on purpose: the same 22,800 training stories, the same tokenizer, the same 1,500 steps, the same random seed, so all four even see the same batches in the same order. Size is the only thing that differs.&lt;/p&gt;

&lt;p&gt;Only two numbers change between them: how many blocks are stacked, and how wide each token's list of numbers is. That is the whole experiment.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="no"&gt;SIZES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="c1"&gt;# name -&amp;gt; (n_layer, n_embd); 4 heads, 256-token window, dropout 0 in all&lt;/span&gt;
 &lt;span class="s2"&gt;"1m"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="s2"&gt;"3m"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;192&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="s2"&gt;"6m"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# Lesson 01's config&lt;/span&gt;
 &lt;span class="s2"&gt;"12m"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;384&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="c1"&gt;# per size: seed 42, build GPT, seed 42 again (same batches), 1,500 steps, then measure&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nl&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ne&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;SIZES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
 &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;train_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nl&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ne&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
 &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;recite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# 50 prompts x 120 tokens, temperature 0.8, top-k 40, seed 1234&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The stories Ella reads three times
&lt;/h2&gt;

&lt;p&gt;The training set is mostly TinyStories, 19,800 short children's stories. Mixed in are 1,000 Momo-and-Ella stories I wrote, and Ella reads each of those three times. Another 200 TinyStories are kept back as a quiz Momo never sees, with no Momo in them at all.&lt;/p&gt;

&lt;p&gt;Those thousand repeated stories are the trap. If you are a pile of dials trying to be less wrong about the next word, and the same story comes round three times, the cheapest trick is to simply remember it. The question is how many dials it takes before that trick becomes available.&lt;/p&gt;

&lt;h2&gt;
  
  
  How you catch a monkey reciting
&lt;/h2&gt;

&lt;p&gt;You cannot ask Momo whether he is copying, so I measure. Each model gets the first ten tokens of 50 training Momo stories and writes 120 more tokens. Then I take what he wrote and find the longest stretch of it that appears, character for character, inside any of the 1,000 training Momo stories. I call that stretch the &lt;strong&gt;run&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A run of 100 characters is about three of the short Momo sentences (they average 32 characters), or roughly a quarter of an average 417-character story. If your median run is past 100, you are not composing. You are reciting.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers, honestly
&lt;/h2&gt;

&lt;p&gt;Here is what the four sizes did, smallest to largest.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Quiz perplexity:&lt;/strong&gt; 16.0, 10.8, 9.0, 7.4. Perplexity is "between how many words is Momo still choosing" on the 200 stories he never saw. Bigger Momo, better guesses, as you would expect. The 6M row lands at 9.0, Lesson 01's figure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Loss on the Momo stories themselves:&lt;/strong&gt; 0.554, 0.247, 0.206, 0.185. This is how surprised each model is by the stories it trained on. The drop between the first two sizes is the big one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Median longest copied run:&lt;/strong&gt; 62, 211.5, 265 and 269 characters.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Generations with a run of 100 or more:&lt;/strong&gt; 6% (3 of 50), 98% (49 of 50), 100%, 100%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Read the last two lines together. The 1M model mostly composes. Its median run is 62 characters, and only 3 of its 50 stories contain a 100-character copy. Its stories wander and mix sentences into things like this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;tell a soft pink banana that&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Then at 3.06 million parameters it flips. Forty-nine of fifty generations contain a copied run of 100 characters or more. The median is 211.5 characters and the longest is 361, most of a whole story. At 6M and 12M every single generation passes 100, and the first sample from both of those models is the very same 266-character paragraph, which turns out to occur word for word in three different training stories.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The check number is 3M.&lt;/strong&gt; The smallest size whose median copied run crosses 100 characters is the 3,060,288-parameter model. Somewhere between 0.95M and 3.06M dials, recitation switched on.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that surprised me
&lt;/h2&gt;

&lt;p&gt;I expected memorisation to show up the way the textbooks say it does: training loss keeps falling, quiz loss starts climbing, a gap opens. It did not. At every size, train loss sat &lt;em&gt;above&lt;/em&gt; quiz loss, the opposite of an overfitting gap. If I had only looked at those two curves I would have said the larger models were fine. They were reciting the whole time.&lt;/p&gt;

&lt;p&gt;That is the one idea I want you to carry away from this post. Memorising a repeated corner of the data need not show up in the headline numbers at all. The Momo-story loss and the copied-run table are where it shows, and you only see it if you go looking.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does not prove
&lt;/h2&gt;

&lt;p&gt;I owe you the disappointing parts too. This is Momo's friendliest possible case for memorising: the thousand Momo stories are built from small pools of stock sentences and read three times each. My run metric cannot tell one story learned by heart from a string of stock sentences that many stories share, so treat "copied run" as how much of the output is lifted from the training set, not as proof of story-by-story recall. I trained four sizes, so I can only say the switch happened between 0.95M and 3.06M parameters, not where. The 6M and 12M medians differ by 4 characters, which is noise. One seed and 50 prompts means no error bars. And none of this predicts where the thresholds sit for GPT-scale models; the research on those uses far larger models and far messier data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run it yourself
&lt;/h2&gt;

&lt;p&gt;All four train in minutes. On an Apple Silicon Mac this run took 0.55, 1.70, 3.79 and 6.33 minutes for the four sizes, and the whole notebook about 10 to 13 minutes. It also runs on a free Colab T4, though I have not timed it there. The full lesson on the site has the glossary, both tables, the short trick that finds the copied runs, and three check-yourself questions, including the one I am still chewing on: how would these numbers change if the thousand Momo stories were genuinely different from one another?&lt;/p&gt;

&lt;h2&gt;
  
  
  Want to actually build it?
&lt;/h2&gt;

&lt;p&gt;This post is the postcard. The lesson is the trip. &lt;strong&gt;Lesson 03 of e=mc²&lt;/strong&gt; on theaivibe.org walks you through every piece with widgets you can poke, a notebook that runs on a free GPU in minutes, and the honest numbers at every step. No account, no paywall, nothing to install.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Start here:&lt;/strong&gt; &lt;a href="https://theaivibe.org/emc2/why-small-llms-memorize-train-four-sizes" rel="noopener noreferrer"&gt;https://theaivibe.org/emc2/why-small-llms-memorize-train-four-sizes&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The whole ten-lesson journey:&lt;/strong&gt; &lt;a href="https://theaivibe.org/emc2" rel="noopener noreferrer"&gt;https://theaivibe.org/emc2&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tomorrow: Lesson 04 takes the public Lesson 01 weights, teaches Momo a brand-new character, and measures exactly what he forgets along the way.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>beginners</category>
    </item>
    <item>
      <title>How Attention Works: I Read All 24 Heads of a Tiny Language Model and Found the One That Watches the Main Character's Name</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Mon, 16 Nov 2026 12:30:01 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/how-attention-works-i-read-all-24-heads-of-a-tiny-language-model-and-found-the-one-that-watches-196m</link>
      <guid>https://dev.to/aiexplore369zoho/how-attention-works-i-read-all-24-heads-of-a-tiny-language-model-and-found-the-one-that-watches-196m</guid>
      <description>&lt;p&gt;&lt;em&gt;Part 2 of ten. Each post is a short, plain-words tour of one lesson from &lt;a href="https://theaivibe.org/emc2" rel="noopener noreferrer"&gt;e=mc²&lt;/a&gt;, a free build-an-LLM-from-scratch course; the full lesson, notebook and numbers are on the site.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Attention is the part of a language model that lets each word look back at the words before it, and yesterday's tiny model is small enough that you can watch it happen.&lt;/strong&gt; In Lesson 01 I trained a 5.9-million-parameter storyteller on a free Colab GPU. Today nothing gets trained. I open those same weights, switch on one flag that records who looked at whom, and read all 24 of its attention heads. One of them, it turns out, spends about 22% of its attention on the main character's name.&lt;/p&gt;

&lt;h2&gt;
  
  
  Momo and Ella, again
&lt;/h2&gt;

&lt;p&gt;Momo the monkey is the model; his dials were set last lesson and stay put tonight. Ella the elephant is the teacher and the memory. She reads him the story, and she keeps a shelf of everything already said. Picture the story told in a circle: each word has a seat, and each word may look at the words that spoke before it, never at the ones still to come.&lt;/p&gt;

&lt;p&gt;Attention is the looking.&lt;/p&gt;

&lt;h2&gt;
  
  
  A question, a card and a handover
&lt;/h2&gt;

&lt;p&gt;Here is the whole idea. Inside one attention head, every word gets three short lists of numbers, 64 numbers each in this model.&lt;/p&gt;

&lt;p&gt;The first list is a &lt;strong&gt;query&lt;/strong&gt;: the question a word asks. When "He" comes up in a story, the question is roughly "who is he?"&lt;/p&gt;

&lt;p&gt;The second is a &lt;strong&gt;key&lt;/strong&gt;: the card a word holds up for every later word to see. "Momo" holds up a card that says, in effect, "I am a name."&lt;/p&gt;

&lt;p&gt;The third is a &lt;strong&gt;value&lt;/strong&gt;: what the word hands over if it gets picked.&lt;/p&gt;

&lt;p&gt;Now suppose I am the word "He". I compare my question to every earlier word's card. The better the match, the higher the score. A step called &lt;strong&gt;softmax&lt;/strong&gt; turns those scores into shares that add up to 100%, and I take the share-weighted blend of everyone's values. If most of my shares went to "Momo", most of what I carry forward came from "Momo".&lt;/p&gt;

&lt;p&gt;Write down those shares for every word in a story and you get an &lt;strong&gt;attention map&lt;/strong&gt;: a table of who looked at whom.&lt;/p&gt;

&lt;p&gt;One honest note: nothing is asked in words. A query is 64 numbers made by a matrix multiplication, and "looking for a name" is a label I read into a head afterwards. Momo was only ever trained to guess the next word.&lt;/p&gt;

&lt;h2&gt;
  
  
  The triangle, and Ella's shelf
&lt;/h2&gt;

&lt;p&gt;Momo must never peek at words after the one he is guessing. The &lt;strong&gt;causal mask&lt;/strong&gt; enforces it: before the softmax, every score for a later word is set to minus infinity, so its share is exactly zero. Lay a whole story's map out and it is a lower triangle. The dark upper-right corner of every heatmap is the rule, not a decision the head made.&lt;/p&gt;

&lt;p&gt;The mask also explains the &lt;strong&gt;KV cache&lt;/strong&gt;, which is just Ella's shelf. Earlier words cannot see later ones, so an earlier word's key and value never change once computed. When Momo writes a story one token at a time, Ella files each token's key and value on the shelf, and the next token computes only its own and reads the rest. Writing 200 tokens costs 200 key-and-value computations instead of 20,100. In the notebook, the shelf's answer matches recomputing everything within 0.0000003. It is a speed trick, not a memory of the story: what Momo knows lives in his dials, not on the shelf.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading a real model's mind
&lt;/h2&gt;

&lt;p&gt;The notebook copies the model from Lesson 01 and adds one flag. With the flag on, each block computes attention the long way and keeps the shares; with it off, it uses PyTorch's fast built-in routine as before. The two agree within 0.00002 in the raw output scores, and every row of every map sums to 1. That flag is the whole change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;forward&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
 &lt;span class="no"&gt;B&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;C&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;
 &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;qkv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;C&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dim&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# a query, key and value per token&lt;/span&gt;
 &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;B&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;n_head&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;C&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;n_head&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transpose&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
 &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;return_attn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="c1"&gt;# the only change from Lesson 01&lt;/span&gt;
 &lt;span class="n"&gt;scores&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt; @ &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transpose&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
 &lt;span class="n"&gt;mask&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;tril&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ones&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;device&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;device&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
 &lt;span class="n"&gt;att&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;softmax&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;masked_fill&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="n"&gt;mask&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"-inf"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dim&lt;/span&gt;&lt;span class="o"&gt;=-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
 &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last_attn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;att&lt;/span&gt; &lt;span class="c1"&gt;# (batch, 4 heads, T, T): who looked at whom&lt;/span&gt;
 &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;att&lt;/span&gt; @ &lt;span class="n"&gt;v&lt;/span&gt;
 &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
 &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;scaled_dot_product_attention&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;is_causal&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then I plot the triangle, plot all 24 heads on one story, and measure every head over the 200 quiz stories Momo never trained on. A main character, by a deliberately crude rule, is the most frequent capitalised name in a story, mentioned at least twice; 159 of the 200 stories have one. For each head I add up how much attention the other words, after the first mention, send back to that name.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;24 heads read:&lt;/strong&gt; 6 blocks of 4 heads, all from the Lesson 01 weights. No training.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Even-spread baseline: 0.052.&lt;/strong&gt; If a head spread its attention evenly, that is roughly how much would land on the name.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The winner, block 5, head 3 (L5H3): 0.221.&lt;/strong&gt; About 22% of its attention on the main character's name, 4.3 times an even spread. L4H4 (0.199) and L6H2 (0.178) are close behind.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17 of the 24 heads&lt;/strong&gt; put less on the name than an even spread would.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Previous-token heads: 0 of 24.&lt;/strong&gt; None puts more than half its attention there; the highest is L1H1 at 0.146. That was the disappointing result, and I am reporting it as is.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;8 to 20 seconds&lt;/strong&gt; for the whole notebook on an Apple Silicon Mac; it also runs on a free Colab T4 or a plain CPU.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it does not mean
&lt;/h2&gt;

&lt;p&gt;Three limits, said plainly. L5H3 puts about 22% of its attention on the name, not all of it; this is a tendency, not a switch. Attention is where a head looks, not proof of what the model relies on; I did not switch any head off to see what breaks. And name position is not controlled: a name early in a story may draw attention partly for being early.&lt;/p&gt;

&lt;p&gt;So no, this does not prove Momo understands who the hero is. It shows, with a real number from a real model, where one of his 24 ways of looking tends to go. That is a smaller claim and a truer one, and it is the first step toward explaining the 61-word story Momo recited in Lesson 01 and could not explain.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this is the lesson that matters
&lt;/h2&gt;

&lt;p&gt;Every large model you have heard of is this mechanism with more heads, more blocks and more zeros. If you have ever wondered how a chat model keeps track of a character across a paragraph, you now have the picture: a circle of words looking back, and an elephant with a shelf.&lt;/p&gt;

&lt;p&gt;The full lesson has the glossary table, the heatmaps of all 24 heads, a clickable toy widget, the ranked table of every head and the notebook that reproduces the 0.221.&lt;/p&gt;

&lt;h2&gt;
  
  
  Want to actually build it?
&lt;/h2&gt;

&lt;p&gt;This post is the postcard. The lesson is the trip. &lt;strong&gt;Lesson 02 of e=mc²&lt;/strong&gt; on theaivibe.org walks you through every piece with widgets you can poke, a notebook that runs on a free GPU in minutes, and the honest numbers at every step. No account, no paywall, nothing to install.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Start here:&lt;/strong&gt; &lt;a href="https://theaivibe.org/emc2/how-attention-works-tiny-storyteller-attention-maps" rel="noopener noreferrer"&gt;https://theaivibe.org/emc2/how-attention-works-tiny-storyteller-attention-maps&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The whole ten-lesson journey:&lt;/strong&gt; &lt;a href="https://theaivibe.org/emc2" rel="noopener noreferrer"&gt;https://theaivibe.org/emc2&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tomorrow: Momo recited a 61-word story word for word in Lesson 01, so I train the same recipe at four sizes and watch for the moment memorising begins.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>beginners</category>
    </item>
    <item>
      <title>Build an LLM From Scratch in 5 Minutes on a Free GPU: I Trained a 5.9-Million-Parameter Storyteller, and Here Is What It Wrote</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Sun, 15 Nov 2026 16:28:04 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/build-an-llm-from-scratch-in-5-minutes-on-a-free-gpu-i-trained-a-59-million-parameter-j2c</link>
      <guid>https://dev.to/aiexplore369zoho/build-an-llm-from-scratch-in-5-minutes-on-a-free-gpu-i-trained-a-59-million-parameter-j2c</guid>
      <description>&lt;p&gt;&lt;em&gt;Part 1 of ten. Each post is a short, plain-words tour of one lesson from &lt;a href="https://theaivibe.org/emc2" rel="noopener noreferrer"&gt;e=mc²&lt;/a&gt;, a free build-an-LLM-from-scratch course; the full lesson, notebook and numbers are on the site.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Most people think building a language model takes a lab, a budget and a team.&lt;/strong&gt; Last month I built one from nothing, on a free Google Colab GPU, in about five minutes end to end. It has 5.9 million parameters, it learned to write bedtime stories, and it is sitting on the Hugging Face Hub right now where you can run it.&lt;/p&gt;

&lt;p&gt;I want to show you what it wrote, what the numbers mean, and why doing this once makes every headline about trillion-parameter models stop sounding like magic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Meet Momo and Ella
&lt;/h2&gt;

&lt;p&gt;The lesson is told through two characters, because the ideas are simpler than the vocabulary.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Momo&lt;/strong&gt; is a young monkey learning to tell stories. Inside his head are millions of tiny dials, all pointing somewhere random, so his first stories are nonsense.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ella&lt;/strong&gt; is an elephant, and elephants never forget. Every night she reads Momo a story one word at a time, stops before each word, lets him guess, and tells him how wrong he was. Then she nudges his dials a little. Thousands of nights later, Momo tells stories.&lt;/p&gt;

&lt;p&gt;That is the whole of machine learning in one paragraph. Momo is the model. His dials are the weights (5.9 million of them here). Ella's stories are the training data. "How wrong were you" is the loss. Nudging the dials is gradient descent. Everything else is detail, and the lesson decodes about forty of those details one at a time.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a language model does, in one sentence
&lt;/h2&gt;

&lt;p&gt;Given the words so far, guess the next one. A story is that guess repeated. A chat answer is the same thing with your question as the words so far.&lt;/p&gt;

&lt;p&gt;When Momo continues "Once upon a time, a little", four things happen. The sentence is bitten into &lt;strong&gt;tokens&lt;/strong&gt;, pieces of text with a number on each (our vocabulary has 4,096). Each token becomes a spot on a &lt;strong&gt;meaning map&lt;/strong&gt;, a list of 256 numbers where similar words sit close together. Then every token looks back at the tokens before it and decides how much to listen to each: "capital" finds "France" and borrows some of its meaning. That is &lt;strong&gt;attention&lt;/strong&gt;, from the 2017 paper that started all this, and it runs six times in a row, four heads each. Finally the last token's sharpened meaning is scored against every word in the vocabulary, and one is picked.&lt;/p&gt;

&lt;p&gt;That is it. That is also, with more stories, more blocks and more GPUs, what the big ones do.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;5.9 million parameters:&lt;/strong&gt; 6 blocks, 4 attention heads, 256-wide, a vocabulary of 4,096 pieces learned from the stories themselves.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;1,500 training steps&lt;/strong&gt;, each a batch of 32 windows of 256 tokens.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;1.2 minutes of training&lt;/strong&gt; on Colab's free T4 GPU; about five minutes for the whole notebook; about two minutes on an Apple Silicon Mac.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Perplexity 9&lt;/strong&gt; on 200 stories the model never saw. Perplexity is "between how many words is Momo still choosing". Random guessing would be 4,096. Watching that number fall from 4,096 to 9 in front of you is the moment the field stops being mysterious.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The 200 quiz stories were held out before training, so nothing the model memorised can flatter the score. An earlier draft got that wrong, and fixing it is why I trust the 9.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it wrote
&lt;/h2&gt;

&lt;p&gt;Given "Lily had a red balloon.", one run produced this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;She ran to her mom and gave her a big bite. They both ate the balloon and had the balloon. It was shiny and yellow. They ran to the park to play with.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Grammatical, kid-flavoured, slightly unhinged. That is exactly what 5.9 million parameters and two minutes buy, and I think it is wonderful.&lt;/p&gt;

&lt;p&gt;Then I gave it "Once upon a time, a little monkey", and it wrote this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Momo was sad because he could not swim like Bo the bear cub. He sat in the tall grass with his feet in the water. Ella said, "Try again, little one. Every time you try, you learn a little more." She held him up with her trunk. Momo kicked and kicked. Soon he could swim all by himself. The end.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Lovely, right? It is also, character for character, the second half of one of the thousand Momo stories I wrote into the training data, which the model read three times. That is &lt;strong&gt;memorising, not inventing&lt;/strong&gt;. A few thousand repeats are easy for 5.9 million dials to store. The lesson shows you how to tell the difference, and why the fix is more unique stories and fewer repeats. Every large model has this problem too; it is just harder to catch when the training set is the internet.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run it in thirty seconds
&lt;/h2&gt;

&lt;p&gt;You do not have to train anything to try it. The weights, tokenizer and the notebook that builds them are public:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;huggingface_hub&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hf_hub_download&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;transformers&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;PreTrainedTokenizerFast&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;

&lt;span class="c1"&gt;# Config, GPT and tell_story() are the ~80 lines from Parts 6 and 8 of the notebook&lt;/span&gt;
&lt;span class="n"&gt;cfg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;hf_hub_download&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"singhpratech/tiny-storyteller"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"config.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;hf_hub_download&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"singhpratech/tiny-storyteller"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"model.pt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;map_location&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"cpu"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;tok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;PreTrainedTokenizerFast&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"singhpratech/tiny-storyteller"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;GPT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_state_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;tell_story&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Once upon a time, a little monkey"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Start prompts the way the stories start ("Once upon a time, a little monkey"). Temperature 0.8 is the default; turn it up and the stories get stranger.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why start this small
&lt;/h2&gt;

&lt;p&gt;Because once you have turned 5.9 million dials yourself and watched perplexity fall from 4,096 to 9, the big numbers stop being magic and start being engineering. The recipe does not change as it scales: tokenizer, embeddings, stacked attention, a next-word head, a loss, an optimiser. Only the zeros do.&lt;/p&gt;

&lt;h2&gt;
  
  
  Want to actually build it?
&lt;/h2&gt;

&lt;p&gt;This post is the postcard. The lesson is the trip. &lt;strong&gt;Lesson 01 of e=mc²&lt;/strong&gt; on theaivibe.org walks you through every piece with widgets you can poke, a notebook that runs on a free GPU in minutes, and the honest numbers at every step. No account, no paywall, nothing to install.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Start here:&lt;/strong&gt; &lt;a href="https://theaivibe.org/emc2/build-an-llm-from-scratch-free-gpu-tiny-storyteller" rel="noopener noreferrer"&gt;https://theaivibe.org/emc2/build-an-llm-from-scratch-free-gpu-tiny-storyteller&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The whole ten-lesson journey:&lt;/strong&gt; &lt;a href="https://theaivibe.org/emc2" rel="noopener noreferrer"&gt;https://theaivibe.org/emc2&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tomorrow's post opens the same tiny model and reads its mind: what its 24 attention heads actually look at.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>beginners</category>
    </item>
    <item>
      <title>AI Embeddings Break Columnar Assumptions in Parquet and Arrow</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:15:44 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/ai-embeddings-break-columnar-assumptions-in-parquet-and-arrow-45p3</link>
      <guid>https://dev.to/aiexplore369zoho/ai-embeddings-break-columnar-assumptions-in-parquet-and-arrow-45p3</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR —&lt;/strong&gt; Columnar engines like DuckDB and Arrow assume you scan a few columns and filter on values. An embedding column violates every one of those assumptions: it is wide, high-entropy, never filtered by value, and always read whole. Treat it as a separate physical tier, use fixed-size layouts, and late-materialize vectors after metadata filters.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Columnar storage is a bet on a specific access pattern: read a few columns out of many, filter on their values, and aggregate. Parquet, Arrow, DuckDB, and GPU dataframe engines are all optimized around that bet. It's a good bet for analytics.&lt;/p&gt;

&lt;p&gt;Then we add an embedding column and quietly lose it.&lt;/p&gt;

&lt;p&gt;My thesis: &lt;strong&gt;an embedding column is the one column type where the columnar assumptions invert, and most "slow AI data prep" is the result of storing it like any other column.&lt;/strong&gt; The fix isn't a faster engine. It's treating vectors as their own physical tier.&lt;/p&gt;

&lt;h2&gt;
  
  
  The access pattern flips
&lt;/h2&gt;

&lt;p&gt;Look at what a normal analytical column gets for free:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Value predicates.&lt;/strong&gt; Min/max statistics, dictionary pages, and bloom filters let the engine skip data based on what's inside.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Compression.&lt;/strong&gt; Low-cardinality strings, sorted integers, and timestamps compress dramatically with dictionary, run-length, and delta encodings.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Narrow reads.&lt;/strong&gt; The query touches three columns out of fifty.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now look at an embedding column:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;No value predicates.&lt;/strong&gt; Nobody writes a filter on the seventh dimension of a vector. Min/max stats on a float inside a vector are noise. You pay for the metadata and never use it.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Poor compression.&lt;/strong&gt; Learned floats are high-entropy. Dictionary and run-length encodings have nothing to grab. Byte-oriented transforms for floats help a bit, and general-purpose codecs buy you little for the CPU they burn.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;All-or-nothing reads.&lt;/strong&gt; A consumer wants the whole vector or none of it. There is no useful "read half of this column."&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the column that is usually the largest in your table, by bytes, gets none of the features that justify a columnar format. You're paying the format's costs (row-group overhead, encoding and decoding, nested-type bookkeeping) without collecting its benefits.&lt;/p&gt;

&lt;h2&gt;
  
  
  Nested lists are the quiet tax
&lt;/h2&gt;

&lt;p&gt;The default way people land embeddings in a table is a variable-length list of floats. In Parquet, nested and repeated data is represented with repetition and definition levels, extra bookkeeping the reader must decode to reconstruct list boundaries. For a variable-length list, that work is legitimate. For a vector where every row has exactly the same dimension, it's pure waste: you're encoding "where does each list end" for data whose answer is a constant.&lt;/p&gt;

&lt;p&gt;Arrow has the right abstraction: a fixed-size list. Every row is the same width, so the whole column is one flat contiguous buffer of floats plus a known stride. That layout is what NumPy, a GPU kernel, or an ANN index wants. Row &lt;code&gt;i&lt;/code&gt; lives at offset &lt;code&gt;i * dim&lt;/code&gt;. No offsets array, no per-row decode.&lt;/p&gt;

&lt;p&gt;The practical rule: &lt;strong&gt;if the dimension is a schema invariant, encode it as one.&lt;/strong&gt; Declare fixed-size at the Arrow layer, and verify what your Parquet writer and reader actually do with it end to end. Whether you get a zero-copy view or a silent conversion back through variable-length lists depends on the toolchain, and the failure mode is a slow pipeline, not an error. Check by measuring a round trip, not by trusting the type name.&lt;/p&gt;

&lt;h2&gt;
  
  
  Split the vector from the metadata
&lt;/h2&gt;

&lt;p&gt;The bigger win is physical, not logical. In a typical embeddings table you have a handful of small, filterable columns (document id, source, timestamp, tenant, language) and one huge opaque blob of floats. Storing them in the same files means every metadata scan drags row groups sized for vectors.&lt;/p&gt;

&lt;p&gt;Row-group sizing is where this bites. A row group is the unit of skipping and the unit of reading. If the vector column dominates the bytes, a row group sized by byte target holds very few rows. Your metadata statistics then cover tiny spans, footers balloon, and a filter that should prune aggressively instead opens thousands of small groups.&lt;/p&gt;

&lt;p&gt;The pattern that works is vertical partitioning:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;A metadata table&lt;/strong&gt; with the filterable columns, laid out normally, sorted or clustered on your common predicates, with healthy row-group sizes.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;A vector table&lt;/strong&gt; keyed by a stable row id, stored fixed-size, uncompressed or lightly compressed, with row groups sized for sequential throughput rather than pruning.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;A late join&lt;/strong&gt; that filters on metadata first, collects the surviving ids, and only then fetches vectors.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is late materialization, the oldest trick in column stores, applied to the one column where it matters most. DuckDB-style engines do a version of this inside a single query. But they can only do it well if the physical layout lets them avoid touching the vector bytes until the end.&lt;/p&gt;

&lt;h2&gt;
  
  
  The GPU makes the layout mistake louder
&lt;/h2&gt;

&lt;p&gt;GPU-accelerated analytics changes the arithmetic, but not in the direction people expect. The kernels are fast. The cost that dominates is moving bytes: from storage to host, from host to device, and across the interconnect. A vector column is the heaviest thing you'll move.&lt;/p&gt;

&lt;p&gt;That has three consequences:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Filter before you transfer.&lt;/strong&gt; Running a metadata predicate on the CPU and shipping only surviving vectors usually beats shipping everything to the device and filtering there. A GPU that sits idle waiting on a transfer isn't accelerating anything.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Contiguity is the feature.&lt;/strong&gt; A fixed-size, flat buffer can be handed to the device in one bulk copy. A list column with offsets forces a gather or a repack first, and that repack happens on the slow side of the bus.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Don't decompress on the critical path.&lt;/strong&gt; Since vectors barely compress, a heavy codec costs CPU cycles to save few bytes. Storing them with light or no compression often lets the read path run at storage speed.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is why "just put the dataframe on the GPU" disappoints on embedding-heavy workloads. The compute was never the bottleneck. The layout and the transfer were.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd actually do
&lt;/h2&gt;

&lt;p&gt;If I were setting up an embedding-bearing dataset today, this is the checklist:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Declare vectors as fixed-size lists at the schema level, and write a test that a round trip preserves that type and avoids a copy.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Keep vectors out of the table you filter and aggregate. Join by stable id.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Size row groups differently for each tier: pruning-friendly for metadata, throughput-friendly for vectors.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Pin the embedding model version next to the vector tier, so a re-embed replaces the whole tier atomically instead of mixing generations in place.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Choose precision deliberately. Storing vectors at lower precision is a bigger lever than any codec, but it changes retrieval quality, so evaluate it rather than assuming.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this is exotic. It's the same discipline column-store designers have always applied: store things the way they're read. The mistake is assuming a vector is "just another column" because the table format let you declare it as one.&lt;/p&gt;

&lt;p&gt;Columnar engines are excellent at what they were built for. An embedding column isn't that. Give it its own tier, and the fast engines you already have get fast again.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>dataengineering</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Why Your Streaming Pipeline Lies About Latency: Event Time vs. Processing Time</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Sat, 14 Nov 2026 12:05:02 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/why-your-streaming-pipeline-lies-about-latency-event-time-vs-processing-time-13m3</link>
      <guid>https://dev.to/aiexplore369zoho/why-your-streaming-pipeline-lies-about-latency-event-time-vs-processing-time-13m3</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR —&lt;/strong&gt; Most streaming pipelines report latency numbers that measure the wrong thing. The gap between event time and processing time is not a minor implementation detail — it is the fundamental reason dashboards look healthy while business logic silently produces stale or wrong results. Understanding watermarks, late data, and window semantics is the only way to build pipelines that are honest about what they know and when they know it.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Every streaming pipeline has a latency number on its dashboard. That number is almost always a lie — not because engineers are careless, but because most pipelines measure the wrong clock. They report how quickly data moves through the system, not how accurately the system reflects the world that generated the data. Those are completely different things, and conflating them is one of the most persistent sources of silent correctness failures in production data infrastructure.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two Clocks, One Pipeline
&lt;/h2&gt;

&lt;p&gt;Every event in a stream carries two timestamps. The first is &lt;strong&gt;event time&lt;/strong&gt;: when something actually happened in the real world — a user clicked a button, a sensor recorded a reading, a transaction cleared. The second is &lt;strong&gt;processing time&lt;/strong&gt;: when the event arrived at a particular stage of your pipeline.&lt;/p&gt;

&lt;p&gt;In a perfect world these would be identical. In practice, they diverge constantly. Mobile clients buffer events while offline and flush them minutes or hours later. IoT devices have clock skew. Upstream services retry failed deliveries. Kafka partitions rebalance and cause brief consumer lag spikes. A single network hiccup can push a batch of events into the next processing window even though they all happened before the window boundary.&lt;/p&gt;

&lt;p&gt;When your monitoring says "p99 latency: 800ms," it almost certainly means the 99th percentile of &lt;em&gt;processing time latency&lt;/em&gt; — the delay from broker ingestion to some downstream sink. It says nothing about how old the events themselves were when they arrived. A pipeline can have sub-second processing latency while routinely operating on data that is five minutes stale.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Windows Make the Problem Concrete
&lt;/h2&gt;

&lt;p&gt;Windowed aggregations are where event-time confusion graduates from theoretical concern to production incident. Consider a tumbling five-minute window that counts transactions per merchant. If you assign windows by processing time, a transaction that happened at 12:04:58 but arrived at 12:05:03 lands in the wrong window. Your 12:00–12:05 count is understated; your 12:05–12:10 count is overstated. The error is small per event, but it compounds across millions of events and it is &lt;em&gt;systematically biased&lt;/em&gt; in the direction of recency — the most recent window always looks artificially inflated because late arrivals from earlier windows keep leaking into it.&lt;/p&gt;

&lt;p&gt;Assign windows by event time instead, and you solve the misclassification problem — but you immediately face a new one: &lt;strong&gt;when do you close the window?&lt;/strong&gt; You cannot wait forever for late events. You need a principled way to say "I am confident enough that all events for this window have arrived; I will emit the result now."&lt;/p&gt;

&lt;p&gt;That mechanism is the watermark.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watermarks Are a Bet, Not a Guarantee
&lt;/h2&gt;

&lt;p&gt;A watermark is a statement of the form: "I believe all events with timestamps earlier than &lt;em&gt;T&lt;/em&gt; have now been observed." It is not a guarantee. It is a heuristic — a bet your pipeline makes about the maximum out-of-orderness of your data stream.&lt;/p&gt;

&lt;p&gt;Systems like Apache Flink let you define watermark strategies explicitly. A bounded-out-of-orderness watermark says: "Assume events arrive at most &lt;em&gt;N&lt;/em&gt; milliseconds late." Set &lt;em&gt;N&lt;/em&gt; too low and you close windows prematurely, discarding late data as it arrives after the watermark has already advanced past its window. Set &lt;em&gt;N&lt;/em&gt; too high and you hold state in memory longer, increase end-to-end latency, and delay downstream consumers from seeing results.&lt;/p&gt;

&lt;p&gt;This is not a parameter you can tune once and forget. The right value for &lt;em&gt;N&lt;/em&gt; depends on your data source, your network topology, your client SDK behavior, and your tolerance for late-data errors — and all of those things change over time. A mobile app update that adds aggressive offline buffering can silently invalidate a watermark strategy that worked fine for a year.&lt;/p&gt;

&lt;p&gt;The honest engineering position is: &lt;strong&gt;watermarks encode assumptions, and those assumptions should be monitored like any other SLO.&lt;/strong&gt; Track the distribution of event-time lag at ingestion. Alert when the 99th percentile of lag exceeds your watermark bound. Otherwise you will not know that your pipeline has started silently dropping late events until a business analyst notices that last Tuesday's numbers look wrong in hindsight.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Late Data Problem Has No Clean Solution
&lt;/h2&gt;

&lt;p&gt;Once you accept that watermarks are bets, you have to decide what to do when the bet is wrong — when an event arrives after its window's watermark has already advanced. You have three options, none of them free:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Drop the event.&lt;/strong&gt; Simple, but you are silently losing data. Acceptable only if you have quantified the business impact and it is genuinely negligible.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Emit a corrected result.&lt;/strong&gt; Reopen the window, recompute, and send a retraction followed by an updated aggregate. This is what Flink's allowed lateness mechanism supports. Downstream consumers must be built to handle retractions — most are not.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Route late events to a side output.&lt;/strong&gt; Preserve them for reconciliation, audit, or replay. This is operationally honest but requires a separate reconciliation process that most teams never actually build.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The worst choice is the most common one: drop late events silently and report a latency number that makes the pipeline look healthy. This is how you end up with fraud detection models trained on subtly incomplete feature windows, or revenue dashboards that quietly undercount during periods of high mobile traffic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Idle Partitions and Watermark Stalls
&lt;/h2&gt;

&lt;p&gt;There is a failure mode that bites almost every team the first time they run a multi-partition Flink or Spark Structured Streaming job on uneven traffic: &lt;strong&gt;watermark stalls caused by idle partitions.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;In most streaming frameworks, the global watermark advances to the minimum watermark across all partitions. If one Kafka partition receives no events for several minutes — because traffic to that shard is genuinely zero, or because a producer is down — the global watermark freezes. Windows stop closing. Downstream results stop emitting. The pipeline appears to be processing data normally but is actually stuck, holding state and waiting for a partition that may not produce events for a long time.&lt;/p&gt;

&lt;p&gt;Flink addresses this with idle source detection: after a configurable timeout with no events, a partition is marked idle and excluded from the watermark minimum calculation. But this is opt-in configuration that many teams never enable, and the timeout value requires the same careful tuning as the watermark bound itself. A timeout that is too short causes watermarks to advance past legitimate gaps in sparse data streams.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Honest Latency Measurement Looks Like
&lt;/h2&gt;

&lt;p&gt;A pipeline that takes event-time semantics seriously instruments at least three things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Event-time lag at ingestion:&lt;/strong&gt; the difference between wall-clock time and the event timestamp when the event is first consumed. This tells you how out-of-order your upstream data actually is.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Watermark lag:&lt;/strong&gt; how far behind the current watermark is relative to wall-clock time. A growing watermark lag means your pipeline is falling behind or stalling on idle partitions.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Late event rate per window:&lt;/strong&gt; what fraction of events for a given window arrive after that window's watermark has closed. This is the direct measure of how often your watermark bet is wrong.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these metrics are exotic. They are all derivable from timestamps that are already present in your events and your framework's internal state. The reason most pipelines do not track them is that setting up processing-time latency monitoring is easy and the numbers look good, while setting up event-time monitoring requires understanding the semantics first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Correctness Conversation Your Team Is Not Having
&lt;/h2&gt;

&lt;p&gt;The deeper issue is cultural. Streaming pipeline correctness is treated as an infrastructure concern — something the platform team handles — rather than a domain-level contract between the pipeline and its consumers. Downstream teams build models and dashboards on top of aggregations without knowing whether those aggregations are event-time correct, what the watermark strategy is, or what happens to late data.&lt;/p&gt;

&lt;p&gt;The fix is to make the contract explicit. Document, for every windowed aggregation your pipeline produces: the window type and size, the watermark strategy and its assumed maximum lateness, the late-data policy, and the expected late-event rate under normal conditions. Treat a change to any of these as a breaking schema change. Version it accordingly.&lt;/p&gt;

&lt;p&gt;That discipline is more valuable than any framework feature. Watermarks, allowed lateness, and side outputs are tools. The judgment about what your data source actually does and what your downstream consumers can actually tolerate — that is the engineering work that determines whether your pipeline is telling the truth.&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>database</category>
      <category>dataops</category>
    </item>
    <item>
      <title>Text-to-SQL Agents Answer Questions Nobody Reviewed. Add Receipts</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Fri, 13 Nov 2026 14:15:49 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/text-to-sql-agents-answer-questions-nobody-reviewed-add-receipts-123h</link>
      <guid>https://dev.to/aiexplore369zoho/text-to-sql-agents-answer-questions-nobody-reviewed-add-receipts-123h</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR —&lt;/strong&gt; A dashboard is a question that humans reviewed, argued over, and corrected. A text-to-SQL agent produces unreviewed questions at scale, and its typical failure is a valid query for the wrong interpretation. Fix this with answer receipts, divergence-triggered clarification, and a path that promotes frequent agent questions into reviewed artifacts.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Most conversations about text-to-SQL agents are about SQL generation: schema linking, join paths, dialect quirks. Those problems are real, and they are also the ones that get better on their own as models improve.&lt;/p&gt;

&lt;p&gt;The problem that doesn't fix itself is this: &lt;strong&gt;a dashboard is a reviewed question, and an agent answer is an unreviewed one.&lt;/strong&gt; Analytics engineers have been relying on that difference for years without naming it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a Dashboard Actually Was
&lt;/h2&gt;

&lt;p&gt;We talk about dashboards as visualization. Their real function was social validation of a query.&lt;/p&gt;

&lt;p&gt;Someone chose the grain. Someone decided what "active user" means, which timezone cuts the day, whether refunds net out, which test accounts get excluded. Then the chart sat in front of dozens of people who knew roughly what the number should be. A wrong figure got noticed because a finance lead said "that can't be right," and the query was fixed. Over months, the artifact converged.&lt;/p&gt;

&lt;p&gt;That loop was never in the architecture diagram. It was implicit error correction performed by humans glancing at a shared artifact repeatedly.&lt;/p&gt;

&lt;p&gt;A conversational agent deletes the loop. The question is asked once, answered once, and consumed by one person who has no strong prior about the answer, which is often exactly why they asked.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Failure Is Interpretation, Not Syntax
&lt;/h2&gt;

&lt;p&gt;Benchmarks for text-to-SQL mostly score whether the generated query returns the same result as a reference query. That is a reasonable proxy for translation. It is a poor proxy for answering a business question, because the reference query encodes one interpretation chosen by whoever wrote the benchmark.&lt;/p&gt;

&lt;p&gt;In production, the dangerous output isn't a query that fails. A failing query is loud. The dangerous output is a query that runs cleanly, returns a plausible number, and answers a slightly different question than the one the user meant. Consider "revenue last quarter":&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Calendar quarter or fiscal quarter?&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Gross, net of refunds, or recognized?&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Booked date or paid date?&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Which timezone defines the boundary?&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Does it include internal or test accounts?&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each choice produces a defensible query. The results differ, sometimes by a little, sometimes a lot. The agent picks one and returns a number with the confidence of a calculator.&lt;/p&gt;

&lt;p&gt;A semantic layer helps here, but only partly. It pins down what a metric means once you've selected it. It does not pin down the time window, the population, or which of three similar metrics the user was reaching for. The ambiguity migrates upward into question interpretation, and that part of the system has no owner.&lt;/p&gt;

&lt;h2&gt;
  
  
  Receipts: Make the Interpretation a Return Value
&lt;/h2&gt;

&lt;p&gt;The first fix is cheap and architectural. An agent should never return a bare number. It should return an &lt;strong&gt;answer receipt&lt;/strong&gt;, a structured record of the choices it made:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;The resolved metric and the definition it bound to, by identifier rather than prose.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The grain, filters, time range, and timezone, stated explicitly, including defaults the user never specified.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The executed SQL.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Input freshness and the row counts at key stages, so an empty join is visible.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Any assumptions the agent made because the question was silent.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The point is not to make users read SQL. Most won't. The point is that the receipt is machine-checkable and human-skimmable. A finance lead can scan "fiscal quarter, net of refunds, UTC" in two seconds and catch the mismatch, which is exactly the glance that a dashboard used to provoke.&lt;/p&gt;

&lt;p&gt;Defaults are the critical part. Every silently applied default is a decision the user didn't make. Surfacing the defaults turns hidden interpretation into reviewable interpretation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ask When Interpretations Diverge, Not When the Model Feels Unsure
&lt;/h2&gt;

&lt;p&gt;The obvious next step is "have the agent ask clarifying questions." Done naively, it's miserable. The agent asks about everything, users learn to click through, and you've built a worse form UI.&lt;/p&gt;

&lt;p&gt;Model self-reported uncertainty is a weak trigger. A better one is observable: &lt;strong&gt;generate several plausible interpretations, execute them, and compare the results.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If the candidate interpretations all return roughly the same number, the ambiguity is harmless and the agent should just answer, with a receipt. If they diverge beyond a tolerance, the ambiguity is material, and the agent should surface the fork: "Net of refunds this is one figure; gross it is another. Which do you want?"&lt;/p&gt;

&lt;p&gt;This reframes clarification as a cost-benefit decision grounded in outcomes instead of vibes. It also has an honest cost, since you run multiple queries per question. That cost is bounded and predictable, and cheap compared to a wrong number making it into a planning meeting. You can limit it by running the alternates only on questions whose terms resolve to more than one candidate definition.&lt;/p&gt;

&lt;h2&gt;
  
  
  Close the Loop: Promote Questions Into Artifacts
&lt;/h2&gt;

&lt;p&gt;Receipts and divergence checks make individual answers safer. They don't rebuild the thing dashboards gave us, which was convergence over time.&lt;/p&gt;

&lt;p&gt;To get that back, log every receipt and cluster the questions. The frequent clusters are your real demand signal, and they should graduate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;An agent answers a question with a receipt.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The same intent shows up repeatedly across users.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;An analytics engineer reviews the interpretation, fixes definitions, and certifies it as a named query or metric.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The agent routes future matches to the certified artifact instead of regenerating SQL.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The dashboard becomes a cache of vetted questions, and the agent becomes the long tail around it. Certified paths get the reliability of reviewed work. Novel questions get flexibility, clearly labeled as unreviewed.&lt;/p&gt;

&lt;p&gt;That label matters. Users should be able to tell "this came from a certified definition" from "I improvised this just now." Trust calibrates correctly only if the system exposes the difference.&lt;/p&gt;

&lt;h2&gt;
  
  
  What This Changes About the Job
&lt;/h2&gt;

&lt;p&gt;If agents write the SQL, the analytics engineer's value doesn't vanish. It moves from writing queries to &lt;strong&gt;curating the question space&lt;/strong&gt;: deciding which interpretations are canonical, retiring definitions that conflict, and reading divergence logs to find where the business itself is ambiguous.&lt;/p&gt;

&lt;p&gt;Those logs are surprisingly revealing. When two interpretations of "active customer" keep producing different numbers, the agent didn't fail. It found a real disagreement in your organization that dashboards had been papering over by letting each team keep its own chart.&lt;/p&gt;

&lt;p&gt;The takeaway for anyone shipping these systems: stop measuring only whether the SQL is right. Measure whether the interpretation was visible, whether material ambiguity was surfaced, and how often an unreviewed question earns its way into a reviewed one. An agent that is right 90% of the time and hides which 10% is worse than one that is right 80% of the time and tells you where to look.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>architecture</category>
      <category>sql</category>
    </item>
    <item>
      <title>LLM Columns in dbt Break Idempotency. Cache by Hash, Not Run</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Thu, 12 Nov 2026 14:15:43 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/llm-columns-in-dbt-break-idempotency-cache-by-hash-not-run-28m3</link>
      <guid>https://dev.to/aiexplore369zoho/llm-columns-in-dbt-break-idempotency-cache-by-hash-not-run-28m3</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR —&lt;/strong&gt; A dbt model is supposed to be a pure function of its inputs. An LLM call inside a model is not, so full refreshes, CI diffs, and tests all quietly stop meaning what they used to. Treat LLM-derived columns as a content-addressed ledger keyed on input, prompt, and model identity, and let dbt join to it instead of regenerating it.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The most seductive thing a modern warehouse can do for an AI team is let you call a model from SQL. Classify a support ticket, extract an entity, summarize a contract. One expression in a SELECT, wrapped in a dbt model, and you have an "enriched" table that refreshes on schedule.&lt;/p&gt;

&lt;p&gt;It works on day one. It is also a quiet violation of the contract that makes dbt worth using. My thesis: &lt;strong&gt;an LLM call is not a transformation, it is an observation, and you should store it like one.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The contract you didn't know you were relying on
&lt;/h2&gt;

&lt;p&gt;dbt's whole workflow rests on an assumption that rarely gets stated: a model is a pure function of its upstream relations plus its code. Same inputs, same SQL, same output. Everything downstream leans on that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Full refresh is safe.&lt;/strong&gt; You can drop and rebuild any table and expect equivalence.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Slim CI works.&lt;/strong&gt; If code and upstream data haven't changed, you can skip the model and trust the existing result.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Diffs are meaningful.&lt;/strong&gt; A change in output means a change in logic or data, so you can review it.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Incremental models are an optimization,&lt;/strong&gt; not a semantic choice. They should converge on what a full rebuild would produce.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Put a model call inside the SELECT and every one of these breaks, each in a slightly different way.&lt;/p&gt;

&lt;h2&gt;
  
  
  How it fails in practice
&lt;/h2&gt;

&lt;p&gt;Start with the full refresh. Someone runs it to fix an unrelated schema issue. Every row is re-sent to the model. Sampling variance, a silently updated model alias, or a tweaked prompt means a few percent of your categorical labels flip. No SQL changed. Dashboards that group by that label move, and nobody can explain why from the git history.&lt;/p&gt;

&lt;p&gt;Then incremental models. They now encode a hidden policy: rows processed last month were labeled by whatever model was live then, and rows processed today by whatever is live now. Your table is a palimpsest of model versions with no column saying which is which. A full rebuild would not reproduce it. The "optimization" has become the source of truth.&lt;/p&gt;

&lt;p&gt;Then CI. Slim CI sees no modified code and skips the model. But if the prompt lives in a variable, a config table, or the endpoint behind an alias, the semantics changed and the checker is blind to it. Conversely, if CI does rebuild the model, you pay for per-row inference on every pull request and get different outputs each time, so the diff is noise.&lt;/p&gt;

&lt;p&gt;Finally, tests. &lt;code&gt;not_null&lt;/code&gt;, &lt;code&gt;unique&lt;/code&gt;, and &lt;code&gt;accepted_values&lt;/code&gt; pass happily on a column that is 15% wrong. These tests check shape. LLM output fails in meaning, which is the one place dbt's built-in tests don't look.&lt;/p&gt;

&lt;h2&gt;
  
  
  Content-addressed enrichment: a ledger, not a column
&lt;/h2&gt;

&lt;p&gt;The fix is not to avoid LLMs in the warehouse. It is to separate &lt;em&gt;generating&lt;/em&gt; an observation from &lt;em&gt;using&lt;/em&gt; it.&lt;/p&gt;

&lt;p&gt;Build an append-only enrichment ledger. Each row is keyed by a hash of everything that determines the output:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;the normalized input text (or its hash),&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;a prompt identifier with its own version, treated as a deployable artifact,&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;a pinned model identifier, never a floating alias,&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the decoding parameters that matter, such as temperature and output schema.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The ledger stores the output, a timestamp, and the key components in plain columns. It is written by a separate job, not by a dbt SELECT. That job asks one question per row: &lt;em&gt;does a ledger entry already exist for this key?&lt;/em&gt; If yes, do nothing. If no, call the model and append.&lt;/p&gt;

&lt;p&gt;Your dbt model then does what dbt is good at: a deterministic join. Inputs plus a pinned (prompt, model) pair select a ledger row. Rebuild it a hundred times and you get the same table, because the nondeterminism was captured once at write time and frozen.&lt;/p&gt;

&lt;p&gt;This is just memoization with a content-addressed key, the same idea behind build caches and artifact stores. The model call becomes a cacheable, replayable side effect, and the transformation layer stays pure.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this buys you
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Upgrades become explicit events.&lt;/strong&gt; Changing a prompt or model means bumping a pinned version in config. That is a reviewable diff in git. The new version backfills into new ledger rows; old rows remain. You can run both versions side by side, compare them, and cut over deliberately, rather than discovering the change in a metric review.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Full refresh becomes free and safe.&lt;/strong&gt; Rebuilding the dbt model costs warehouse compute for a join, not inference for every row. The model is only re-invoked for genuinely new keys.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;CI gets honest.&lt;/strong&gt; Because prompt and model identity are part of the model's config, state comparison sees the change. And in CI you can point the join at a small frozen fixture of ledger rows, so tests are deterministic and cost nothing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lineage includes the model.&lt;/strong&gt; Every enriched value can answer "which prompt and which model produced you?" That matters when a downstream consumer, human or another agent, asks why a record is labeled the way it is.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tests that check meaning
&lt;/h2&gt;

&lt;p&gt;The ledger gives you something to test against, and the tests have to be statistical, not structural. A few that pay for themselves:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Distribution stability.&lt;/strong&gt; Compare the label mix of a new version's outputs to the previous version on the same inputs. A big shift is a review trigger, not necessarily a failure.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Golden-set agreement.&lt;/strong&gt; Keep a small, human-labeled sample and assert a minimum agreement rate on every version bump.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Self-consistency on a slice.&lt;/strong&gt; Re-run a sample of keys and measure how often the output flips. High flip rate means the task is under-specified or the decoding is too loose, and no downstream join will fix that.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Schema conformance at write time.&lt;/strong&gt; Reject malformed outputs before they enter the ledger, so the warehouse never stores something it can't type.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These run on the ledger job, not on every dbt build. dbt's role is to enforce that downstream models only consume rows from approved (prompt, model) pairs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this fits with the semantic layer
&lt;/h2&gt;

&lt;p&gt;Semantic layers define metrics over governed columns. If one of those columns is an LLM-derived label, then the metric inherits its nondeterminism. "Tickets by category" changing between Tuesday's and Wednesday's queries is the sort of bug that destroys trust in the whole layer. Pinning the enrichment version into the metric's definition makes it a first-class dimension of the metric: &lt;em&gt;category as of classifier v3&lt;/em&gt;. That is a boring, auditable statement, and boring is the goal.&lt;/p&gt;

&lt;h2&gt;
  
  
  The objection: it's more plumbing
&lt;/h2&gt;

&lt;p&gt;It is. A ledger table, a writer job, and a version registry are more moving parts than one SQL expression. But the single-expression version defers that cost rather than eliminating it. You pay later, in unreproducible dashboards, surprise inference bills from a routine refresh, and debugging sessions where the honest answer is "the model changed and we didn't record it."&lt;/p&gt;

&lt;p&gt;The rule of thumb I'd give any team: if a value came from a model, it is data you &lt;em&gt;collected&lt;/em&gt;, not data you &lt;em&gt;computed&lt;/em&gt;. Collected data gets an append-only log, provenance, and a deliberate process for replacing it. Computed data gets dbt. Keep the two on opposite sides of a join and both stay trustworthy.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>architecture</category>
      <category>llm</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Machine Learning Models Never Throw, So Data Bugs Ship Silently</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Wed, 11 Nov 2026 14:15:49 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/machine-learning-models-never-throw-so-data-bugs-ship-silently-495j</link>
      <guid>https://dev.to/aiexplore369zoho/machine-learning-models-never-throw-so-data-bugs-ship-silently-495j</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR —&lt;/strong&gt; A model is a total function: it returns a plausible answer for almost any input, including broken ones. That makes upstream data outages invisible to conventional monitoring. The fix is to treat imputation and fallback rates as first-class SLIs, carry provenance for every feature value, and give the serving path an explicit abstain option.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Ordinary software fails loudly. Pass a null where a parser expects a string and you get a stack trace, an alert, and a ticket. Machine learning systems have no such reflex. A model is a total function. Hand it a vector, any vector, and it hands back a number.&lt;/p&gt;

&lt;p&gt;That is the root of most data quality incidents I have seen in production ML. The data was wrong, nothing threw, and the model answered anyway. My thesis: &lt;strong&gt;the robustness we build into models and feature layers is exactly what makes data failures invisible.&lt;/strong&gt; Observability for ML starts by undoing some of that robustness on purpose.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Outage That Looks Like Normal Traffic
&lt;/h2&gt;

&lt;p&gt;Here is a pattern worth knowing by heart. A feature lookup depends on an upstream service. The service gets slow, the lookup times out, and the client library, reasonably, returns a configured default. The model receives a zero where it expected a rolling average of recent activity.&lt;/p&gt;

&lt;p&gt;A gradient-boosted tree takes the branch it learned for that value. A neural net computes a perfectly finite logit. Latency is fine, error rate is zero, and the score distribution moves a little. Nobody pages anyone, because nothing in the request path was an error.&lt;/p&gt;

&lt;p&gt;Every layer behaved correctly in isolation. The client's default was documented. The model handled the value because training data contained missingness too. The serving layer returned a valid response. The failure exists only in the composition, and no single component owns it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Graceful Degradation Destroys Information at the Boundary
&lt;/h2&gt;

&lt;p&gt;Each fallback is a lossy conversion of an &lt;em&gt;error&lt;/em&gt; into a &lt;em&gt;value&lt;/em&gt;. Once the conversion happens, the downstream system cannot tell "this user has no history" from "we failed to fetch this user's history." Those are different facts with different correct behaviors, and they now share the same bytes.&lt;/p&gt;

&lt;p&gt;Aircraft instrumentation gets this right. A failed sensor does not display zero altitude. It displays a flag, and pilots are trained to treat the flag as information. A feature pipeline that substitutes a default for a failed read is the equivalent of painting zero on the altimeter.&lt;/p&gt;

&lt;p&gt;Three common mechanisms do this conversion:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Client-side defaults&lt;/strong&gt; in feature-store SDKs that swallow timeouts.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Imputation steps&lt;/strong&gt; baked into the model's preprocessing, which run identically whether the value was missing for a legitimate reason or an infrastructural one.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Stale reads&lt;/strong&gt;, where a cache or a lagging materialization returns an old but well-formed value. This is the hardest to catch because nothing is even missing.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Schema validation and null-rate checks on the training table do not see any of this. They validate the data at rest. The damage happens at read time, per request, in the serving path.&lt;/p&gt;

&lt;h2&gt;
  
  
  Instrument Provenance, Not Just Values
&lt;/h2&gt;

&lt;p&gt;The fix is not more checks on columns. It is carrying a small amount of metadata alongside every feature value so the serving layer can say how it got that value. At minimum, each value should be tagged as one of:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;observed&lt;/strong&gt;: fetched fresh from the source of truth.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;stale&lt;/strong&gt;: fetched, but older than its freshness budget.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;imputed&lt;/strong&gt;: filled by a deliberate, learned or configured rule.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;defaulted&lt;/strong&gt;: filled because a read failed.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;From those tags you can compute the metrics that matter. The most useful is the &lt;strong&gt;fallback rate per feature, per segment&lt;/strong&gt;. Freshness at &lt;em&gt;read&lt;/em&gt; time is the next one, not freshness at write time. A table that was updated on schedule can still be served from a replica that is hours behind.&lt;/p&gt;

&lt;p&gt;Segmentation is not optional. Aggregate fallback rates are dominated by the majority cohort, and outages are rarely uniform. A regional dependency fails and one geography gets defaulted features while the global number barely moves. Slice by whatever dimension your infrastructure fails along: region, tenant, entity type, upstream source.&lt;/p&gt;

&lt;p&gt;You also need a baseline that separates &lt;em&gt;expected&lt;/em&gt; missingness from &lt;em&gt;induced&lt;/em&gt; missingness. New users legitimately have no history. The signal is the fallback rate conditioned on cohort, compared against that cohort's normal level. An alert on "null rate above X" is noisy. An alert on "defaulted rate for established users departed from its own history" is not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Give the Serving Path a Way to Say No
&lt;/h2&gt;

&lt;p&gt;Observability that only records the problem is a postmortem tool. The stronger move is to make the serving path act on provenance. Instead of a binary serve-or-error, define three outcomes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Serve&lt;/strong&gt;: inputs are within contract, use the primary model.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Degrade&lt;/strong&gt;: some features are defaulted or stale beyond a threshold, so route to a simpler model trained to not depend on them, or to a rules baseline.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Abstain&lt;/strong&gt;: too much of the input is untrustworthy, so return an explicit "no prediction" that the caller must handle.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is a circuit breaker, applied to data instead of dependencies. The key design choice is that the degrade path must be a model you trained and evaluated for that regime, not an accident. A model trained with random feature dropout can handle a missing feature on purpose. A model that merely tolerates one is guessing.&lt;/p&gt;

&lt;p&gt;Abstain is the uncomfortable one, because product teams read it as downtime. But a confident wrong score is usually more expensive than an honest absence, especially when the score feeds another automated decision. Downstream systems can only behave sensibly if the abstention is visible to them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Same Failure Wears an LLM Costume
&lt;/h2&gt;

&lt;p&gt;This is not just a tabular-model problem. In a RAG system, a retrieval failure that returns zero documents does not stop the model. It answers fluently from parametric memory, and the response looks identical in shape to a grounded one. An agent whose tool returns an empty list will often continue as if the empty list were a finding.&lt;/p&gt;

&lt;p&gt;The same discipline applies. Log whether context was retrieved, how much, and how fresh. Track the &lt;strong&gt;empty-context rate&lt;/strong&gt; and the &lt;strong&gt;tool-returned-nothing rate&lt;/strong&gt; as SLIs. Give the generation step an abstain path ("I could not find supporting material") that the pipeline enforces rather than hopes for. Fluency is the LLM equivalent of a finite logit: it tells you the function ran, not that the input was sound.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to Do Monday
&lt;/h2&gt;

&lt;p&gt;Start with an audit, not a platform. Walk the serving path for one model and list every place an error can become a value: SDK defaults, preprocessing imputers, cache fallbacks, &lt;code&gt;try/except&lt;/code&gt; blocks that return a constant. Each one is a silent failure site.&lt;/p&gt;

&lt;p&gt;Then, in order:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Emit a provenance tag for every feature on every request, even if you only sample the logs.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Dashboard the defaulted and stale rate by cohort next to your latency and error panels.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Set a budget, and treat breaching it as an incident even when accuracy proxies look fine.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Train the fallback model deliberately, then wire the breaker to it.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Models are built to be forgiving, and that is a virtue in the weights and a vice in the system. If the system never complains, it is not healthy. It just has no way to tell you.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>architecture</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Flink Watermarks Are a Hidden Hyperparameter in Real-Time ML</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Tue, 10 Nov 2026 14:15:49 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/flink-watermarks-are-a-hidden-hyperparameter-in-real-time-ml-50d9</link>
      <guid>https://dev.to/aiexplore369zoho/flink-watermarks-are-a-hidden-hyperparameter-in-real-time-ml-50d9</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR —&lt;/strong&gt; A watermark in a Kafka and Flink pipeline decides how much of the world a real-time feature has seen when the model reads it. Training on a batch replay sees a more complete world than production did, which creates a directional, systematic skew. Treat lateness policy as a versioned model parameter, log what serving actually saw, and measure lateness per source.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Most teams tune the learning rate, the context window, and the retrieval top-k. Almost nobody tunes the watermark. Yet in a real-time ML system built on Kafka and Flink, the watermark decides how much of reality a feature has seen at the moment the model reads it. It shapes your predictions as much as any model setting does, and it never shows up in an experiment tracker.&lt;/p&gt;

&lt;p&gt;My thesis: &lt;strong&gt;the lateness policy of your stream processor is a hyperparameter of your model, and it should be versioned, logged, and evaluated like one.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What a watermark actually promises
&lt;/h2&gt;

&lt;p&gt;Event time is when something happened. Processing time is when your system found out. In a clean world they are the same. In the real world they diverge because of mobile clients that go offline, retries, partner feeds delivered in batches, cross-region replication, and consumer lag after a deploy.&lt;/p&gt;

&lt;p&gt;A watermark is the stream processor's heuristic claim: "I believe I have seen everything with an event time earlier than T." It is a guess, usually a bounded out-of-orderness assumption such as "events arrive at most N seconds late." When the watermark passes the end of a window, the window fires. Events that show up afterward are dropped, routed to a side output, or trigger a corrected result, depending on how you configured allowed lateness.&lt;/p&gt;

&lt;p&gt;Take a feature like "number of card transactions in the last ten minutes." The value the model sees at request time is not the true count. It is the count of events that arrived before the watermark closed the relevant window, plus whatever you do with stragglers. That is a different quantity, and it depends on a number somebody picked during initial development.&lt;/p&gt;

&lt;h2&gt;
  
  
  The skew is directional, not random
&lt;/h2&gt;

&lt;p&gt;Training/serving skew is usually described as a bug: two code paths computing the same thing differently. Watermark skew is subtler because both paths can be correct and still disagree.&lt;/p&gt;

&lt;p&gt;Here is the typical setup. Serving reads features computed by the live streaming job. Training data comes from a batch job that replays the full event log, or from a warehouse table built after the fact. By the time the batch job runs, every late event has landed. The training feature reflects the complete history. The serving feature reflects the history as of the watermark.&lt;/p&gt;

&lt;p&gt;So the training set is systematically &lt;em&gt;cleaner and more complete&lt;/em&gt; than what production ever serves. The model learns, for example, that a count of seven means something specific. In production, the same underlying situation produces a count of five, because two events were still in flight. The model is not seeing noise. It is seeing a consistent undercount that training told it never happens.&lt;/p&gt;

&lt;p&gt;It gets worse because late data is rarely random. Events arrive late for reasons that correlate with the label:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Offline or poorly connected clients often differ in behavior from well-connected ones.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Retry storms happen during incidents, which are exactly when anomalies cluster.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Batched partner feeds carry a distinct population with its own base rate.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Fraudulent activity may be deliberately shaped to look like normal, delayed traffic.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the late tail is informative, then dropping it online and including it offline produces a skew aligned with the signal you care about. Aggregate metrics will not flag this, since the features look healthy and the pipeline is green. The model just underperforms in a way that looks like drift.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two failure modes that live in Kafka partitions
&lt;/h2&gt;

&lt;p&gt;Watermark behavior across partitions adds operational traps. A stream processor typically advances its watermark based on the slowest input. That has two consequences.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;An idle partition stalls everything.&lt;/strong&gt; If one Kafka partition stops receiving data, because of a low-traffic key range or an upstream outage, the combined watermark can stop advancing. Windows stop firing. Your features go stale while the job reports healthy throughput. Serving keeps happily reading the last computed values.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix for that creates the opposite problem.&lt;/strong&gt; Configuring an idle timeout lets the watermark move on without the quiet partition. That unblocks the pipeline, but when the partition wakes up, its events are now late relative to a watermark that has moved past them. You traded staleness for silent data loss, and the loss concentrates in whichever keys live on the partition that went quiet.&lt;/p&gt;

&lt;p&gt;Neither behavior is a bug. Both are defaults interacting with traffic shape. That is why they belong in your model's risk register rather than only in the platform team's runbook.&lt;/p&gt;

&lt;h2&gt;
  
  
  Treat lateness as a model parameter
&lt;/h2&gt;

&lt;p&gt;The goal is not to eliminate lateness. Waiting longer for completeness costs latency, and a fraud decision that arrives after the transaction clears is worth little. The goal is to make the tradeoff explicit and make training match it. Several practices follow from that.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Log what serving saw.&lt;/strong&gt; The most reliable training data for a real-time model is the feature vector actually served at prediction time, joined to the label later. It encodes your real watermark behavior, including incident days and idle-partition stalls. Reconstructing features after the fact is the path that produces the skew.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;If you must replay, replay under the same policy.&lt;/strong&gt; Run the same streaming logic over recorded events with the same lateness configuration, deterministically in event time. A batch SQL reimplementation with no lateness cutoff is a different feature, even if it shares a name.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Version the lateness config with the model.&lt;/strong&gt; Allowed lateness, out-of-orderness bounds, and idle timeouts should be pinned alongside the model artifact. Changing the watermark delay for a latency win is a model change. Ship it with an evaluation, not a quiet config edit.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Measure lateness per source.&lt;/strong&gt; Track the distribution of event-time-to-arrival gaps for each upstream, not one global p99. A tail that looks fine overall can hide one feed that is consistently minutes behind.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Count what you drop.&lt;/strong&gt; Route late events to a side output and monitor the volume, ideally sliced by the features and segments your model cares about. A rising dropped-event rate is an early warning that precedes any accuracy metric.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Expose freshness to the model or the policy around it.&lt;/strong&gt; Passing a "data age" or completeness signal as a feature lets the model discount a count it knows is partial. At minimum, serving logic can fall back to a safer default when freshness exceeds a threshold.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The cost curve belongs to the business, not the platform
&lt;/h2&gt;

&lt;p&gt;Who decides how long to wait? Platform teams tend to pick a watermark delay that keeps latency dashboards green. Data scientists assume features are complete because the offline evaluation said so. Nobody owns the gap between them.&lt;/p&gt;

&lt;p&gt;The right owner is whoever can price the tradeoff. For some use cases, such as recommendations or ranking, a slightly undercounted feature costs little, and low latency wins. For others, such as fraud or risk, the late tail may carry the most valuable signal, and a longer wait or a two-stage design is justified: a fast provisional score now, a corrected score when the window closes. Streaming engines support emitting early results and refinements, and ML systems should use that capability deliberately rather than treating a window as a single-shot answer.&lt;/p&gt;

&lt;p&gt;The broader point is that "real-time" is not a property you get from Kafka and Flink. It is a negotiated position between freshness and completeness, and that position leaks directly into model behavior. If you can recite your model's learning rate but not its allowed lateness, you are tuning the wrong knob and ignoring the one that production is tuning for you.&lt;/p&gt;

&lt;p&gt;Put the watermark in the experiment config. Put the served features in the training set. Then the skew has nowhere to hide.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>architecture</category>
      <category>machinelearning</category>
      <category>dataengineering</category>
    </item>
    <item>
      <title>A RAG Index Is a Materialized View, So Stop Treating It Like a Cache</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Mon, 09 Nov 2026 14:15:46 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/a-rag-index-is-a-materialized-view-so-stop-treating-it-like-a-cache-3bmp</link>
      <guid>https://dev.to/aiexplore369zoho/a-rag-index-is-a-materialized-view-so-stop-treating-it-like-a-cache-3bmp</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR —&lt;/strong&gt; Enterprise RAG indexes are derived data over lakehouse tables, which makes them materialized views with all the maintenance problems that implies. Cron re-embedding, a single freshness number, and text copied into the vector store all fail in predictable ways. Store pointers and content hashes, hydrate from the source at query time, and pin answers to a snapshot.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Ask a team how their enterprise RAG index stays fresh and you will usually hear the word "refresh." A nightly job scans the lakehouse, chunks documents, embeds them, and upserts into a vector store. When someone complains the assistant quoted last quarter's numbers, the fix is to run the job more often.&lt;/p&gt;

&lt;p&gt;That is cache thinking, and it is the wrong mental model. A cache is a disposable copy of something you can always re-fetch. A RAG index over lakehouse data is a &lt;strong&gt;derived dataset with its own schema, its own cost to rebuild, and its own correctness contract&lt;/strong&gt;. That is a materialized view. Once you say it that way, most of the hard problems already have names.&lt;/p&gt;

&lt;h2&gt;
  
  
  The unit mismatch nobody budgets for
&lt;/h2&gt;

&lt;p&gt;The lakehouse tracks change at the level of files, partitions, and rows. Retrieval operates on chunks. Embedding is the expensive step, so you want to re-embed only what actually changed. But a changed row does not tell you which chunks changed, and a rewritten file does not tell you which rows did.&lt;/p&gt;

&lt;p&gt;Naive pipelines resolve this by re-embedding everything touched by a changed file. Compaction and rewrite operations, which are routine table maintenance, then look like mass edits. Your embedding bill spikes because someone optimized file layout, not because any content changed.&lt;/p&gt;

&lt;p&gt;The fix is the same one incremental view maintenance has always used: a stable key and a content hash.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Every chunk gets a deterministic ID derived from the source record key and its position or span within it.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Every chunk stores a hash of its normalized content plus the chunking and embedding configuration.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;On each run, compute the desired set of (ID, hash) pairs from the change feed, diff against the index, and embed only the delta.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A file rewrite that changes no content produces zero embedding calls. A chunking-config change produces a full rebuild, and it should, because that is a schema migration, not a refresh.&lt;/p&gt;

&lt;h2&gt;
  
  
  Freshness is a vector, not a number
&lt;/h2&gt;

&lt;p&gt;"The index is 20 minutes behind" is almost never a true statement. An index fed by five source tables has five watermarks, and they advance independently. The CRM table may be seconds old while the contracts table last landed yesterday because an upstream job failed.&lt;/p&gt;

&lt;p&gt;This matters because RAG answers routinely join facts across sources. If one retrieved chunk reflects Tuesday's pricing and another reflects Thursday's discount policy, the model will confidently synthesize a statement that was never true at any point in time. This is a torn read, and databases solved it decades ago with snapshot isolation.&lt;/p&gt;

&lt;p&gt;Table formats with snapshots give you the primitive. Practical consequences:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Track a watermark per source,&lt;/strong&gt; expressed as the snapshot or version of the table that was last fully indexed.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Expose it at query time.&lt;/strong&gt; Retrieval results should carry the source version each chunk came from, so the application can see that the evidence spans different points in time.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Surface staleness to the model and the user.&lt;/strong&gt; An "as of" annotation in the context costs a few tokens and prevents a class of silent errors. For some queries, such as anything involving balances or inventory, a stale source should trigger a fallback to a live query rather than a hedged answer.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Not every question needs consistency across sources. But you cannot decide which ones do if the index does not tell you what it knows and when it knew it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Store pointers, hydrate at read time
&lt;/h2&gt;

&lt;p&gt;The most consequential design choice is what the vector store actually holds. The default is to put the chunk text next to the embedding and return it directly. It is simple and fast, and it makes the index a second copy of your data that has to be kept correct forever.&lt;/p&gt;

&lt;p&gt;The alternative is late binding. The index stores the embedding, the chunk ID, the source key, and the content hash. Vector search is used only for &lt;em&gt;candidate generation&lt;/em&gt;. Before anything reaches the prompt, a hydration step fetches the current record from the lakehouse by key and checks the hash.&lt;/p&gt;

&lt;p&gt;That one step buys three things at once:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Freshness repair.&lt;/strong&gt; If the hash no longer matches, the chunk is stale. Drop it, or re-chunk that record inline, and enqueue it for reindexing. The index can lag without the answer lagging.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Deletion correctness.&lt;/strong&gt; A record deleted at the source simply fails to hydrate. You are no longer racing a tombstone through the pipeline to stop leaking something that should be gone.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;A single enforcement point for access.&lt;/strong&gt; Hydration goes through the same governed read path as every other consumer, with row filters and column masks applied using the caller's identity. The index never has to model your permission system, which it cannot do faithfully anyway.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The cost is real. Hydration adds a read per candidate, and you will over-fetch candidates because some will be filtered out. Over-fetch factors and tail latency of the hydration store become things you tune. I think that is a good trade. Latency is a budget you can spend; a leaked or fabricated fact is not recoverable by tuning.&lt;/p&gt;

&lt;h2&gt;
  
  
  What breaks when you ignore this
&lt;/h2&gt;

&lt;p&gt;The failures are quiet, which is why teams tolerate them for months.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Phantom facts.&lt;/strong&gt; Updated records leave old chunks behind because the upsert keyed on position, not stable ID. Retrieval returns both versions and the model picks one.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Embedding cost drift.&lt;/strong&gt; Maintenance on the lakehouse turns into re-embedding on the index, and nobody connects the two.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Unreproducible answers.&lt;/strong&gt; Someone asks why the assistant said X last week. Without snapshot versions on the retrieved evidence, you cannot reconstruct what the index returned, so you cannot debug it or audit it.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Silent partial outages.&lt;/strong&gt; One upstream table stalls. The index is "up," retrieval works, and one slice of the corpus is simply getting older. Without per-source watermarks, no alert ever fires.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The design checklist
&lt;/h2&gt;

&lt;p&gt;If you are building or auditing a lakehouse-backed RAG system, ask these questions in order:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Can I derive every chunk deterministically from (source key, source version, chunk config, embedding model)?&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Does the index update from a change feed, with content hashes preventing no-op re-embedding?&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Do I have a watermark per source, and does retrieval output carry it?&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Does the final prompt contain only text that was hydrated from the source under the caller's identity?&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Can I replay a past query against a past snapshot?&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this is novel. It is view maintenance, snapshot reads, and late binding, applied to a new kind of derived table. The novelty is that the consumer is a language model that will happily paper over every inconsistency with fluent prose. Databases surface staleness as wrong numbers; LLMs hide it. That is the strongest argument for building the discipline into the index itself, because the model will never tell you the index is out of date.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>architecture</category>
      <category>machinelearning</category>
      <category>rag</category>
    </item>
    <item>
      <title>Vector Databases at Warehouse Scale Quietly Break Top-K Search</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Sun, 08 Nov 2026 14:16:01 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/vector-databases-at-warehouse-scale-quietly-break-top-k-search-3ag6</link>
      <guid>https://dev.to/aiexplore369zoho/vector-databases-at-warehouse-scale-quietly-break-top-k-search-3ag6</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR —&lt;/strong&gt; Running vector search at warehouse scale forces you to partition the index by tenant, time, or shard — but that partitioning breaks the global top-k guarantee vector search is supposed to provide. You end up stacking two approximations: the ANN algorithm's recall loss, and a merge-across-shards ranking error that almost nobody measures. The fix isn't a better index, it's treating top-k as a distributed query problem with its own correctness budget.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Every vector database pitch includes a promise: give me a query embedding, I'll give you the k nearest neighbors. That promise is already approximate — HNSW, IVF, and their cousins trade exact nearest-neighbor search for speed, and everyone building on them accepts that trade. What almost nobody accounts for is the second approximation that shows up the moment you scale past a single index: partitioning.&lt;/p&gt;

&lt;p&gt;At warehouse scale — hundreds of millions to billions of vectors, multi-tenant, continuously ingesting — you cannot keep one monolithic index. You shard it. And sharding doesn't just change where data lives. It changes what "top-k" means, in ways that degrade retrieval quality silently, long after your latency dashboards look fine.&lt;/p&gt;

&lt;h2&gt;
  
  
  Top-K Is a Global Claim, Not a Local One
&lt;/h2&gt;

&lt;p&gt;When someone asks for the top 10 nearest vectors to a query, the implicit claim is global: out of everything in the collection, these are the closest 10. A single HNSW graph over the full dataset approximates that global claim reasonably well — recall loss comes from the graph's traversal heuristics, not from missing data.&lt;/p&gt;

&lt;p&gt;The moment you split that graph into shards — by tenant, by time window, by geography, by hash — you've changed the query from "find the 10 closest vectors in the dataset" to "find the 10 closest vectors in each shard, then merge." Those are not the same operation, and the gap between them grows with shard count and with how skewed your data distribution is across shards.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Warehouse Scale Forces the Split
&lt;/h2&gt;

&lt;p&gt;This isn't optional at scale. A few forces push every vector database toward partitioning once you're past the single-node, single-tenant demo:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Index build cost.&lt;/strong&gt; HNSW graphs and IVF clusters are expensive to build and even more expensive to rebuild. Past a certain size, you shard so that ingestion and compaction stay tractable instead of becoming a full-collection rebuild every time.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Tenant isolation.&lt;/strong&gt; Multi-tenant RAG systems shard by tenant for both performance and data-governance reasons — one noisy tenant shouldn't degrade another's query latency, and one tenant's vectors shouldn't leak into another's results through a shared index.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Hot/cold separation.&lt;/strong&gt; Time-partitioned data — logs, chat history, event embeddings — gets sharded by time so recent data stays in fast, memory-resident indexes while older shards move to cheaper storage tiers.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Replica and placement constraints.&lt;/strong&gt; Distributed deployments shard for the same reasons distributed databases always shard: no single node can hold everything in memory, and you want parallelism across the fleet.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these are mistakes. They're the correct engineering response to scale. But each one turns a single approximate search into a fan-out-and-merge problem, and the merge step is where correctness quietly leaks out.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Merge Problem Nobody Benchmarks
&lt;/h2&gt;

&lt;p&gt;Here's the mechanic. You fan a query out to &lt;em&gt;n&lt;/em&gt; shards, each shard's index returns its local top-k using its own approximate search, and a coordinator merges those &lt;em&gt;n&lt;/em&gt; lists by score to produce the final top-k. This works correctly only if two conditions hold: every shard's local top-k actually contains the true global top-k candidates from that shard, and scores are comparable across shards.&lt;/p&gt;

&lt;p&gt;Both conditions fail in practice, in ways that compound rather than cancel.&lt;/p&gt;

&lt;p&gt;First, local top-k from an ANN index is already a lossy sample of that shard's true top-k. If you ask each shard for its local top 10 and merge, you're assuming the true global top 10 is distributed such that no shard needed to contribute more than 10 strong candidates. At warehouse scale with skewed data — one tenant's shard is denser in the query's semantic neighborhood than others — that assumption breaks. The shard holding the actual best matches might return a locally-correct top 10 that excludes some of the globally-best vectors simply because the local index's graph traversal didn't surface them given the query's entry point.&lt;/p&gt;

&lt;p&gt;Second, score comparability across shards is not guaranteed unless every shard uses identical index parameters, identical distance metrics, and ideally was built from the same embedding model version. In practice, shards get rebuilt at different times, with different index configs, sometimes even different quantization settings for storage efficiency. A cosine similarity score of 0.82 from a freshly rebuilt HNSW shard and a 0.82 from an older, more heavily quantized shard are not the same confidence of match — but the merge step treats them as if they were.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where This Actually Bites
&lt;/h2&gt;

&lt;p&gt;This failure mode doesn't show up as an outage. It shows up as a slow, hard-to-diagnose quality regression. A RAG system's retrieval quality drifts downward as the underlying collection grows and gets re-sharded, even though the embedding model hasn't changed and the index build logs show no errors. Support tickets say "the assistant used to find the right document, now it sometimes doesn't," and the on-call engineer checks latency, checks index health, checks the embedding pipeline — and finds nothing wrong, because nothing in the usual observability surface captures merge-stage ranking error.&lt;/p&gt;

&lt;p&gt;It's worse in filtered search, which is the norm in production RAG rather than the exception. A query that's scoped to a tenant plus a metadata filter plus a time range effectively narrows the candidate shards further, sometimes down to a single shard with a thin population of matching vectors. The local top-k from that thin shard can be a poor approximation of relevance, but the system has no signal that it degraded — the query still returns exactly k results with plausible-looking scores.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Actually Helps
&lt;/h2&gt;

&lt;p&gt;You can't eliminate the shard-and-merge tradeoff at warehouse scale, but you can budget for it instead of ignoring it.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Over-fetch per shard.&lt;/strong&gt; Ask each shard for more than k results — commonly a multiple of k — before merging, so local recall loss has room to be corrected at the merge stage rather than baked in before the coordinator ever sees the candidates.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Normalize scores at build time, not query time.&lt;/strong&gt; Pin distance metric, quantization scheme, and embedding model version per shard, and track those as metadata the coordinator can check before trusting a cross-shard score comparison.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Measure recall against brute force, periodically, on a sample.&lt;/strong&gt; Latency and error-rate dashboards won't catch this. A scheduled job that compares sharded top-k against an exact brute-force search over a sampled subset is the only reliable early-warning system for merge-stage degradation.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Treat shard rebalancing as a correctness event, not just a capacity event.&lt;/strong&gt; When you re-shard for storage or tenancy reasons, re-run the recall benchmark. A rebalance that improves write throughput can silently worsen retrieval quality if it changes how semantically similar vectors get distributed across shards.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Vector search was sold as a drop-in replacement for exact nearest-neighbor lookup, with the understanding that you'd trade a little recall for a lot of speed. That trade is fine in isolation. The problem is that warehouse-scale deployments don't make that trade once — they make it twice, once inside the index and once across the fleet of shards that index gets split into, and only the first one is ever in the benchmark deck.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>architecture</category>
      <category>devops</category>
      <category>programming</category>
    </item>
    <item>
      <title>Data Pipelines Optimize for Throughput. AI Training Needs Determinism</title>
      <dc:creator>AI Explore</dc:creator>
      <pubDate>Sat, 07 Nov 2026 14:16:05 +0000</pubDate>
      <link>https://dev.to/aiexplore369zoho/data-pipelines-optimize-for-throughput-ai-training-needs-determinism-h6g</link>
      <guid>https://dev.to/aiexplore369zoho/data-pipelines-optimize-for-throughput-ai-training-needs-determinism-h6g</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR —&lt;/strong&gt; Data pipelines built for analytics are fault-tolerant by reordering records on retry, parallel reads, and backfills — and nobody notices because SUM() doesn't care about order. Model training does. Nondeterministic ingestion order silently changes which near-duplicates survive dedup, which examples a curriculum sees first, and whether a 'reproduced' run actually reproduces anything. Reliable AI pipelines need an explicit order contract, not just a schema contract.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Every data pipeline team has a quiet assumption baked into its retry logic: order doesn't matter. A partition fails, you reprocess it. A backfill runs out of sequence. Parallel workers read from object storage in whatever order the scheduler hands out shards. None of this breaks a dashboard. &lt;code&gt;SUM()&lt;/code&gt; and &lt;code&gt;COUNT(DISTINCT)&lt;/code&gt; are commutative. Your warehouse doesn't know or care that row 4,201 arrived before row 11,003 this time instead of last time.&lt;/p&gt;

&lt;p&gt;Model training is not commutative. Feed the same underlying data through the same pipeline twice, and if the order of records differs, you can get two meaningfully different training sets — not because anyone changed a filter or a threshold, but because the pipeline's fault-tolerance mechanisms silently reordered things along the way. Nobody wrote a bug. The pipeline did exactly what it was designed to do. It just wasn't designed for a consumer that's sensitive to sequence.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where Order Secretly Changes the Output
&lt;/h2&gt;

&lt;p&gt;Three places this bites teams, in order of how rarely anyone checks for it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Deduplication.&lt;/strong&gt; Near-duplicate detection (fuzzy hashing, embedding similarity, canonicalization) almost always keeps "the first one seen" and drops the rest. First seen according to what order? If your ingestion job reads from five parallel shards and interleaves them nondeterministically, which duplicate survives as canonical changes from run to run. The dataset size is identical. The dedup count is identical. The actual &lt;em&gt;set&lt;/em&gt; of records is not.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Curriculum and batching.&lt;/strong&gt; Anything that assumes examples arrive in a meaningful sequence — difficulty-ordered curricula, recency-weighted streaming fine-tunes, even simple "last N days" windowing against a watermark — depends on stable ordering to mean what you think it means. A late-arriving record that lands one ingestion cycle later than expected doesn't just shift by one row; it can cross an epoch boundary and end up training a different checkpoint entirely.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Checkpoint resume and backfill.&lt;/strong&gt; Resuming a failed ingestion job from an offset, then replaying a backfill against a different partitioning scheme, is routine operational practice. It's also routinely nondeterministic, because the replay almost never guarantees the exact same interleaving as the original run.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this shows up in the checks teams actually run — row counts, null rates, schema conformance, checksum on aggregate stats. Those checks are built to catch the kind of errors analytics pipelines produce: a column goes missing, a type changes, a join fans out. They're structurally blind to the kind of error training pipelines produce: the &lt;em&gt;same&lt;/em&gt; schema, the &lt;em&gt;same&lt;/em&gt; row count, a &lt;em&gt;different&lt;/em&gt; set of surviving records.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why This Was Never a Problem Before
&lt;/h2&gt;

&lt;p&gt;Shuffling isn't new to ML — every training loop shuffles data on purpose, with a seed, specifically to randomize gradient updates in a controlled way. That's intentional and reproducible: same seed, same shuffle, same result. The problem isn't that order changes. It's that &lt;em&gt;two different kinds&lt;/em&gt; of reordering are happening at once, and only one of them is under your control.&lt;/p&gt;

&lt;p&gt;There's the shuffle you chose — seeded, logged, reversible. And there's the shuffle the pipeline gave you for free — retries, parallel reads, dynamic partition assignment, out-of-order late arrivals in a streaming source. The second kind is invisible because it lives below the abstraction your training code operates on. By the time your dataloader sees the data, both kinds of reordering have already happened, and you can only account for the one you designed.&lt;/p&gt;

&lt;h3&gt;
  
  
  Kafka's Partition Guarantee Isn't a Global Guarantee
&lt;/h3&gt;

&lt;p&gt;This gets worse with streaming sources feeding continuous fine-tuning or RL-style feedback loops. A system like Kafka gives you strict ordering &lt;em&gt;within&lt;/em&gt; a partition — a genuinely useful guarantee that teams correctly rely on. But a training job consuming across partitions, or reading from a topic that's been repartitioned since the last run, has no global ordering guarantee at all. Treating partition-level ordering as if it were dataset-level ordering is a common, understandable, and expensive mistake.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a Reliable Pipeline Actually Promises
&lt;/h2&gt;

&lt;p&gt;"Reliable" in the analytics sense means: the pipeline runs without error and the numbers reconcile. For a pipeline that feeds model training, reliable has to mean something stronger: given the same source data and the same configuration, the pipeline produces the same resulting dataset — not just the same row count and schema, but the same actual set of records, with the same dedup outcomes and the same sequence where sequence is semantically meaningful.&lt;/p&gt;

&lt;p&gt;That's a different contract, and it has to be designed in, not discovered after a training run won't reproduce. A few concrete practices:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sort before you dedup.&lt;/strong&gt; Never let dedup logic depend on arrival order from parallel readers. Sort by a stable key — content hash, source timestamp, a deterministic tiebreaker — before applying any "keep first" rule. This turns an accidental, infrastructure-dependent outcome into a defined, testable one.
**Freeze ing&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>architecture</category>
      <category>devops</category>
    </item>
  </channel>
</rss>
