<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Bhargav </title>
    <description>The latest articles on DEV Community by Bhargav  (@bharqav).</description>
    <link>https://dev.to/bharqav</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4090971%2F357ff43e-9933-4d14-8ccd-f350ea1ef50d.png</url>
      <title>DEV Community: Bhargav </title>
      <link>https://dev.to/bharqav</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/bharqav"/>
    <language>en</language>
    <item>
      <title>Built my own LLM, take a look at the entire journey in this post!</title>
      <dc:creator>Bhargav </dc:creator>
      <pubDate>Sat, 14 Nov 2026 03:16:41 +0000</pubDate>
      <link>https://dev.to/bharqav/built-my-own-llm-take-a-look-at-the-entire-journey-in-this-post-1mkj</link>
      <guid>https://dev.to/bharqav/built-my-own-llm-take-a-look-at-the-entire-journey-in-this-post-1mkj</guid>
      <description>&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/bharqav/i-trained-a-757m-parameter-llm-from-scratch-then-fine-tuned-qwen-instead-heres-everything-i-3edp" class="crayons-story__hidden-navigation-link"&gt;I Trained a 757M-Parameter LLM From Scratch, Then Fine-Tuned Qwen Instead. Here's Everything I Learned.&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/bharqav" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4090971%2F357ff43e-9933-4d14-8ccd-f350ea1ef50d.png" alt="bharqav profile" class="crayons-avatar__image"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/bharqav" class="crayons-story__secondary fw-medium m:hidden"&gt;
              Bhargav 
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                Bhargav 
                
                
              
              &lt;div id="story-author-preview-content-5019934" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/bharqav" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4090971%2F357ff43e-9933-4d14-8ccd-f350ea1ef50d.png" class="crayons-avatar__image" alt=""&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;Bhargav &lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/bharqav/i-trained-a-757m-parameter-llm-from-scratch-then-fine-tuned-qwen-instead-heres-everything-i-3edp" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Nov 12&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/bharqav/i-trained-a-757m-parameter-llm-from-scratch-then-fine-tuned-qwen-instead-heres-everything-i-3edp" id="article-link-5019934"&gt;
          I Trained a 757M-Parameter LLM From Scratch, Then Fine-Tuned Qwen Instead. Here's Everything I Learned.
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/ai"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;ai&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/machinelearning"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;machinelearning&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/python"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;python&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/llm"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;llm&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
            &lt;a href="https://dev.to/bharqav/i-trained-a-757m-parameter-llm-from-scratch-then-fine-tuned-qwen-instead-heres-everything-i-3edp#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              1&lt;span class="hidden s:inline"&gt;&amp;nbsp;comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            17 min read
          &lt;/small&gt;
          
          &lt;div class="only-sidebar-menu-item trusted-visible-block print-hidden align-center"&gt;
                
                    
                
          &lt;/div&gt;

        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;


&lt;/div&gt;


</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>How jepa works and how I operated Qwen to build it out</title>
      <dc:creator>Bhargav </dc:creator>
      <pubDate>Sat, 14 Nov 2026 03:15:54 +0000</pubDate>
      <link>https://dev.to/bharqav/how-jepa-works-and-how-i-operated-qwen-to-build-it-out-2olb</link>
      <guid>https://dev.to/bharqav/how-jepa-works-and-how-i-operated-qwen-to-build-it-out-2olb</guid>
      <description>&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/bharqav/jepa-explained-what-it-actually-does-differently-1p59" class="crayons-story__hidden-navigation-link"&gt;JEPA, Explained: What It Actually Does Differently&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/bharqav" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4090971%2F357ff43e-9933-4d14-8ccd-f350ea1ef50d.png" alt="bharqav profile" class="crayons-avatar__image" width="96" height="96"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/bharqav" class="crayons-story__secondary fw-medium m:hidden"&gt;
              Bhargav 
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                Bhargav 
                
                
              
              &lt;div id="story-author-preview-content-5033096" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/bharqav" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4090971%2F357ff43e-9933-4d14-8ccd-f350ea1ef50d.png" class="crayons-avatar__image" alt="" width="96" height="96"&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;Bhargav &lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/bharqav/jepa-explained-what-it-actually-does-differently-1p59" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Nov 14&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/bharqav/jepa-explained-what-it-actually-does-differently-1p59" id="article-link-5033096"&gt;
          JEPA, Explained: What It Actually Does Differently
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/machinelearning"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;machinelearning&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/ai"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;ai&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/deeplearning"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;deeplearning&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/python"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;python&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
          &lt;a href="https://dev.to/bharqav/jepa-explained-what-it-actually-does-differently-1p59" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left"&gt;
            &lt;div class="multiple_reactions_aggregate"&gt;
              &lt;span class="multiple_reactions_icons_container"&gt;
                  &lt;span class="crayons_icon_container"&gt;
                    &lt;img src="https://assets.dev.to/assets/sparkle-heart-5f9bee3767e18deb1bb725290cb151c25234768a0e9a2bd39370c382d02920cf.svg" width="24" height="24"&gt;
                  &lt;/span&gt;
              &lt;/span&gt;
              &lt;span class="aggregate_reactions_counter"&gt;1&lt;span class="hidden s:inline"&gt;&amp;nbsp;reaction&lt;/span&gt;&lt;/span&gt;
            &lt;/div&gt;
          &lt;/a&gt;
            &lt;a href="https://dev.to/bharqav/jepa-explained-what-it-actually-does-differently-1p59#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              &lt;span class="hidden s:inline"&gt;Add&amp;nbsp;Comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            9 min read
          &lt;/small&gt;
          
          &lt;div class="only-sidebar-menu-item trusted-visible-block print-hidden align-center"&gt;
                
                    
                
          &lt;/div&gt;

        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;


&lt;/div&gt;


</description>
      <category>ai</category>
      <category>deeplearning</category>
      <category>llm</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>JEPA, Explained: What It Actually Does Differently</title>
      <dc:creator>Bhargav </dc:creator>
      <pubDate>Sat, 14 Nov 2026 03:15:16 +0000</pubDate>
      <link>https://dev.to/bharqav/jepa-explained-what-it-actually-does-differently-1p59</link>
      <guid>https://dev.to/bharqav/jepa-explained-what-it-actually-does-differently-1p59</guid>
      <description>&lt;p&gt;I spent a while building a JEPA-style system on top of a truncated Qwen3 backbone, and I want to walk through what it actually is, not as a hype pitch, but as an explanation of the mechanism, what it's good at, and where it genuinely struggles. If you already know what next-token prediction is but have never seen a joint-embedding objective up close, this is written for you.&lt;/p&gt;

&lt;h2&gt;
  
  
  The idea in one paragraph
&lt;/h2&gt;

&lt;p&gt;A transformer trained the usual way learns by predicting the next token: given everything so far, guess the exact next word. JEPA (Joint Embedding Predictive Architecture, a framing Yann LeCun has written about) learns something different: given part of an input, predict the representation of the missing part, not the missing part itself. No vocabulary, no softmax over tens of thousands of token options, no cross-entropy loss. Just two numeric vectors that need to end up close to each other.&lt;/p&gt;

&lt;p&gt;That sounds like a small change. It isn't, and the rest of this post is about why.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why predicting tokens and predicting meaning aren't the same thing
&lt;/h2&gt;

&lt;p&gt;Here's the distinction that actually matters. If a model is trained to predict "bank" after "river," it can get very good at that by learning statistical patterns in text, which word tends to follow which, without ever needing anything you'd call an understanding of what a riverbank is. Token prediction is a proxy task. It correlates with understanding, but it isn't the same thing as understanding, and a model can get a long way on the proxy without the real thing.&lt;/p&gt;

&lt;p&gt;JEPA sidesteps the proxy. Instead of asking "what's the exact next token," it asks "what does the representation of the missing piece look like," and lets the model find whatever internal structure makes that predictable. The hope is that structure ends up looking more like genuine semantic understanding, because there's no shortcut through surface-level word statistics to get there.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvjq13wq1872figqpb8sn.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvjq13wq1872figqpb8sn.webp" alt="Two ways to learn from text" width="800" height="463"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A transformer's loss function is cross-entropy over the vocabulary: did you guess the exact right word, out of tens of thousands of options. JEPA's loss is a distance in embedding space, usually mean-squared error or cosine distance between two vectors. That's the whole difference, mechanically. Everything else follows from it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The architecture, piece by piece
&lt;/h2&gt;

&lt;p&gt;A JEPA system needs three components, and understanding why each one exists is more useful than just naming them.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpytdutmves16zc0pdi59.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpytdutmves16zc0pdi59.webp" alt="Pipeline" width="800" height="560"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The context encoder&lt;/strong&gt; sees the visible part of the input (the part that wasn't masked out) and turns it into a representation. In my setup this is a truncated Qwen3 backbone, using only the early layers rather than the full stack, because the full model's VRAM footprint doesn't fit a single consumer GPU for this kind of training.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The predictor&lt;/strong&gt; is a small network that takes the context encoder's output and tries to guess what the representation of the masked part should look like. It's deliberately lightweight, a couple of linear layers with a nonlinearity between them, because its job isn't to re-derive meaning from scratch, just to extrapolate from what the context encoder already extracted.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The target encoder&lt;/strong&gt; produces the actual representation of the masked part, the thing the predictor is trying to match. This is the piece that makes JEPA work at all, and it deserves its own section.&lt;/p&gt;

&lt;h2&gt;
  
  
  The target encoder trick: why it doesn't just collapse
&lt;/h2&gt;

&lt;p&gt;Here's a problem you'd hit immediately if you tried the naive version of this idea. If you train a network to predict "a representation that matches some other network's output," and both networks are learning at the same time, the easiest way to drive the loss to zero is for both networks to collapse to outputting the same constant vector for every input. Loss goes to zero, and the model has learned nothing. This is a real, well-documented failure mode in joint-embedding self-supervised learning, not a hypothetical edge case.&lt;/p&gt;

&lt;p&gt;The fix is the target encoder. It's a copy of the context encoder's weights, updated not by gradient descent but by an exponential moving average (EMA) of the context encoder's weights over time. Critically, no gradients ever flow into it directly: it's detached from the loss, sometimes called a &lt;strong&gt;stop-gradient&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjv6nud17psgf5ncsu29t.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjv6nud17psgf5ncsu29t.webp" alt="What flows and what doesn't" width="799" height="411"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This matters because the target now moves slowly and smoothly, shaped by training but never directly chasing the loss itself. The predictor has something real and slowly-evolving to aim at, instead of a target that can cheat its way to triviality. Removing the stop-gradient is not a minor tweak, it's the piece that prevents the whole system from collapsing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watching for collapse, because it's a real risk, not a footnote
&lt;/h2&gt;

&lt;p&gt;Collapse doesn't always look dramatic. Loss can go down steadily while the model is quietly mapping every input to nearly the same point in representation space, and if you're only watching the loss curve, you won't notice until downstream performance is clearly broken.&lt;/p&gt;

&lt;p&gt;The way to catch it is to track &lt;strong&gt;effective rank&lt;/strong&gt;, roughly, how many independent directions the representations are actually using. If every input collapses toward the same vector, effective rank drops toward 1. If the model is using its full representational capacity, rank climbs toward the dimensionality of the embedding space. Checking this at multiple levels, token-level, pooled-sentence level, and at both the target and predictor outputs, gives you an early warning that loss alone won't.&lt;/p&gt;

&lt;p&gt;I have effective-rank numbers from an earlier draft of this project, but I can't confirm they come from the final configuration, so I'm leaving them out rather than presenting them as fact.&lt;/p&gt;

&lt;h2&gt;
  
  
  The training loop
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg7tgjsldbywba14qflbh.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg7tgjsldbywba14qflbh.webp" alt="Training loop" width="800" height="425"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Each step looks like this: take a batch of text, mask out some contiguous spans, run the visible tokens through the context encoder, run that output through the predictor, and compute the loss against what the target encoder produces for the actual (unmasked) input. Gradients flow backward through the predictor and the context encoder. The target encoder gets updated afterward, not through backprop, but by nudging its weights toward the context encoder's current weights via the EMA rule. Repeat for however many steps you've budgeted, checkpointing along the way.&lt;/p&gt;

&lt;p&gt;The EMA momentum (how slowly the target moves) is a real hyperparameter with real consequences. Too low, and the target chases the context encoder too closely, reintroducing the collapse risk. Too high, and the target barely moves, so the predictor gets almost no useful signal to learn from. The values I've seen work in this range sit close to the target encoder updating very slowly relative to the context encoder, in the neighborhood of 0.996 to 0.999, but I'd treat that as a starting point to tune, not a universal constant.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two tracks, one backbone
&lt;/h2&gt;

&lt;p&gt;This repo actually contains two separate systems that happen to share the same base model.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb8euce9a9qzsdzwcynff.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb8euce9a9qzsdzwcynff.webp" alt="Repo tracks" width="800" height="503"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Track 1&lt;/strong&gt; is a decision model: given a state description, a question, and a list of candidate answers, it returns a calibrated probability for each candidate in a single forward pass, without generating any text. That's the system I wrote about in an earlier post, including the honest numbers on where it struggles (many-candidate questions, in particular).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Track 2&lt;/strong&gt; is the JEPA pretraining experiment described in this post, a representation-learning method, not a decision-making system. It's evaluated differently too, with linear probes and k-nearest-neighbor classification on the learned representations, rather than decision accuracy.&lt;/p&gt;

&lt;p&gt;They share a backbone and some infrastructure, but they're answering different questions: Track 1 asks "can this model make good decisions," Track 2 asks "can this training method learn useful representations without labels."&lt;/p&gt;

&lt;h2&gt;
  
  
  Design choices and the actual tradeoffs behind them
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Truncating the backbone.&lt;/strong&gt; Using only the early layers of Qwen3 instead of the full 28 keeps the VRAM budget inside what a single consumer GPU can handle. The tradeoff is real: deeper layers typically encode more abstract semantic structure, and truncating trades some of that away for the ability to train at all on accessible hardware.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;LoRA instead of full fine-tuning.&lt;/strong&gt; Low-Rank Adaptation injects small trainable matrices into the attention layers instead of updating every parameter in the model. This is the difference between training a small fraction of the parameters versus all of them, and on an 8GB GPU, it's often the difference between a training run that fits in memory and one that doesn't.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sequence packing instead of padding.&lt;/strong&gt; Padding every short sequence out to a fixed length wastes a lot of compute on padding tokens that carry no signal. Packing concatenates multiple shorter texts together and splits at exact boundaries instead, which keeps more of each batch doing useful work, with a block-causal attention mask ensuring sequences packed together don't leak information across their boundary.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this genuinely breaks or gets uncertain
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Multi-candidate scaling, on the decision side.&lt;/strong&gt; As I covered in the earlier post, accuracy drops as the number of candidates grows, down into the high 30s percent by 8 candidates. That's a real limitation of the current scoring approach, not a minor footnote.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Order invariance is precise but not absolute.&lt;/strong&gt; The scoring head is designed so that permuting candidates doesn't change their individual scores, and at the head level this holds extremely tightly. But once you re-tokenize a request with candidates in a genuinely different order, the backbone's attention computation involves floating-point operations in a different order too, and that introduces a small amount of drift end-to-end. It's small enough not to change which answer gets selected in testing, but it's not a mathematical guarantee, and I'd rather say that plainly than round it up to "perfectly invariant."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Representation collapse is a standing risk, not a solved problem.&lt;/strong&gt; The monitoring (effective rank tracking) catches it, and the stop-gradient plus EMA target architecturally resists it, but this is a known, documented failure mode across this entire family of methods, not something unique to this implementation that's been fully eliminated.&lt;/p&gt;

&lt;h2&gt;
  
  
  A sketch of the training step, for intuition
&lt;/h2&gt;

&lt;p&gt;This isn't a verbatim excerpt from the actual codebase, just a simplified sketch to make the mechanism concrete:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# illustrative only, not the real implementation&lt;/span&gt;
&lt;span class="n"&gt;context_repr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;context_encoder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;masked_input&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;predicted_target&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;predictor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context_repr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;no_grad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;target_repr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;target_encoder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;full_input&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# stop-gradient&lt;/span&gt;

&lt;span class="n"&gt;loss&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;mse_loss&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;predicted_target&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target_repr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;loss&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;backward&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;optimizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# target encoder updates by EMA, not backprop&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p_target&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p_context&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;target_encoder&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context_encoder&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;p_target&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mul_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;momentum&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p_context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;momentum&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;No vocabulary, no softmax, no cross-entropy anywhere in this loop. That absence is the entire point.&lt;/p&gt;
&lt;h2&gt;
  
  
  Try the decision model
&lt;/h2&gt;

&lt;p&gt;The decision model (Track 1) is the part of this repo that's furthest along, though I want to be precise about its current state rather than call it finished: security hardening (auth, rate limiting, safetensors-only checkpoints, pinned model revisions) is implemented and locally tested as of this writing, but hasn't yet been confirmed by CI on a clean push, and broader reliability work (multi-seed reruns, load testing) is still open.&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/bharqav/mini-jepa.git
&lt;span class="nb"&gt;cd&lt;/span&gt; mini-jepa
make quickstart
curl -X POST http://127.0.0.1:8000/v1/decide \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"Authorization: Bearer &amp;lt;your JEV_API_TOKEN&amp;gt;"&lt;/span&gt; \
  -d &lt;span class="s1"&gt;'{
    "state": "Production is down.",
    "questions": [{
      "id": "q1",
      "type": "choice",
      "question": "Severity?",
      "candidates": [
        {"id": "p1", "text": "P1"},
        {"id": "p2", "text": "P2"},
        {"id": "p3", "text": "P3"}
      ]
    }]
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;If you're running locally without setting &lt;code&gt;JEV_API_TOKEN&lt;/code&gt;, you'll need &lt;code&gt;JEV_DEV_MODE=1&lt;/code&gt; instead, and the service will log a warning that auth is off. It won't serve requests unauthenticated otherwise.&lt;/p&gt;
&lt;h2&gt;
  
  
  What I'd actually take away from this
&lt;/h2&gt;

&lt;p&gt;JEPA isn't a drop-in replacement for transformers, and I don't think it's useful to frame it as a horse race. It's a different bet about what the training signal should be: instead of getting good at guessing exact words, get good at building a representation space where prediction in that space is possible. Whether that bet pays off at larger scale is genuinely an open question, not something a single small-scale repo on an 8GB laptop GPU settles either way. What this project does show is the mechanism working end to end, with the collapse risk real and monitored rather than hand-waved away, and with the limitations (order-invariance precision, multi-candidate scaling) reported honestly instead of smoothed over.&lt;/p&gt;

&lt;p&gt;The code, the full architecture, and what's actually verified so far are on GitHub:&lt;/p&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/bharqav" rel="noopener noreferrer"&gt;
        bharqav
      &lt;/a&gt; / &lt;a href="https://github.com/bharqav/mini-jepa" rel="noopener noreferrer"&gt;
        mini-jepa
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      Order-invariant decision models and a JEPA-style latent predictor on a truncated Qwen3 backbone
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div&gt;
&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;mini-jepa&lt;/h1&gt;
&lt;/div&gt;

&lt;div class="markdown-heading"&gt;
&lt;h3 class="heading-element"&gt;Order-invariant decision models and a JEPA-style latent predictor on a truncated Qwen3 backbone&lt;/h3&gt;
&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Zero-token inference · Near-exact candidate-order invariance · Calibrated probabilities · EMA-target self-supervision&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/bharqav/mini-jepa/actions/workflows/ci.yml" rel="noopener noreferrer"&gt;&lt;img src="https://github.com/bharqav/mini-jepa/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"&gt;&lt;/a&gt;
&lt;a href="https://github.com/bharqav/mini-jepa/actions/workflows/codeql.yml" rel="noopener noreferrer"&gt;&lt;img src="https://github.com/bharqav/mini-jepa/actions/workflows/codeql.yml/badge.svg?branch=main" alt="CodeQL"&gt;&lt;/a&gt;
&lt;a href="https://www.python.org/downloads/" rel="nofollow noopener noreferrer"&gt;&lt;img src="https://camo.githubusercontent.com/83581bcff9ca73f74aa3258af455bb2729fe5dd71fce2e6b4908a40ffa674af0/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f707974686f6e2d332e31312532422d626c75653f6c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465" alt="Python 3.11+"&gt;&lt;/a&gt;
&lt;a href="https://github.com/bharqav/mini-jepa/LICENSE" rel="noopener noreferrer"&gt;&lt;img src="https://camo.githubusercontent.com/e2ec9f9b84cbfa4a84d3c943ea5dcaf295c8cee0c665120898bd30ca552b1e39/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c6963656e73652d4d49542d79656c6c6f772e737667" alt="License: MIT"&gt;&lt;/a&gt;
&lt;a href="https://github.com/astral-sh/ruff" rel="noopener noreferrer"&gt;&lt;img src="https://camo.githubusercontent.com/ebb58fd361fda3f2ed4bd261bdeff116a084a6ad25150070a600f7a5243eec01/68747470733a2f2f696d672e736869656c64732e696f2f656e64706f696e743f75726c3d68747470733a2f2f7261772e67697468756275736572636f6e74656e742e636f6d2f61737472616c2d73682f727566662f6d61696e2f6173736574732f62616467652f76322e6a736f6e" alt="Code style: ruff"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;&lt;code&gt;mini-jepa&lt;/code&gt; turns a pretrained causal language model into something that
&lt;strong&gt;answers questions instead of completing text&lt;/strong&gt;. Give it a state, a question
and a set of candidate answers; it returns a calibrated probability
distribution, in a single forward pass, generating &lt;strong&gt;zero tokens&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;It also contains a JEPA-style latent predictor that learns representations by
predicting masked spans in a target encoder's embedding space, rather than by
predicting tokens.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Naming:&lt;/strong&gt; the distribution is &lt;code&gt;mini-jepa&lt;/code&gt;; the importable package is
&lt;code&gt;jev_qwen&lt;/code&gt; (&lt;code&gt;from jev_qwen.infer import DecisionEngine&lt;/code&gt;).&lt;/p&gt;
&lt;/blockquote&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Why build this?&lt;/strong&gt; Autoregressive generation is a poor fit for structured
decision problems: you pay for tokens you did not ask for, you get invalid
output that must be parsed and validated, and you cannot permute the options…&lt;/p&gt;
&lt;/blockquote&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/bharqav/mini-jepa" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


</description>
      <category>machinelearning</category>
      <category>ai</category>
      <category>deeplearning</category>
      <category>python</category>
    </item>
    <item>
      <title>I Trained a 757M-Parameter LLM From Scratch, Then Fine-Tuned Qwen Instead. Here's Everything I Learned.</title>
      <dc:creator>Bhargav </dc:creator>
      <pubDate>Thu, 12 Nov 2026 01:15:11 +0000</pubDate>
      <link>https://dev.to/bharqav/i-trained-a-757m-parameter-llm-from-scratch-then-fine-tuned-qwen-instead-heres-everything-i-3edp</link>
      <guid>https://dev.to/bharqav/i-trained-a-757m-parameter-llm-from-scratch-then-fine-tuned-qwen-instead-heres-everything-i-3edp</guid>
      <description>&lt;p&gt;Most "I built an LLM" posts show you a loss curve going down and stop there.&lt;/p&gt;

&lt;p&gt;This one is different, because my from-scratch model &lt;strong&gt;lost&lt;/strong&gt;, and I think the &lt;em&gt;why&lt;/em&gt; is the most useful thing I can teach you.&lt;/p&gt;

&lt;p&gt;Over the last few weeks I built two language models on the same task (empathetic, supportive dialogue) and compared them head to head:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;A GPT-style transformer trained from random initialization&lt;/strong&gt;: 253M and 757M parameter versions, written in PyTorch, trained on my own hardware.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Qwen2.5-0.5B-Instruct fine-tuned with QLoRA&lt;/strong&gt;, in two stages, then merged back into a single fp16 checkpoint.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Around both I built a serving layer (FastAPI, OpenAI-compatible endpoints, Docker, Prometheus) and a verification harness so every number in this post can be reproduced from the repo.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Repo:&lt;/strong&gt; &lt;a href="https://github.com/bharqav/SLM" rel="noopener noreferrer"&gt;github.com/bharqav/SLM&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;By the end of this article you'll understand:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;how a decoder-only LLM works, from tokens to logits to sampled text&lt;/li&gt;
&lt;li&gt;how I built and trained one, including the data mistakes that wrecked my first runs&lt;/li&gt;
&lt;li&gt;what QLoRA is, with the actual arithmetic&lt;/li&gt;
&lt;li&gt;why a 0.5B pretrained model beat my 757M scratch model, and why that isn't the paradox it looks like&lt;/li&gt;
&lt;li&gt;what I got wrong, measured wrong, and had to correct along the way&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Let's start with the machine itself.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 1: How an LLM actually works
&lt;/h2&gt;

&lt;p&gt;Strip away the hype and a language model does exactly one thing:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Given a sequence of tokens, produce a probability distribution over the next token.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Everything else (chat, code completion, "reasoning") is that one operation applied repeatedly. Here's the full pipeline.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Text becomes tokens
&lt;/h3&gt;

&lt;p&gt;Neural networks eat numbers, not strings. A &lt;strong&gt;tokenizer&lt;/strong&gt; splits text into chunks called tokens and maps each to an integer ID.&lt;/p&gt;

&lt;p&gt;My scratch model uses GPT-2's byte-pair encoding (BPE) via &lt;code&gt;tiktoken&lt;/code&gt;, with a vocabulary of 50,257 tokens. Qwen uses its own tokenizer with a vocabulary of 151,936.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;tiktoken&lt;/span&gt;

&lt;span class="n"&gt;enc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tiktoken&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_encoding&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"gpt2"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"I have been feeling really down lately"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;              &lt;span class="c1"&gt;# a list of integers&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# back to the original text&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;BPE starts from bytes and repeatedly merges the most frequent pairs, so common words become single tokens and rare words get split into pieces. That's why a model can handle any string, including made-up words and code.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Tokens become vectors
&lt;/h3&gt;

&lt;p&gt;Each token ID indexes a row in a giant lookup table called the &lt;strong&gt;embedding matrix&lt;/strong&gt;. For my 757M model that's &lt;code&gt;50,257 x 1,536&lt;/code&gt;, about 77 million parameters, just to turn IDs into vectors.&lt;/p&gt;

&lt;p&gt;But there's a problem. Attention (coming up next) doesn't know word order by itself. "Dog bites man" and "man bites dog" would look identical. So we also inject &lt;strong&gt;position information&lt;/strong&gt;.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;My GPT-style model uses &lt;strong&gt;learned positional embeddings&lt;/strong&gt;: another lookup table, &lt;code&gt;256 x 1,536&lt;/code&gt;, added to the token vectors.&lt;/li&gt;
&lt;li&gt;Qwen uses &lt;strong&gt;RoPE&lt;/strong&gt; (rotary position embeddings), which rotates query/key vectors by position-dependent angles instead of adding a learned table. It extrapolates to longer contexts far more gracefully.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Step 3: The transformer block
&lt;/h3&gt;

&lt;p&gt;The model is a stack of identical &lt;strong&gt;transformer blocks&lt;/strong&gt;. Mine has 24 of them. Each block has two sub-layers wrapped in residual connections:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;x = x + Attention( LayerNorm(x) )
x = x + MLP( LayerNorm(x) )
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two ideas hide in those two lines.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Residual connections&lt;/strong&gt; (&lt;code&gt;x = x + ...&lt;/code&gt;) mean each layer only has to learn a &lt;em&gt;correction&lt;/em&gt; to what's already flowing through. Gradients also get a clean highway back to early layers, which is what makes 24+ layer networks trainable at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pre-norm&lt;/strong&gt; (normalizing &lt;em&gt;before&lt;/em&gt; each sub-layer, not after) keeps activations well-scaled and training stable, especially as depth grows.&lt;/p&gt;

&lt;h4&gt;
  
  
  Self-attention: tokens talking to each other
&lt;/h4&gt;

&lt;p&gt;For every token, attention asks: &lt;em&gt;which earlier tokens matter for predicting what comes next?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Each token's vector is projected into three vectors:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Query (Q):&lt;/strong&gt; "what am I looking for?"&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Key (K):&lt;/strong&gt; "what do I contain?"&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Value (V):&lt;/strong&gt; "what do I pass along if you attend to me?"&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Attention scores are &lt;code&gt;softmax(Q @ K^T / sqrt(d))&lt;/code&gt;, and the output is the score-weighted sum of the values. Here's the heart of my implementation, simplified:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;CausalSelfAttention&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Module&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n_embd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n_head&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dropout&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;n_head&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;n_head&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;c_attn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Linear&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n_embd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;n_embd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# Q, K, V in ONE matmul&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;c_proj&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Linear&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n_embd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n_embd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;      &lt;span class="c1"&gt;# output projection&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dropout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dropout&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;forward&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="no"&gt;B&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;C&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;
        &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;c_attn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;C&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dim&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="c1"&gt;# reshape to (batch, heads, time, head_dim)&lt;/span&gt;
        &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;B&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;n_head&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;C&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;n_head&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transpose&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                   &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="c1"&gt;# is_causal=True applies the lower-triangular mask&lt;/span&gt;
        &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;scaled_dot_product_attention&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;is_causal&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;dropout_p&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dropout&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;training&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transpose&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;contiguous&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;B&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;C&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;c_proj&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details matter here.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Causal masking.&lt;/strong&gt; During training, the model sees a whole sequence at once, so we must stop token &lt;em&gt;t&lt;/em&gt; from peeking at token &lt;em&gt;t+1&lt;/em&gt;, or it would just copy the answer. &lt;code&gt;is_causal=True&lt;/code&gt; applies a lower-triangular mask so each position only attends to itself and the past.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Multiple heads.&lt;/strong&gt; With 1,536 dimensions and 24 heads, each head works in a 64-dimensional subspace. Different heads can specialize: one tracks bracket matching, another subject-verb agreement, another something no human has labeled. We run them in parallel and concatenate.&lt;/p&gt;

&lt;h4&gt;
  
  
  The MLP: where tokens think alone
&lt;/h4&gt;

&lt;p&gt;Attention &lt;em&gt;mixes information between tokens&lt;/em&gt;. The MLP then processes each token's vector &lt;em&gt;independently&lt;/em&gt;: expand to 4x the width (1,536 → 6,144), apply a nonlinearity (GELU in my model), project back down. A large share of a transformer's parameters, and much of its stored "knowledge", live in these MLPs.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Vectors become a distribution
&lt;/h3&gt;

&lt;p&gt;After the final block and a last LayerNorm, each position's vector is multiplied by the embedding matrix &lt;em&gt;again&lt;/em&gt;, this time in reverse, to produce one score (a &lt;strong&gt;logit&lt;/strong&gt;) per vocabulary token. Softmax turns logits into probabilities.&lt;/p&gt;

&lt;p&gt;Sharing the same matrix for input embedding and output projection is called &lt;strong&gt;weight tying&lt;/strong&gt;. It saves ~77M parameters and acts as a regularizer, because the embedding space has to be useful for both reading and predicting.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5: Training is just next-token prediction
&lt;/h3&gt;

&lt;p&gt;Training data is a long stream of tokens. For every position, the target is simply the &lt;em&gt;next&lt;/em&gt; token. The loss is cross-entropy between the predicted distribution and that target:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;logits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                                  &lt;span class="c1"&gt;# (B, T, vocab)&lt;/span&gt;
&lt;span class="n"&gt;loss&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cross_entropy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;logits&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;logits&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;ignore_index&lt;/span&gt;&lt;span class="o"&gt;=-&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;loss&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;backward&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;optimizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's it. No labels beyond the text itself. This is why LLM pretraining scales: the internet is the dataset.&lt;/p&gt;

&lt;p&gt;A handy way to read the loss: &lt;strong&gt;perplexity = exp(loss)&lt;/strong&gt;. My best 24-layer empathy checkpoint hit a validation loss of 3.9259, which is a perplexity of about 50.7. Loosely: the model is as uncertain as if it were choosing between ~51 equally likely tokens at each step.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 6: Generation is sampling in a loop
&lt;/h3&gt;

&lt;p&gt;To generate text, you feed in a prompt, get a distribution over the next token, &lt;strong&gt;sample&lt;/strong&gt; one, append it, and repeat. How you sample matters enormously:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;sample_next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;logits&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;generated_ids&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rep_penalty&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# penalize tokens we've already produced&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;generated_ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;logits&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;logits&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;rep_penalty&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;logits&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;logits&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;rep_penalty&lt;/span&gt;
    &lt;span class="n"&gt;logits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;logits&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;temperature&lt;/span&gt;                   &lt;span class="c1"&gt;# &amp;lt;1 = sharper, &amp;gt;1 = flatter&lt;/span&gt;
    &lt;span class="n"&gt;topk_vals&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;topk_idx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;topk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;logits&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# keep only the k best&lt;/span&gt;
    &lt;span class="n"&gt;probs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;softmax&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;topk_vals&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dim&lt;/span&gt;&lt;span class="o"&gt;=-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;topk_idx&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;multinomial&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;probs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Temperature&lt;/strong&gt; rescales confidence. Low values give safe, repetitive text; high values give creative and sometimes unhinged text.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Top-k&lt;/strong&gt; discards the long tail of unlikely tokens.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Repetition penalty&lt;/strong&gt; fights the classic small-model failure of looping on the same phrase.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  A note on the KV cache
&lt;/h3&gt;

&lt;p&gt;Notice something wasteful: when generating token 101, the keys and values for tokens 1-100 haven't changed. A &lt;strong&gt;KV cache&lt;/strong&gt; stores them so each new token costs one token's worth of compute instead of re-processing the whole prefix.&lt;/p&gt;

&lt;p&gt;Qwen (via Hugging Face) uses a KV cache. My from-scratch model does &lt;strong&gt;not&lt;/strong&gt;, and as you'll see in the benchmarks, that single omission matters a lot.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 2: Building the from-scratch model
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The architecture
&lt;/h3&gt;

&lt;p&gt;I deliberately followed GPT-2's design so every piece would be something I could fully explain. Two sizes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;253M model:&lt;/strong&gt; 16 layers, 16 heads, 1,024-dim embeddings&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;757M model:&lt;/strong&gt; 24 layers, 24 heads, 1,536-dim embeddings (about 680M non-embedding parameters)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Shared settings: 256-token context, 4x MLP expansion, GELU, pre-norm LayerNorm with bias, dropout 0.1, tied embeddings, learned positions, and gradient checkpointing, which trades extra compute for roughly 40% less activation memory.&lt;/p&gt;

&lt;p&gt;One initialization detail worth knowing, straight from the GPT-2 paper. The output projections inside each block (&lt;code&gt;c_proj&lt;/code&gt;) are initialized with a &lt;em&gt;smaller&lt;/em&gt; standard deviation, &lt;code&gt;0.02 / sqrt(2 * n_layer)&lt;/code&gt;. Every block adds to the residual stream, so without this scaling the stream's variance grows with depth and early training gets chaotic.&lt;/p&gt;

&lt;p&gt;You can sanity-check the parameter count with back-of-envelope math: each block has roughly &lt;code&gt;12 x d^2&lt;/code&gt; parameters (4d² for attention's Q/K/V/output, 8d² for the MLP). For d = 1,536 that's ~28.3M per block, x 24 blocks = ~680M. Add the 77M embedding table and you land right at ~757M.&lt;/p&gt;

&lt;h3&gt;
  
  
  Training run 1: the humbling CPU phase
&lt;/h3&gt;

&lt;p&gt;My first pretraining ran on &lt;strong&gt;CPU only&lt;/strong&gt;, float32, 8 threads, at about &lt;strong&gt;215 tokens per second&lt;/strong&gt; on a ~2.5M-token code-and-prose corpus. Two runs totaling roughly 2h44m got the 253M model to a best validation loss of &lt;strong&gt;4.2803&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That number is not good. It's the kind of model that produces fluent-looking nonsense. But it was a complete, working end-to-end loop, and everything after it built on that foundation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Training run 2: the GPU empathy probe
&lt;/h3&gt;

&lt;p&gt;Next I trained the 24-layer, 757M configuration on empathetic dialogue (EmpatheticDialogues + ESConv, about 2.1M train tokens) on an RTX 5060 with bfloat16 mixed precision. Effective batch: 32 sequences x 256 tokens = 8,192 tokens per step.&lt;/p&gt;

&lt;p&gt;Validation loss over time:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;step 200: 4.8194&lt;/li&gt;
&lt;li&gt;step 400: 4.4651&lt;/li&gt;
&lt;li&gt;step 600: 4.1942&lt;/li&gt;
&lt;li&gt;step 800: 4.0373&lt;/li&gt;
&lt;li&gt;step 1000: &lt;strong&gt;3.9259&lt;/strong&gt; (best, perplexity ~50.7)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Good progress. But when I actually &lt;em&gt;read the outputs&lt;/em&gt;, something was badly wrong.&lt;/p&gt;

&lt;h3&gt;
  
  
  The data bug that taught me the most
&lt;/h3&gt;

&lt;p&gt;The model's replies were non-sequiturs. It would answer "I feel like nobody understands me" with something fluent and completely unrelated. The loss curve looked healthy. The behavior was broken.&lt;/p&gt;

&lt;p&gt;Digging into the corpus, I found two structural defects:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;No speaker markers.&lt;/strong&gt; Turns were joined with a bare newline, so the model never learned &lt;em&gt;which side of the conversation it was writing&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Consecutive same-role turns.&lt;/strong&gt; There were 2,511 runs of back-to-back user (or back-to-back assistant) messages, which taught the model that one message can be followed by an unrelated one.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Lesson: a falling validation loss only tells you the model is getting better at the distribution you gave it. It says nothing about whether that distribution was the right one.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The fix was a new preprocessing script that rebuilds the corpus with an explicit format:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User: &amp;lt;text&amp;gt;
Assistant: &amp;lt;text&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;with consecutive same-role turns merged, Alpaca instruction/response pairs added in the same format (so the model can answer general questions, not just offer support), over-long answers dropped, and a 1M-token slice of code appended raw. The model tells the two modes apart by prompt shape: a bare code prefix continues as code, a &lt;code&gt;User:/Assistant:&lt;/code&gt; prefix continues as dialogue. Dialogue and Alpaca splits are disjoint, so validation is leak-free.&lt;/p&gt;

&lt;p&gt;The clean corpus came to ~5.93M tokens: 22,364 dialogues, 44,481 QA pairs, and the 1M code tokens.&lt;/p&gt;

&lt;p&gt;I also used &lt;code&gt;--init-from&lt;/code&gt; to load only the model weights and reset the optimizer. When your data distribution changes, resuming old optimizer state means carrying momentum tuned for a different dataset.&lt;/p&gt;

&lt;p&gt;At inference there's a matching trick. The model happily keeps writing after it finishes its reply, inventing the next &lt;code&gt;User:&lt;/code&gt; turn. So I cut the generated text at the first &lt;code&gt;User:&lt;/code&gt; or &lt;code&gt;Assistant:&lt;/code&gt; marker after the answer, and train with an EOS token at the end of each sequence so sampling can stop cleanly.&lt;/p&gt;

&lt;h3&gt;
  
  
  Distilling Qwen into the small model
&lt;/h3&gt;

&lt;p&gt;Even with clean structure, the small model had to invent its own style from very little data. So I tried &lt;strong&gt;distillation&lt;/strong&gt;: run &lt;code&gt;Qwen2.5-0.5B-Instruct&lt;/code&gt; over 20,000 Alpaca instructions (temperature 0.3), then mix those answers into the training corpus. Now the 253M model learns to imitate a stronger teacher's answers.&lt;/p&gt;

&lt;p&gt;Validation loss across these runs on the clean corpus fell from 4.0473 to 3.2289 (the CLI/API default checkpoint, at step 8,000) and to 2.9197 for the later v3 checkpoint.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;An important caveat:&lt;/strong&gt; those numbers are &lt;em&gt;not&lt;/em&gt; comparable with the 3.9259 from the empathy-only run. Different corpus, different validation set, different task mix. Validation loss is only comparable between models evaluated on the same data with the same tokenizer. I'll come back to this, because it nearly misled me.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 3: Why I fine-tuned Qwen
&lt;/h2&gt;

&lt;p&gt;At this point I had a working scratch pipeline and a decent understanding of the whole stack. So why bring in Qwen?&lt;/p&gt;

&lt;p&gt;Because the honest question was: &lt;strong&gt;what would it take to build the best possible empathetic assistant under my constraints?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here's the gap I was up against.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Scale of pretraining.&lt;/strong&gt; My scratch model saw millions of tokens. Qwen2.5 was pretrained on trillions. That's several orders of magnitude more data, and that's where grammar, world knowledge, code ability, and instruction-following come from. Fine-tuning on 23K dialogues can steer a model with all that capability. It can't conjure it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Architecture upgrades.&lt;/strong&gt; Qwen isn't just "bigger GPT-2". Its design (0.5B parameters, 24 layers, hidden size 896) includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;RoPE&lt;/strong&gt; positions instead of a learned table&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;RMSNorm&lt;/strong&gt; instead of LayerNorm&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grouped-query attention&lt;/strong&gt;: 14 query heads but only 2 key/value heads, which shrinks the KV cache dramatically&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;gated MLP&lt;/strong&gt; (896 → 4,864 → 896)&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;151,936-token vocabulary&lt;/strong&gt; that handles many languages and code efficiently&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Fit for the size bracket.&lt;/strong&gt; At 494M parameters, Qwen2.5-0.5B-Instruct sits between my 253M and 757M scratch models. Same order of magnitude, so the comparison is meaningful, even if it isn't size-parity.&lt;/p&gt;

&lt;p&gt;And importantly, I framed this correctly in the repo: it is &lt;strong&gt;not&lt;/strong&gt; a claim that one model is a distilled or compressed version of the other. It's a side-by-side of two real training pipelines on the same task.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fine-tuning without a data center: QLoRA
&lt;/h3&gt;

&lt;p&gt;Full fine-tuning updates every weight, which needs optimizer state (several times the model size) in GPU memory. QLoRA makes it fit on a consumer card with two ideas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. LoRA (Low-Rank Adaptation).&lt;/strong&gt; Freeze the original weight matrix &lt;code&gt;W&lt;/code&gt; and learn a small update expressed as the product of two skinny matrices:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;W' = W + (alpha / r) * B @ A
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;where &lt;code&gt;A&lt;/code&gt; is &lt;code&gt;r x d_in&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt; is &lt;code&gt;d_out x r&lt;/code&gt;, with a tiny rank &lt;code&gt;r&lt;/code&gt;. Instead of training a 896 x 896 matrix (802,816 values), a rank-8 adapter trains &lt;code&gt;8 x 896 + 896 x 8&lt;/code&gt; = 14,336 values.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Quantization.&lt;/strong&gt; Store the frozen base weights in 4-bit NF4 format with double quantization, and compute in bf16. The base model shrinks to a fraction of its memory footprint, while the small adapter trains in higher precision.&lt;/p&gt;

&lt;p&gt;Let me show the arithmetic on my adapter, because it's a satisfying check. Qwen's &lt;code&gt;q_proj&lt;/code&gt; maps 896 → 896 and &lt;code&gt;v_proj&lt;/code&gt; maps 896 → 128 (2 KV heads x 64 dims). At rank 8, per layer:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;q_proj: 8 x (896 + 896) = 14,336&lt;/li&gt;
&lt;li&gt;v_proj: 8 x (896 + 128) = 8,192&lt;/li&gt;
&lt;li&gt;per layer: 22,528, and x 24 layers = &lt;strong&gt;540,672 trainable parameters&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That matches the 540,672 scalars in my saved adapter exactly, about &lt;strong&gt;0.11% of the 494M base model&lt;/strong&gt;, and the whole adapter file is ~2.1 MB.&lt;/p&gt;

&lt;h3&gt;
  
  
  Stage 1: empathy SFT
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Data:&lt;/strong&gt; ~23,427 training dialogues / 461 validation dialogues (EmpatheticDialogues ~23K exchanges + ESConv ~910 multi-turn support dialogs), hash-split by dialogue to prevent leakage&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Setup:&lt;/strong&gt; 4-bit NF4 base, bf16 compute, AdamW at 1e-4 with warmup and cosine decay, batch 1 with 16 gradient-accumulation steps, 1,024-token sequences, 2 epochs&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost:&lt;/strong&gt; 2,928 optimizer steps in about 3.6 hours&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Result:&lt;/strong&gt; final train loss 2.1938; validation loss 2.7739 after epoch 1 and 2.7650 after epoch 2&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Stage 2: a small steering pass
&lt;/h3&gt;

&lt;p&gt;A second, much smaller pass used 770 rows of hand-written steering dialogues mixed with replay data from stage 1: 96 steps, 175 seconds, learning rate 5e-5.&lt;/p&gt;

&lt;p&gt;Here's where I have to be honest, because the repo is.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stage 2 did not measurably improve validation loss.&lt;/strong&gt; Stage 1 ended at 2.7650; stage 2 at 2.7655. And the stage-2 validation set was the &lt;em&gt;same&lt;/em&gt; 461-row file used in stage 1, so it isn't an untouched hold-out anyway. Validation loss was never going to capture what steering is for (policy and behavior), but I also have no automated behavioral gate in the repo that proves it worked. My stored behavioral probe records predate the latest retrain and no script reproduces them. I've listed that as a known gap rather than pretend otherwise.&lt;/p&gt;

&lt;p&gt;I also hit a subtle config surprise: the stage-2 run was launched with &lt;code&gt;--init-adapter&lt;/code&gt;, and my training script ignores &lt;code&gt;--lora-r&lt;/code&gt; and friends in that path, loading the stage-1 config instead. So the trained stage-2 adapter is rank-8 over &lt;code&gt;q_proj&lt;/code&gt;/&lt;code&gt;v_proj&lt;/code&gt;, even though its log records the CLI values (r=16, alpha=32). Always verify what your adapter config &lt;em&gt;actually&lt;/em&gt; is, not what you typed on the command line.&lt;/p&gt;

&lt;h3&gt;
  
  
  Merging
&lt;/h3&gt;

&lt;p&gt;For deployment I merge the adapter back into the base weights and save a single fp16 checkpoint (988,097,536 bytes). The merged model loads as a plain &lt;code&gt;AutoModelForCausalLM&lt;/code&gt;: no PEFT wrapper, no adapter step at load time, and zero extra inference cost, since the architecture is identical to the base.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;"slm-ft/merged_fp16/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;torch_dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;device_map&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"auto"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;tokenizer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"slm-ft/merged_fp16/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Part 4: Head-to-head results
&lt;/h2&gt;

&lt;p&gt;All benchmarks: CPU, float32, 8 threads, identical prompts, &lt;code&gt;max_new_tokens=40&lt;/code&gt;, temperature 0.7, top-k 20, top-p 0.9, repetition penalty 1.2, seed 1337. Reproducible from &lt;code&gt;benchmarks/generation_matrix.json&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Speed and memory
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Generation throughput:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;SLM 16L (253M): &lt;strong&gt;12.54 tok/s&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Qwen2.5-0.5B-Instruct (base): 9.92 tok/s&lt;/li&gt;
&lt;li&gt;Qwen + QLoRA merged: 10.62 tok/s&lt;/li&gt;
&lt;li&gt;SLM 24L (757M): 4.69 tok/s&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Memory after load (RSS):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;SLM 16L: 2,907 MB&lt;/li&gt;
&lt;li&gt;Qwen base: 3,631 MB&lt;/li&gt;
&lt;li&gt;Qwen merged: 3,923 MB&lt;/li&gt;
&lt;li&gt;SLM 24L: 6,078 MB&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Reading these correctly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The &lt;strong&gt;16L model is genuinely faster&lt;/strong&gt; than Qwen: fewer parameters, so less memory to stream per token. It's 1.26x faster at generation and 2.5x faster per forward pass.&lt;/li&gt;
&lt;li&gt;The &lt;strong&gt;24L model is 2.1x slower than Qwen despite having only ~1.5x the parameters.&lt;/strong&gt; The reason is the missing KV cache. It re-encodes the entire context at every generated token, while Qwen reuses cached keys and values. This is the single most actionable finding for anyone building their own model: &lt;em&gt;implement the KV cache.&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Don't over-read the merged-vs-base gap.&lt;/strong&gt; The merged model measured 7% faster than base in the final run, but across earlier runs of the same script the direction reversed (-11% and -6%). Same architecture, same FLOPs, so per-token cost should be identical. I treat that gap as unmeasured noise, not a result. CPU timings on my machine vary by roughly 10% run to run, so the &lt;em&gt;ratios&lt;/em&gt; are the finding, not the absolutes.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Quality
&lt;/h3&gt;

&lt;p&gt;Speed is where my scratch models can compete. Quality is where they can't.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;SLM 24L:&lt;/strong&gt; On the empathetic prompt it stays on topic in a stilted, all-lowercase register ("i am sorry to hear that, but i hope you get through it soon..."). On code prompts it falls apart: &lt;code&gt;def preprocess(text):&lt;/code&gt; produced "You have an appointment?" and &lt;code&gt;class ChatMessage:&lt;/code&gt; produced nothing at all. That's the predictable result of training only on empathy data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;SLM 16L:&lt;/strong&gt; Short, on-topic empathetic replies ("That is good to hear! I hope everything goes well for you."). On code prompts it produces prose or fragments, not code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Qwen2.5-0.5B-Instruct:&lt;/strong&gt; Coherent and on-topic on every prompt, including syntactically valid code completions for all four code prompts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Qwen merged fine-tune:&lt;/strong&gt; Same behavior as the base on all five benchmark prompts. The good news: the narrow empathy SFT did &lt;strong&gt;not&lt;/strong&gt; wreck its code ability, which is the main regression risk of fine-tuning on a small, specialized corpus. Whether it &lt;em&gt;improves&lt;/em&gt; empathy still needs the behavioral gate I haven't built yet.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Bottom line:&lt;/strong&gt; neither from-scratch model matches Qwen's output quality on these prompts. The project's quality-oriented path is the QLoRA fine-tune. The scratch models are the educational path, and the 16L one is a perfectly reasonable fast model for short supportive replies.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why this isn't a paradox
&lt;/h3&gt;

&lt;p&gt;A 757M model losing to a 494M one sounds backwards until you separate the three things that determine quality: &lt;strong&gt;parameters, data, and compute.&lt;/strong&gt; My 757M model had more parameters but trained on roughly a million times less data than Qwen saw, for a few thousand steps on a single consumer GPU. Parameters are capacity; data and compute are what &lt;em&gt;fill&lt;/em&gt; that capacity. A large empty model is just a large empty model.&lt;/p&gt;

&lt;p&gt;That's the real reason fine-tuning wins at small scale: you rent someone else's trillions of training tokens and spend your own tiny budget only on the last mile.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 5: Serving it and verifying it
&lt;/h2&gt;

&lt;p&gt;A model nobody can call is a notebook. So I built the serving layer properly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;FastAPI server&lt;/strong&gt; (&lt;code&gt;slm_api.py&lt;/code&gt;) with:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;/generate&lt;/code&gt; plus &lt;strong&gt;OpenAI-compatible&lt;/strong&gt; &lt;code&gt;/v1/completions&lt;/code&gt; and &lt;code&gt;/v1/chat/completions&lt;/code&gt;, with SSE streaming variants, so existing OpenAI clients can point at it&lt;/li&gt;
&lt;li&gt;Bearer-token auth, per-IP rate limiting (HTTP 429), request timeouts (HTTP 504), and restricted CORS&lt;/li&gt;
&lt;li&gt;Input and output &lt;strong&gt;safety guardrails&lt;/strong&gt; that reject harmful prompts and scan generated text. For a model aimed at emotional-support conversations I wanted that layer to exist from day one&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/health&lt;/code&gt;, &lt;code&gt;/model/info&lt;/code&gt;, and a Prometheus &lt;code&gt;/metrics&lt;/code&gt; endpoint tracking request counts, latency histograms, tokens generated, and model-loaded state&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Deployment:&lt;/strong&gt; a slim Docker image (CPU-only PyTorch wheels, because the default Linux wheel bundles several GB of CUDA runtime I wasn't going to use), non-root user, health checks, and a production &lt;code&gt;docker-compose&lt;/code&gt; stack with Prometheus and resource limits.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Verification harness:&lt;/strong&gt; one command, &lt;code&gt;python harness.py&lt;/code&gt;, runs the test suite, a setup-artifact check, the live comparison, and a dependency audit, then exits non-zero if any step fails. There's also seeded generation (&lt;code&gt;seed=1337&lt;/code&gt;), pinned dependency constraints, pre-commit hooks, and a held-out perplexity evaluation script.&lt;/p&gt;

&lt;p&gt;I added the harness after a mistake, which brings me to the part I think matters most.&lt;/p&gt;




&lt;h2&gt;
  
  
  Part 6: What I got wrong (and fixed)
&lt;/h2&gt;

&lt;p&gt;I kept a "provenance note" every time a number in my own docs turned out to be wrong. There were several:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A wrong validation loss.&lt;/strong&gt; An earlier table said "step 500, val loss 3.9733" for the 24-layer model. That number belonged to a &lt;em&gt;different&lt;/em&gt; run (a 458M, 20-layer model). The correct shipped result is step 1000, 3.9259. I only caught it by tracing each figure back to its log file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A double-counted memory figure.&lt;/strong&gt; My benchmark reported 5,616 MB for the merged Qwen model. The harness had left the &lt;em&gt;base&lt;/em&gt; model resident in the same process, so both rows carried the base's ~2 GB. After freeing it, the real number is 3,923 MB.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A benchmark "result" that was noise.&lt;/strong&gt; Merged vs. base speed flipped direction across runs. If I'd published only one run, I'd have "found" a speedup that doesn't exist.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An adapter config that didn't match its log&lt;/strong&gt;, as described above.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A fine-tune that's "verified" only by loss.&lt;/strong&gt; The behavioral gate is still missing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these were catastrophic, because each got caught by the same habit: &lt;strong&gt;every number in the docs must trace to an artifact in the repo, and every claim must say what it does and doesn't prove.&lt;/strong&gt; If you take one engineering lesson from this post, take that one. It's more valuable than any architecture trick.&lt;/p&gt;




&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;LLMs are simpler than they look.&lt;/strong&gt; Embeddings, attention, MLPs, residuals, and a softmax, applied next-token-at-a-time. Building the whole thing once makes every paper easier to read.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Data quality beats everything.&lt;/strong&gt; My model looked healthy on the loss curve while giving non-sequiturs, and the cause was missing speaker markers and merged turns in the corpus.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validation loss is only comparable on the same data and tokenizer.&lt;/strong&gt; Don't compare numbers across corpora.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Implement the KV cache.&lt;/strong&gt; It was the difference between "slightly slower than Qwen" and "2x slower."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;At small scale, fine-tune instead of pretraining.&lt;/strong&gt; Parameters are capacity; pretraining data is what fills it. QLoRA let me adapt a capable model with 0.11% trainable parameters in an afternoon.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure honestly.&lt;/strong&gt; Run benchmarks more than once, label what's noise, free your memory between measurements, and write down what you &lt;em&gt;didn't&lt;/em&gt; verify.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  What's next
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;An automated &lt;strong&gt;behavioral evaluation&lt;/strong&gt; for the merged model (the biggest gap)&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;KV cache&lt;/strong&gt; for the from-scratch model&lt;/li&gt;
&lt;li&gt;Standard zero-shot benchmarks (HellaSwag, ARC, PIQA, and others) for a fairer capability comparison than my own prompt set&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Try it yourself
&lt;/h2&gt;

&lt;p&gt;Everything (the model code, training scripts, QLoRA pipeline, benchmarks, API server, Docker setup, and harness) is open source under the MIT license:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/bharqav/SLM" rel="noopener noreferrer"&gt;github.com/bharqav/SLM&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;License note:&lt;/strong&gt; the training data includes EmpatheticDialogues (CC BY-NC-4.0) and ESConv (CC BY-NC-SA), so models trained on it carry non-commercial constraints. Check the repo's License section before using any weights commercially.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If you build your own, I'd love to hear what broke first. In my experience it's never the architecture; it's the data.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Questions or corrections? Drop a comment, or open an issue on the repo. If a number in this post doesn't match the artifacts, the artifacts win.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>llm</category>
    </item>
    <item>
      <title>I Built a Latent Diffusion Model That Fits in 5.5MB and Runs Without a GPU</title>
      <dc:creator>Bhargav </dc:creator>
      <pubDate>Tue, 10 Nov 2026 09:39:01 +0000</pubDate>
      <link>https://dev.to/bharqav/i-built-a-latent-diffusion-model-that-fits-in-55mb-and-runs-without-a-gpu-25eb</link>
      <guid>https://dev.to/bharqav/i-built-a-latent-diffusion-model-that-fits-in-55mb-and-runs-without-a-gpu-25eb</guid>
      <description>&lt;p&gt;Stable Diffusion needs an A100 to feel fast. It ships with 860 million parameters, a UNet backbone, and a multi-gigabyte footprint that assumes you have real GPU infrastructure behind it.&lt;/p&gt;

&lt;p&gt;I wanted to know how small a diffusion model could get before it stopped working at all. The answer, after a lot of ablation studies I'll show you below, is &lt;strong&gt;TinyDiffusion 128&lt;/strong&gt;: a 1.45 million parameter Diffusion Transformer that fits in 5.55MB of FP32 weights, runs in under 200ms on a plain CPU in the project's fastest benchmark configuration (more on how that number relates to the others below), and needs less than 25MB of RAM to do it.&lt;/p&gt;

&lt;p&gt;This isn't a toy that only draws noise. It generates coherent 128x128 images from text prompts, and every TinyDiffusion number in this post is pulled directly from the project's own evaluation suite and ablation reports, not estimated or rounded up for effect. The figures for other models are reference values from published or documented sources, and they're labelled that way where they appear.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is TinyDiffusion 128?
&lt;/h2&gt;

&lt;p&gt;A quality-first, CPU-native, micro-scale latent diffusion system. It replaces the heavy UNet architecture most diffusion models use with a Diffusion Transformer (DiT) operating on an 8x compressed latent space, which is most of where the size reduction comes from.&lt;/p&gt;

&lt;p&gt;The core numbers, as documented in the repository:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Attribute&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Architecture&lt;/td&gt;
&lt;td&gt;Diffusion Transformer (DiT)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Parameters&lt;/td&gt;
&lt;td&gt;1.45 million&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;FP32 weight size&lt;/td&gt;
&lt;td&gt;5.55 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Peak RAM&lt;/td&gt;
&lt;td&gt;&amp;lt; 25 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CPU inference&lt;/td&gt;
&lt;td&gt;sub-200 ms for a full 12-step generation (fastest benchmark configuration)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Nothing here needs a GPU. It runs on Windows, macOS, and Linux with pure CPU inference, and there's a zero-dependency C++17 runtime for embedded or edge deployment on top of the Python package.&lt;/p&gt;

&lt;h2&gt;
  
  
  How It Actually Works
&lt;/h2&gt;

&lt;p&gt;The pipeline has three stages: turn the text prompt into a conditioning signal, run that signal through a DiT backbone across a handful of denoising steps, then decode the result into a real image.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Conditioning.&lt;/strong&gt; A prompt like "a ruby red circle on a white background" goes through a &lt;code&gt;TinyTokenizer&lt;/code&gt; and &lt;code&gt;TinyTextEncoder&lt;/code&gt;, producing a single context embedding (a 1x64 vector) that steers everything that follows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Latent diffusion sampling.&lt;/strong&gt; Gaussian noise starts in a compressed 1x4x16x16 latent space. The TinyDiT backbone predicts the noise to remove at each step, guided by both the timestep and the text embedding, and a DDIM scheduler runs this for 10 to 15 steps rather than the hundreds a naive diffusion process would need. Classifier-free guidance (CFG) blends a conditioned and unconditioned prediction at a scale of 3.5 by default, which the project's own ablation study found to be the best trade-off between boundary sharpness and saturation, without introducing clipping artifacts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Reconstruction.&lt;/strong&gt; Once the latent is denoised, a &lt;code&gt;TinyVAE&lt;/code&gt; decoder upscales it 8x back into a full 128x128 RGB image.&lt;/p&gt;

&lt;p&gt;The model can also be exported to a single-file binary format (&lt;code&gt;.tdiff&lt;/code&gt;) and run through the standalone C++ runtime, which uses raw pointer arithmetic and arena allocators with no external dependencies.&lt;/p&gt;

&lt;h2&gt;
  
  
  How It Was Evaluated
&lt;/h2&gt;

&lt;p&gt;A model this small is easy to dismiss as not really working. So before anything else, it's worth looking at how the project measured whether it actually does, rather than just trusting the claim.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Target&lt;/th&gt;
&lt;th&gt;What it checks&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fréchet Distance (FID)&lt;/td&gt;
&lt;td&gt;301.64&lt;/td&gt;
&lt;td&gt;&amp;lt; 350&lt;/td&gt;
&lt;td&gt;Similarity to 792 reference samples&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inception Score (IS)&lt;/td&gt;
&lt;td&gt;2.17 ± 0.06&lt;/td&gt;
&lt;td&gt;&amp;gt; 2.0&lt;/td&gt;
&lt;td&gt;Class salience, batch variety&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mean brightness&lt;/td&gt;
&lt;td&gt;166.9 / 255&lt;/td&gt;
&lt;td&gt;&amp;gt; 140&lt;/td&gt;
&lt;td&gt;No dark-image collapse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spatial variance&lt;/td&gt;
&lt;td&gt;36.2&lt;/td&gt;
&lt;td&gt;&amp;gt; 25&lt;/td&gt;
&lt;td&gt;Edge contrast, not flat output&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Shannon entropy&lt;/td&gt;
&lt;td&gt;6.65 bits&lt;/td&gt;
&lt;td&gt;&amp;gt; 4.0&lt;/td&gt;
&lt;td&gt;Information density, pixel balance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dark image bias (&amp;lt; 80 RGB)&lt;/td&gt;
&lt;td&gt;0.1% (1/792)&lt;/td&gt;
&lt;td&gt;&amp;lt; 5.0%&lt;/td&gt;
&lt;td&gt;Color variety across the gallery&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These are the project's own self-reported targets and results, evaluated against 792 procedurally generated reference samples, not an independent third-party benchmark. Worth keeping in mind as you read the rest of this: a small FID-style check against your own reference set tells you the model is internally consistent and not collapsing to noise, but it isn't the same claim as matching a large, diverse, human-curated dataset.&lt;/p&gt;

&lt;p&gt;A note on the FID number itself: 301.64 will look high next to scores in published papers, and it shouldn't be compared with them. FID depends on the feature extractor and on the reference set, and it is biased upward when the sample count is small (here, 792 images). A score computed on 128x128 procedural images against a small custom reference set is not on the same scale as one computed on a large benchmark such as ImageNet or COCO. That's why the target is a loose threshold (&amp;lt; 350): it works as a sanity check that the model hasn't collapsed, not as a quality ranking.&lt;/p&gt;

&lt;h2&gt;
  
  
  The CFG Ablation: Why 3.5 Is the Default
&lt;/h2&gt;

&lt;p&gt;Classifier-free guidance scale controls the tradeoff between literal prompt-following and how saturated and sharp the output looks. The project ran a sweep across five values:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;CFG scale&lt;/th&gt;
&lt;th&gt;Spatial variance&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1.0&lt;/td&gt;
&lt;td&gt;41.03&lt;/td&gt;
&lt;td&gt;Low contrast, under-conditioned&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2.0&lt;/td&gt;
&lt;td&gt;42.98&lt;/td&gt;
&lt;td&gt;Balanced, mild boundary softness&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3.5&lt;/td&gt;
&lt;td&gt;48.05&lt;/td&gt;
&lt;td&gt;Strong boundary sharpness, no clipping&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7.0&lt;/td&gt;
&lt;td&gt;48.70&lt;/td&gt;
&lt;td&gt;Highest variance, but minor saturation artifacts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10.0&lt;/td&gt;
&lt;td&gt;43.10&lt;/td&gt;
&lt;td&gt;Over-guided, edge clipping appears&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The curve isn't monotonic. Pushing guidance higher doesn't keep helping past a point, it starts clipping and losing fidelity. Spatial variance is technically highest at 7.0 (48.70), but 3.5 reaches within about 1.3% of that (48.05) without the saturation artifacts that appear at 7.0 and the edge clipping at 10.0. That trade-off, rather than the single highest number, is why 3.5 is the default instead of the maximum the sweep tested.&lt;/p&gt;

&lt;h2&gt;
  
  
  TinyDiT vs TinyUNet: Why Transformer Won
&lt;/h2&gt;

&lt;p&gt;The project also benchmarked its own DiT backbone against a ConvNet-based UNet alternative, on the same hardware:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;TinyDiT&lt;/th&gt;
&lt;th&gt;TinyUNet&lt;/th&gt;
&lt;th&gt;Advantage&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Parameters&lt;/td&gt;
&lt;td&gt;1.45M&lt;/td&gt;
&lt;td&gt;1.54M&lt;/td&gt;
&lt;td&gt;DiT, 5.8% smaller&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;FP32 size&lt;/td&gt;
&lt;td&gt;5.55 MB&lt;/td&gt;
&lt;td&gt;5.89 MB&lt;/td&gt;
&lt;td&gt;DiT&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Single-step latency&lt;/td&gt;
&lt;td&gt;31.81 ms&lt;/td&gt;
&lt;td&gt;57.99 ms&lt;/td&gt;
&lt;td&gt;DiT, 1.82x faster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sampling throughput&lt;/td&gt;
&lt;td&gt;31.4 it/s&lt;/td&gt;
&lt;td&gt;17.2 it/s&lt;/td&gt;
&lt;td&gt;DiT, +82.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12-step CFG total&lt;/td&gt;
&lt;td&gt;763.5 ms&lt;/td&gt;
&lt;td&gt;1391.8 ms&lt;/td&gt;
&lt;td&gt;DiT&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Despite being the newer, less battle-tested architecture choice for a project this size, the transformer backbone won on every axis measured here: fewer parameters, smaller file, and roughly 1.8x faster per step.&lt;/p&gt;

&lt;h2&gt;
  
  
  How It Compares to Larger Diffusion Models
&lt;/h2&gt;

&lt;p&gt;This is the table most people will want to see, so it deserves the most caution. These are the project's documented comparison figures. The TinyDiffusion row is this project's own measurement; the other rows are reference values:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Parameters&lt;/th&gt;
&lt;th&gt;FP32 size&lt;/th&gt;
&lt;th&gt;CPU latency/step&lt;/th&gt;
&lt;th&gt;Total (steps)&lt;/th&gt;
&lt;th&gt;RAM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;TinyDiffusion 128 (ours)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1.45 M&lt;/td&gt;
&lt;td&gt;5.55 MB&lt;/td&gt;
&lt;td&gt;13.13 ms&lt;/td&gt;
&lt;td&gt;~178 ms (12 steps)&lt;/td&gt;
&lt;td&gt;&amp;lt; 25 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SnapFusion&lt;/td&gt;
&lt;td&gt;280 M&lt;/td&gt;
&lt;td&gt;1,120 MB&lt;/td&gt;
&lt;td&gt;~350 ms&lt;/td&gt;
&lt;td&gt;~1,800 ms (8 steps)&lt;/td&gt;
&lt;td&gt;~1.5 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MobileDiffusion (Google)&lt;/td&gt;
&lt;td&gt;520 M&lt;/td&gt;
&lt;td&gt;2,080 MB&lt;/td&gt;
&lt;td&gt;~280 ms&lt;/td&gt;
&lt;td&gt;~1,200 ms (6 steps)&lt;/td&gt;
&lt;td&gt;~2.2 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mini-SD&lt;/td&gt;
&lt;td&gt;390 M&lt;/td&gt;
&lt;td&gt;1,560 MB&lt;/td&gt;
&lt;td&gt;~680 ms&lt;/td&gt;
&lt;td&gt;~5,400 ms (15 steps)&lt;/td&gt;
&lt;td&gt;~2.8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stable Diffusion v1.5&lt;/td&gt;
&lt;td&gt;860 M&lt;/td&gt;
&lt;td&gt;3,440 MB&lt;/td&gt;
&lt;td&gt;~2,800 ms&lt;/td&gt;
&lt;td&gt;~28,000 ms (20 steps)&lt;/td&gt;
&lt;td&gt;~4.5 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A few things worth being precise about here, since it's easy for a table like this to imply more than it shows:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The rows for SnapFusion, MobileDiffusion, Mini-SD, and Stable Diffusion v1.5 are reference figures, not benchmarks re-run on the same machine as TinyDiffusion. Only the TinyDiffusion row reflects this project's own measurement.&lt;/li&gt;
&lt;li&gt;The step counts differ across models (12 steps for TinyDiffusion vs 20 for SD v1.5), so the total-time comparison isn't a perfectly controlled apples-to-apples test. The per-step latency column (13.13ms vs ~2,800ms) is the fairer number to anchor on, and it still puts TinyDiffusion roughly two orders of magnitude faster per step on CPU.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this changes the underlying result, which is real: a 1.45M parameter model fits in CPU cache in a way an 860M parameter model simply cannot. But "213x faster per step" is a more honest framing than "150x faster overall," since the latter quietly compares different step budgets.&lt;/p&gt;

&lt;h3&gt;
  
  
  A note on the two sets of timing numbers
&lt;/h3&gt;

&lt;p&gt;The latency figures in this post come from two different benchmark setups, and they don't match each other:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;th&gt;Per-step latency&lt;/th&gt;
&lt;th&gt;12-step total&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Comparison table above&lt;/td&gt;
&lt;td&gt;13.13 ms&lt;/td&gt;
&lt;td&gt;~178 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TinyDiT vs TinyUNet benchmark&lt;/td&gt;
&lt;td&gt;31.81 ms&lt;/td&gt;
&lt;td&gt;763.5 ms (with CFG)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The second row is consistent with classifier-free guidance doubling the work: 763.5 ms over 12 steps is about 63.6 ms per step, roughly twice the 31.81 ms single-step figure, because each step runs a conditioned and an unconditioned prediction. The first row is the faster configuration, and it is where the "under 200 ms" headline comes from. I haven't reconciled the two on identical hardware and settings, so treat the headline as the project's best measured case rather than a guarantee for every configuration.&lt;/p&gt;

&lt;p&gt;The per-step comparison with Stable Diffusion holds up either way: using the slower 31.81 ms figure instead of 13.13 ms still gives roughly 88x faster per step (about 2,800 ms vs 31.81 ms), instead of 213x.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try It
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/bharqav/tiny-diffusion.git
&lt;span class="nb"&gt;cd&lt;/span&gt; tiny-diffusion
python -m venv venv
&lt;span class="nb"&gt;source&lt;/span&gt; venv/bin/activate    &lt;span class="c"&gt;# or venv\Scripts\activate on Windows&lt;/span&gt;
pip install -e &lt;span class="s2"&gt;".[dev]"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fastest way to see it work is the interactive prompt studio:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;tiny-diffusion interactive
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Enter Prompt: a glowing gold star on a deep space background
TinyDiffusion Sampling: 100%|##########| 10/10 [00:00&amp;lt;00:00, 26.2it/s]
Success! Image saved to: output/result_1.png
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or generate directly from the CLI with explicit sampling parameters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;tiny-diffusion generate \
  --prompt &lt;span class="s2"&gt;"a ruby red circle on a white background"&lt;/span&gt; \
  --steps 12 \
  --guidance 3.5 \
  --seed 42 \
  --output output/ruby_circle.png
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or from Python:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;tiny_diffusion&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;TinyDiffusionPipeline&lt;/span&gt;

&lt;span class="n"&gt;pipe&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;TinyDiffusionPipeline&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;checkpoint_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"checkpoints"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"dit"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;image&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;pipe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"a vibrant magenta hexagon on a golden glow background"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;num_inference_steps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;guidance_scale&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;3.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;513&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;image&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"output/magenta_hexagon.png"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For the standalone C++ runtime:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cmake -B build
cmake --build build --config Release
./build/test_runtime
./build/tiny-diffusion-cpp
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Repository Structure
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tiny-diffusion/
├── tiny_diffusion/           # Canonical Python package
│   ├── cli.py                # Unified CLI entrypoint
│   ├── models/               # DiT, UNet, VAE, text encoder
│   ├── sampling/             # DDIM scheduler, sampling pipeline
│   ├── data/                 # Tokenizer and dataset abstractions
│   ├── data_factory/         # Procedural synthetic data generation
│   ├── training/             # VAE, DiT, and distributed training loops
│   ├── evaluation/           # FID, IS, Shannon entropy metrics
│   └── compression/          # INT8 quantization, .tdiff export
├── runtime/                  # Pure C++17 zero-dependency inference engine
│   ├── include/              # arena.h, tensor.h, model.h, kernels.h
│   ├── src/                  # C++ implementation
│   └── tests/                # C++ unit tests
├── output/                   # 1,000-image gallery, manifests, benchmark reports
├── benchmarks/               # Ablation and comparative benchmark runners
├── scripts/                  # Gallery generator, ops profiler, training pipeline
└── tests/                    # 27 passing pytest unit tests
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Lessons Learned
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Parameter count isn't the only thing that matters, shape does too.&lt;/strong&gt; TinyDiT beat TinyUNet with fewer parameters and a smaller file, but the real win was architecture: the transformer's structure mapped better to CPU cache behavior than the convolutional alternative did, which is why the latency gap (1.82x) is bigger than the parameter gap (5.8%).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Guidance scale has a peak, not a ceiling.&lt;/strong&gt; It would be easy to assume more CFG guidance is strictly better. The ablation shows clearly that it isn't: quality rises, peaks around 3.5 to 7.0, then degrades into clipping artifacts past that.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Self-evaluation needs an honest frame.&lt;/strong&gt; An FID score against your own 792-sample reference set is a real, useful signal that the model isn't collapsing or stuck on noise. But it's a different claim from matching a benchmark built on a large external dataset, and it's worth saying so directly rather than letting the number imply more than it supports.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Compare the thing that's actually controlled.&lt;/strong&gt; The per-step latency comparison against larger models is a clean, apples-to-apples number. The total-time comparison isn't, because the step counts differ. Both are useful, but only one of them is a fair test, and it matters which one you lead with.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Real Takeaway
&lt;/h2&gt;

&lt;p&gt;A 5.55MB model that runs in under 200ms on a CPU in its fastest configuration isn't trying to replace Stable Diffusion. It's answering a narrower question: how much can you strip away from a diffusion model's architecture, parameter count, and latent space before image generation stops working, and how do you prove it's still working once you have?&lt;/p&gt;

&lt;p&gt;The ablations and evaluation suite are the actual contribution here, as much as the model itself. Anyone can claim a small model works. Showing the CFG sweep, the architecture comparison, and the FID numbers against a stated reference set is what makes that claim checkable.&lt;/p&gt;

&lt;p&gt;The code, the 1,000-image gallery, and the full benchmark reports are on GitHub: &lt;a href="https://github.com/bharqav/tiny-diffusion" rel="noopener noreferrer"&gt;github.com/bharqav/tiny-diffusion&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>cpp</category>
    </item>
    <item>
      <title>What Does It Take to Run a 30B Model Without a GPU?</title>
      <dc:creator>Bhargav </dc:creator>
      <pubDate>Tue, 10 Nov 2026 09:33:07 +0000</pubDate>
      <link>https://dev.to/bharqav/what-does-it-take-to-run-a-30b-model-without-a-gpu-5bo0</link>
      <guid>https://dev.to/bharqav/what-does-it-take-to-run-a-30b-model-without-a-gpu-5bo0</guid>
      <description>&lt;p&gt;Most people interact with large language models through frameworks that hide much of the inference stack. Those frameworks are powerful, but they can make it difficult to see what happens between loading a model and generating the next token.&lt;/p&gt;

&lt;p&gt;So I built &lt;strong&gt;Quantr&lt;/strong&gt;, an experimental LLM inference engine written primarily in C.&lt;/p&gt;

&lt;p&gt;The project explores quantized model weights, GGUF loading, CPU execution, low-level kernels, context management, and a runtime that can be inspected without making PyTorch or BLAS mandatory dependencies. Its CPU path does not require a GPU.&lt;/p&gt;

&lt;p&gt;This is not a claim that a small binary makes every large model fit into a small amount of RAM. The interesting engineering challenge is understanding which parts of inference can be made lighter, which trade-offs are involved, and how to verify that the implementation is doing what it claims.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Repository:&lt;/strong&gt; &lt;a href="https://github.com/bharqav/quantr-in-c" rel="noopener noreferrer"&gt;github.com/bharqav/quantr-in-c&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What Is Quantr?
&lt;/h2&gt;

&lt;p&gt;Quantr is a C-based inference runtime built to explore quantized transformer models in the GGUF format.&lt;/p&gt;

&lt;p&gt;The repository brings together several pieces of an inference stack:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;GGUF metadata and tensor loading.&lt;/li&gt;
&lt;li&gt;Memory-mapped model-file access where supported.&lt;/li&gt;
&lt;li&gt;Quantized tensor representations, including Q4_K, Q5_K, Q6_K, and Q8_0 paths.&lt;/li&gt;
&lt;li&gt;Scalar CPU computation and available SIMD kernels.&lt;/li&gt;
&lt;li&gt;Tokenization, model execution, sampling, and context-management code.&lt;/li&gt;
&lt;li&gt;Experimental speculative-decoding and backend-verification utilities.&lt;/li&gt;
&lt;li&gt;Tests for selected numerical operations and runtime components.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The core idea is simple: make the inference path explicit. Instead of treating the runtime as a black box, I wanted to understand the model format, memory layout, numerical operations, and CPU execution decisions directly.&lt;/p&gt;

&lt;p&gt;Quantr runs as a normal user-space program under an operating system. I sometimes describe the project as low-level or bare-metal-inspired, but it is not an operating-system-free bare-metal runtime.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Build an Inference Engine from Scratch?
&lt;/h2&gt;

&lt;p&gt;An inference engine is not just a matrix multiplication loop. It has to coordinate several layers of work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Model representation:&lt;/strong&gt; parse metadata and understand tensor layouts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory management:&lt;/strong&gt; load weights and manage intermediate activations and attention state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Numerical computation:&lt;/strong&gt; implement operations such as normalization, attention, and feed-forward layers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Quantization:&lt;/strong&gt; reduce weight storage while accounting for numerical error and dequantization requirements.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hardware execution:&lt;/strong&gt; use SIMD instructions where available while retaining a fallback path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runtime behavior:&lt;/strong&gt; handle tokenization, sampling, context limits, and generation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verification:&lt;/strong&gt; test individual operations and establish whether complete model outputs are correct.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Working in C makes these boundaries visible. It also removes some of the safety nets provided by higher-level frameworks. Tensor shapes, memory ownership, numerical precision, and architecture-specific behavior become your responsibility.&lt;/p&gt;

&lt;p&gt;That is what makes the project useful as a systems-learning exercise: every shortcut has a cost, and every performance claim needs evidence.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Weight and Memory Problem
&lt;/h2&gt;

&lt;p&gt;Model weights are often the largest component of inference memory, but they are not the only component.&lt;/p&gt;

&lt;p&gt;For a rough estimate, a model with 30 billion parameters stored at FP16 needs around 60 GB for its weights alone. At an idealized four bits per parameter, the raw weight payload would be around 15 GB before block metadata, tensor-specific formats, and runtime overhead.&lt;/p&gt;

&lt;p&gt;That distinction matters because several different optimizations are often discussed as if they all reduce the same quantity.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Technique&lt;/th&gt;
&lt;th&gt;What it can improve&lt;/th&gt;
&lt;th&gt;What it does not automatically solve&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Weight quantization&lt;/td&gt;
&lt;td&gt;Reduces the storage required for supported weight tensors&lt;/td&gt;
&lt;td&gt;KV-cache memory, intermediate buffers, or all numerical error&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mixture-of-Experts routing&lt;/td&gt;
&lt;td&gt;Reduces the number of experts evaluated for a token when supported by the model and runtime&lt;/td&gt;
&lt;td&gt;The need to represent and access the model's expert weights&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Memory mapping&lt;/td&gt;
&lt;td&gt;Avoids an eager copy of the entire file into a separate user-space buffer and lets the OS page data as needed&lt;/td&gt;
&lt;td&gt;Physical memory use, page faults, or memory-bandwidth limits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Smaller model architecture&lt;/td&gt;
&lt;td&gt;Reduces parameter storage and usually compute requirements&lt;/td&gt;
&lt;td&gt;It is not the same model running more efficiently&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;KV-cache optimization&lt;/td&gt;
&lt;td&gt;Can reduce memory used to retain attention keys and values&lt;/td&gt;
&lt;td&gt;Weight storage or other runtime allocations&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The important lesson is that &lt;strong&gt;file size, virtual address space, resident memory, and total system memory are different measurements.&lt;/strong&gt; A model file being memory-mapped does not mean the model has no physical-memory cost.&lt;/p&gt;

&lt;p&gt;Any claim that a large model runs within a particular RAM budget should be backed by a reproducible test that identifies the exact model file, quantization, context length, hardware, operating system, and memory metric.&lt;/p&gt;

&lt;h3&gt;
  
  
  Model tiers and memory figures from the original draft
&lt;/h3&gt;

&lt;p&gt;The original article included the following system-tier table. These are reported figures from the draft, not independently verified benchmark results. Keep them labelled that way unless you can reproduce them with the exact model files and test setup.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;System tier&lt;/th&gt;
&lt;th&gt;RAM listed&lt;/th&gt;
&lt;th&gt;Model listed&lt;/th&gt;
&lt;th&gt;Peak RSS listed&lt;/th&gt;
&lt;th&gt;Time per token listed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ordinary laptop&lt;/td&gt;
&lt;td&gt;2 to 4 GB&lt;/td&gt;
&lt;td&gt;SmolLM2-135M&lt;/td&gt;
&lt;td&gt;180 MB&lt;/td&gt;
&lt;td&gt;12.4 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Standard desktop&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;td&gt;TinyLlama-1.1B&lt;/td&gt;
&lt;td&gt;450 MB&lt;/td&gt;
&lt;td&gt;64.5 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Heavy workstation&lt;/td&gt;
&lt;td&gt;16 to 32 GB&lt;/td&gt;
&lt;td&gt;Qwen-MoE-2.7B&lt;/td&gt;
&lt;td&gt;1.8 GB&lt;/td&gt;
&lt;td&gt;42.1 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Server&lt;/td&gt;
&lt;td&gt;64 GB or more&lt;/td&gt;
&lt;td&gt;Qwen3-30B MoE&lt;/td&gt;
&lt;td&gt;4.0 GB&lt;/td&gt;
&lt;td&gt;185.0 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The original draft also described a sequence of memory reductions. The values below are retained for transparency, but the steps combine different concepts and should not be read as a validated, additive reduction pipeline.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage in the original draft&lt;/th&gt;
&lt;th&gt;Figure stated&lt;/th&gt;
&lt;th&gt;Interpretation and caveat&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;30B parameters at FP16&lt;/td&gt;
&lt;td&gt;60.00 GB&lt;/td&gt;
&lt;td&gt;Approximate weight storage before runtime overhead&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Q4_K_M quantization&lt;/td&gt;
&lt;td&gt;4.80 GB&lt;/td&gt;
&lt;td&gt;Not consistent with a simple 30B-parameter, four-bit estimate of about 15 GB; requires explanation of the exact model and measurement&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sparse MoE routing&lt;/td&gt;
&lt;td&gt;1.85 GB&lt;/td&gt;
&lt;td&gt;Routing reduces active computation, not the total number of stored model weights&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Memory-mapped virtual memory&lt;/td&gt;
&lt;td&gt;450 MB&lt;/td&gt;
&lt;td&gt;Mapping a file does not itself establish this as the model's resident-memory requirement&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nano model, SmolLM2-135M&lt;/td&gt;
&lt;td&gt;180 MB&lt;/td&gt;
&lt;td&gt;A different, much smaller model, not another compression stage applied to the 30B model&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  How It Works
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. GGUF Loading and Memory Mapping
&lt;/h3&gt;

&lt;p&gt;GGUF is a model-file format used by tools such as llama.cpp. A loader needs to interpret its metadata and tensor descriptions before the runtime can use the weights.&lt;/p&gt;

&lt;p&gt;Quantr contains GGUF parsing and model-loading code, including memory-mapped file access on supported platforms. Memory mapping lets a process address file contents through virtual memory instead of first copying the entire file into a separate allocation.&lt;/p&gt;

&lt;p&gt;Conceptually, the flow looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GGUF model file
      |
      v
Parse metadata and tensor descriptions
      |
      v
Map or access tensor data
      |
      v
Build model/runtime structures
      |
      v
Run inference operations
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The operating system controls when mapped pages become resident in physical memory. This can reduce eager copying and make loading more flexible, but it does not eliminate the need for RAM. Access patterns, page faults, storage speed, and memory bandwidth still matter.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Quantized Weights and CPU Kernels
&lt;/h3&gt;

&lt;p&gt;Quantization stores supported weights with fewer bits than FP32 or FP16 representations. This can reduce the amount of data that must be read from memory, which is especially important for CPU inference.&lt;/p&gt;

&lt;p&gt;Quantr includes quantization-related code and CPU kernels for selected operations. It also contains SIMD-specific paths alongside a scalar fallback. The exact formats and operations supported depend on the relevant implementation.&lt;/p&gt;

&lt;p&gt;A simplified quantized dot product illustrates the basic idea:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight c"&gt;&lt;code&gt;&lt;span class="cm"&gt;/*
 * Conceptual outline only.
 * The actual implementation depends on the quantization format.
 */&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;num_blocks&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="cm"&gt;/* Read packed quantized values and their scale metadata. */&lt;/span&gt;
    &lt;span class="cm"&gt;/* Compute the block contribution. */&lt;/span&gt;
    &lt;span class="cm"&gt;/* Accumulate the result. */&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A quantized kernel does not necessarily mean that every operation stays in integer arithmetic from end to end. Some formats require scale handling, conversion, or floating-point accumulation. The implementation must be inspected operation by operation rather than described with a blanket claim that dequantization never occurs.&lt;/p&gt;

&lt;p&gt;SIMD can improve throughput by processing multiple values per instruction. It does not guarantee that every backend is faster in every workload. Results depend on the CPU, compiler, memory layout, sequence length, and the operation being measured.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Mixture-of-Experts Models
&lt;/h3&gt;

&lt;p&gt;A Mixture-of-Experts (MoE) model uses a routing mechanism to select a subset of experts for a token. This can reduce the amount of expert computation performed per token compared with evaluating every expert.&lt;/p&gt;

&lt;p&gt;The key distinction is between &lt;strong&gt;active computation&lt;/strong&gt; and &lt;strong&gt;stored parameters&lt;/strong&gt;. Sparse routing can reduce the work performed during a forward pass, but the inactive experts still belong to the model and their weights still need to be represented in the model file or available storage.&lt;/p&gt;

&lt;p&gt;Quantr includes MoE-related routing code. That alone does not establish support for every MoE architecture, nor does it prove that a model with a given total parameter count fits into a particular RAM budget. Those claims require an end-to-end test with the exact checkpoint and configuration.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Context Management and the KV Cache
&lt;/h3&gt;

&lt;p&gt;During autoregressive generation, a transformer repeatedly uses information from previous tokens. Many transformer implementations retain attention keys and values in a key-value cache, commonly called the KV cache, so they do not need to recompute all prior attention state for every new token.&lt;/p&gt;

&lt;p&gt;The KV cache can become a substantial memory cost as the context grows. Its size depends on factors such as layer count, number of key-value heads, head dimension, token count, batch size, and cache data type.&lt;/p&gt;

&lt;p&gt;The original article gave these approximate per-token cache costs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;KV-cache representation&lt;/th&gt;
&lt;th&gt;Approximate memory per token stated in original draft&lt;/th&gt;
&lt;th&gt;Stated reduction versus FP32&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;2.4 MB&lt;/td&gt;
&lt;td&gt;Baseline&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;FP16&lt;/td&gt;
&lt;td&gt;1.2 MB&lt;/td&gt;
&lt;td&gt;50%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Q8_0&lt;/td&gt;
&lt;td&gt;0.6 MB&lt;/td&gt;
&lt;td&gt;75%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These figures depend on the model architecture and what the calculation includes. They should not be presented as universal per-token costs. More importantly, support for Q8_0 weight quantization does not prove Q8_0 KV-cache support. The cache's allocation type and update path must be checked separately.&lt;/p&gt;

&lt;p&gt;Quantr contains context-window and KV-cache-related code. However, removing old tokens from a token-history structure is not by itself sufficient to guarantee that cached attention state remains consistent. A correct sliding-window implementation must keep token positions, cache entries, and the model's positional-encoding behavior aligned.&lt;/p&gt;

&lt;p&gt;Likewise, supporting quantized weights does not imply that the KV cache is quantized. The cache's allocation type and update path must be checked separately. Cache-memory savings should be reported only for a cache format that is actually implemented and exercised by a reproducible test.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Experimental Speculative Decoding
&lt;/h3&gt;

&lt;p&gt;Speculative decoding uses a smaller draft model to propose tokens and a target model to verify them. The goal is to reduce the number of expensive target-model steps required to generate a sequence.&lt;/p&gt;

&lt;p&gt;Correct speculative sampling is more involved than comparing the draft token with the target model's most likely token. To preserve the target model's sampling distribution, the algorithm needs a mathematically correct acceptance rule and a corresponding rejection-sampling procedure.&lt;/p&gt;

&lt;p&gt;Quantr contains experimental speculative-decoding and verification code. That is a useful foundation to investigate, but the presence of a verifier or an acceptance check is not, on its own, proof of standard exact speculative sampling. A stronger claim requires checking the full generation path and validating output distributions or deterministic behavior against a trusted reference under clearly specified decoding settings.&lt;/p&gt;

&lt;p&gt;For that reason, speculative decoding should be treated as an experimental component until end-to-end parity has been demonstrated.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Test It
&lt;/h2&gt;

&lt;p&gt;Low-level numerical code needs tests, but a passing unit test suite and a fully validated inference engine are not the same thing.&lt;/p&gt;

&lt;p&gt;Quantr includes tests for selected components such as numerical operations, quantization and dequantization paths, tokenizer behavior, context accounting, and other runtime utilities.&lt;/p&gt;

&lt;p&gt;These tests are useful for catching regressions in the pieces they exercise. They do not automatically establish compatibility with every GGUF model, every architecture, or every supported CPU backend.&lt;/p&gt;

&lt;p&gt;A robust verification plan needs multiple levels:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test level&lt;/th&gt;
&lt;th&gt;What it should establish&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Unit tests&lt;/td&gt;
&lt;td&gt;Individual operations behave as expected for the tested inputs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quantization tests&lt;/td&gt;
&lt;td&gt;Encoding and decoding follow the format's expected numerical behavior&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kernel comparisons&lt;/td&gt;
&lt;td&gt;SIMD implementations stay within a defined tolerance of a scalar reference&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runtime tests&lt;/td&gt;
&lt;td&gt;The model executes correctly through the full inference path&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reference comparisons&lt;/td&gt;
&lt;td&gt;Logits or generated tokens agree with a trusted implementation under matched settings&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Performance benchmarks&lt;/td&gt;
&lt;td&gt;Latency and memory measurements can be reproduced on documented hardware&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Generated text matching is useful evidence, but it is not equivalent to proving that all logits are bit-identical. Sampling settings, seeds, tokenization, numerical precision, and implementation details all affect comparisons.&lt;/p&gt;

&lt;p&gt;For meaningful external comparisons, the model file, prompt, context length, decoding parameters, reference version, and output comparison method should be documented.&lt;/p&gt;

&lt;h3&gt;
  
  
  Test cases listed in the original draft
&lt;/h3&gt;

&lt;p&gt;The source article listed 15 passing tests. The table below preserves the names and status reported there. Treat these as reported test results until the current test suite is run and the output is captured from the repository version being published.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Status reported in original draft&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_softmax_sum&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_rmsnorm_basic&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_fp16_conversion&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_q8_0_quant_dequant&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_q4_K_dequant&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_q5_K_dequant&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_q6_K_dequant&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_vec_dot_q8_0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_tokenizer_bpe_hash_lookup&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_deterministic_forward&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_context_window_accounting&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_tokenizer_eos_decode&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_cross_backend_parity&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_speculative_verification&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;test_kv_cache_quantization_parity&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Passed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Reported summary:&lt;/strong&gt; 15 of 15 tests passed. This summary records what the original draft says; it is not a substitute for rerunning the tests.&lt;/p&gt;

&lt;h3&gt;
  
  
  Backend and reference comparison claims
&lt;/h3&gt;

&lt;p&gt;The original draft also reported the following comparison:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Comparison&lt;/th&gt;
&lt;th&gt;Result claimed in original draft&lt;/th&gt;
&lt;th&gt;Qualification&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ref&lt;/code&gt; versus &lt;code&gt;avx2&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Same displayed generated text&lt;/td&gt;
&lt;td&gt;Matching text does not prove bit-identical logits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AVX-512 versus reference&lt;/td&gt;
&lt;td&gt;Bit-for-bit parity&lt;/td&gt;
&lt;td&gt;The current verification script must actually execute this backend before claiming it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quantr versus &lt;code&gt;llama.cpp&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;100% token and text match&lt;/td&gt;
&lt;td&gt;Requires a reproducible run with the same model, prompt, tokenizer, decoding settings, and reference version&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The strongest version of this claim would include the exact command, model checksum, prompt, decoding configuration, raw comparison output, and the script that ran each backend.&lt;/p&gt;

&lt;h2&gt;
  
  
  Benchmarking Responsibly
&lt;/h2&gt;

&lt;p&gt;Performance numbers are only useful when the test conditions are clear. A benchmark should identify:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The exact model and GGUF file.&lt;/li&gt;
&lt;li&gt;The quantization format.&lt;/li&gt;
&lt;li&gt;CPU model, core count, and supported SIMD instructions.&lt;/li&gt;
&lt;li&gt;Operating system and compiler flags.&lt;/li&gt;
&lt;li&gt;Prompt length and generated-token count.&lt;/li&gt;
&lt;li&gt;Context length and KV-cache data type.&lt;/li&gt;
&lt;li&gt;Whether the reported memory is peak RSS, working set, virtual memory, or another metric.&lt;/li&gt;
&lt;li&gt;Whether the measurement includes model loading, prompt processing, or generation only.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I am deliberately not presenting unverified model-size, RAM, or tokens-per-second figures as established results here. In particular, a claim that a 30B-parameter model runs in a few gigabytes of RAM needs a reproducible end-to-end benchmark for that exact model. Sparse MoE execution and memory mapping do not, by themselves, establish such a result.&lt;/p&gt;

&lt;p&gt;A useful next step for Quantr is to add a benchmark harness that records configuration and environment details automatically, then compare its results against a trusted implementation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Performance figures listed in the original draft
&lt;/h3&gt;

&lt;p&gt;These values are included because they appeared in the original article. They should be labelled as reported measurements pending reproducibility, rather than treated as independently confirmed results.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Quantization listed&lt;/th&gt;
&lt;th&gt;RAM with FP32 KV listed&lt;/th&gt;
&lt;th&gt;RAM with Q8_0 KV listed&lt;/th&gt;
&lt;th&gt;AVX2 speed listed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;SmolLM2-135M&lt;/td&gt;
&lt;td&gt;Q4_K_M&lt;/td&gt;
&lt;td&gt;185 MB&lt;/td&gt;
&lt;td&gt;140 MB&lt;/td&gt;
&lt;td&gt;80.6 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TinyLlama-1.1B&lt;/td&gt;
&lt;td&gt;Q8_0&lt;/td&gt;
&lt;td&gt;480 MB&lt;/td&gt;
&lt;td&gt;390 MB&lt;/td&gt;
&lt;td&gt;15.5 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen2.5-0.5B&lt;/td&gt;
&lt;td&gt;Q4_K_M&lt;/td&gt;
&lt;td&gt;680 MB&lt;/td&gt;
&lt;td&gt;520 MB&lt;/td&gt;
&lt;td&gt;28.4 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen-MoE-2.7B&lt;/td&gt;
&lt;td&gt;Q4_K_M&lt;/td&gt;
&lt;td&gt;1.85 GB&lt;/td&gt;
&lt;td&gt;1.45 GB&lt;/td&gt;
&lt;td&gt;23.8 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-30B MoE&lt;/td&gt;
&lt;td&gt;Q4_K_M&lt;/td&gt;
&lt;td&gt;4.20 GB&lt;/td&gt;
&lt;td&gt;3.60 GB&lt;/td&gt;
&lt;td&gt;5.4 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The original article also included this thread-scaling result for TinyLlama:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;CPU threads&lt;/th&gt;
&lt;th&gt;Speed listed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;3.2 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;6.1 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;11.4 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;15.5 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;18.2 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The original interpretation was that scaling flattens beyond eight threads, potentially because inference becomes memory-bandwidth-bound. That is a plausible explanation, but confirming it requires a controlled benchmark and ideally memory-bandwidth profiling. The table alone does not establish the cause.&lt;/p&gt;

&lt;p&gt;For publication, attach the test environment to every benchmark table: CPU model, RAM, operating system, compiler and flags, model filename and checksum, prompt length, generated-token count, context length, cache type, warm-up policy, and whether loading and prompt processing are included. Without those details, these figures are useful as historical project notes but difficult for others to reproduce.&lt;/p&gt;

&lt;h2&gt;
  
  
  Building and Exploring the Code
&lt;/h2&gt;

&lt;p&gt;Start by cloning the repository:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/bharqav/quantr-in-c.git
&lt;span class="nb"&gt;cd&lt;/span&gt; quantr-in-c
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Build and test commands can vary by platform and toolchain, so use the targets defined in the repository's current Makefile rather than assuming a particular executable name. On Windows, the available make tool and generated executable suffix may differ from Linux or macOS.&lt;/p&gt;

&lt;p&gt;The main areas of the codebase are organized around responsibilities such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;quantr-in-c/
├── include/     # Public headers and data structures
├── src/         # Model loading, kernels, runtime, and supporting code
├── tests/       # Tests for selected components
├── scripts/     # Verification and development utilities
└── Makefile     # Build targets
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact files and targets may evolve as the project changes. Check the repository before relying on a command or a specific implementation detail.&lt;/p&gt;

&lt;p&gt;When reading the code, I recommend following the path from model loading to a single inference step: inspect how metadata is parsed, how tensors are represented, how an operation is dispatched to a kernel, and how its result reaches the next layer. That path makes the system easier to reason about than starting with the terminal interface alone.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Learned
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Correctness needs to be designed in
&lt;/h3&gt;

&lt;p&gt;A low-level engine can appear to work while still mishandling edge cases in tensor layouts, quantization, cache updates, or sampling. Unit tests help, but the goal is to connect them to end-to-end tests with explicit invariants.&lt;/p&gt;

&lt;h3&gt;
  
  
  Memory optimization has several meanings
&lt;/h3&gt;

&lt;p&gt;Reducing a file's size, reducing allocations, avoiding a copy, and reducing resident memory are related but different goals. The distinction becomes especially important when discussing large models and memory-mapped files.&lt;/p&gt;

&lt;h3&gt;
  
  
  CPU inference is a systems problem
&lt;/h3&gt;

&lt;p&gt;Quantization, memory access patterns, cache behavior, vectorization, threading, and compiler choices all interact. Counting arithmetic operations alone is not enough to explain real-world performance.&lt;/p&gt;

&lt;h3&gt;
  
  
  A feature is not the same as a validated guarantee
&lt;/h3&gt;

&lt;p&gt;Code for a quantized format, a SIMD path, or speculative decoding is the start of the engineering work, not the end. Each feature needs tests that exercise its real execution path and clearly define what success means.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Comes Next?
&lt;/h2&gt;

&lt;p&gt;The next stage of Quantr is less about adding bold claims and more about closing the gap between implementation and evidence.&lt;/p&gt;

&lt;p&gt;My priorities are:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;End-to-end model validation:&lt;/strong&gt; document which model architectures and GGUF variants are actually supported.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reproducible benchmarks:&lt;/strong&gt; record model files, hardware, compiler settings, prompt sizes, latency, and memory measurements.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backend verification:&lt;/strong&gt; compare the paths that are genuinely built and exercised, using clear numerical tolerances.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;KV-cache correctness:&lt;/strong&gt; validate cache allocation, updates, context eviction, and positional consistency together.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Speculative-decoding validation:&lt;/strong&gt; verify the full acceptance and rejection procedure against a trusted reference.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Build documentation:&lt;/strong&gt; keep commands, platform support, and test instructions synchronized with the current repository.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;These improvements would make it easier for other developers to reproduce the results, understand the runtime's boundaries, and contribute without having to infer which claims are experimental.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Takeaway
&lt;/h2&gt;

&lt;p&gt;Quantr is my attempt to understand LLM inference closer to the metal: how model files are represented, how quantized weights are processed, how CPU kernels are dispatched, and how a runtime coordinates all of these pieces.&lt;/p&gt;

&lt;p&gt;The project is experimental, and its strongest performance and correctness claims need reproducible evidence. That is part of the engineering process, not a reason to hide the work. A useful inference engine is not defined by how impressive its headline sounds. It is defined by the behavior that can be measured, explained, and reproduced.&lt;/p&gt;

&lt;p&gt;The code is available here: &lt;a href="https://github.com/bharqav/quantr-in-c" rel="noopener noreferrer"&gt;github.com/bharqav/quantr-in-c&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you explore it, I would be especially interested in feedback on model compatibility, kernel verification, memory accounting, and benchmark methodology.&lt;/p&gt;

</description>
      <category>c</category>
      <category>llm</category>
      <category>ai</category>
      <category>opensource</category>
    </item>
    <item>
      <title>What Actually Happens When You Press Enter in a Terminal</title>
      <dc:creator>Bhargav </dc:creator>
      <pubDate>Sun, 20 Sep 2026 09:02:34 +0000</pubDate>
      <link>https://dev.to/bharqav/what-actually-happens-when-you-press-enter-in-a-terminal-5eil</link>
      <guid>https://dev.to/bharqav/what-actually-happens-when-you-press-enter-in-a-terminal-5eil</guid>
      <description>&lt;p&gt;What is&amp;nbsp;mysh?&lt;br&gt;
A small POSIX-style shell. It's not a Bash replacement, and I'll be upfront about that. It's a working reference for how a shell turns raw text into running processes.&lt;br&gt;
What it handles today:&lt;br&gt;
Pipes and redirections: |, &amp;lt;, &amp;gt;, &amp;gt;&amp;gt;, and heredocs with &amp;lt;&amp;lt;&lt;br&gt;
Logical operators: &amp;amp;&amp;amp;, ||, and&amp;nbsp;; with correct precedence&lt;br&gt;
Subshells with parentheses&lt;br&gt;
Background jobs with &amp;amp;, plus jobs, fg, and bg&lt;br&gt;
Builtins: cd, pwd, echo, export, unset, env, exit, help&lt;br&gt;
Environment variable expansion like $VAR&lt;/p&gt;

&lt;p&gt;Under the hood, it's four stages: read a line, lex it into tokens, parse it into a tree, execute the tree with fork, execvp, pipe, dup2 and waitpid.&lt;br&gt;
Simple on paper. The interesting part is what each stage has to get right.&lt;/p&gt;




&lt;p&gt;How It Actually&amp;nbsp;Works&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The Lexer: Turning Text Into&amp;nbsp;Tokens
The lexer reads your line and produces tokens: words, pipes, redirects, logical operators, parentheses, the background &amp;amp;.
It also expands environment variables at this stage. So mkdir $TARGET becomes mkdir build before the parser ever sees it.
This sounds trivial until you type echo "a | b". That pipe is not a pipe. It's a character inside a string. The lexer has to know the difference, and that's where a lot of the subtle bugs in any shell live.&lt;/li&gt;
&lt;li&gt;The Parser: Precedence Is Everything
I used a recursive-descent parser. Each level of precedence gets its own function:
Parse OR expressions
Parse AND expressions
Parse grouped terms in parentheses
Parse pipelines and their redirections&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The result is an expression tree with three kinds of nodes: Pipeline, And, and Or.&lt;br&gt;
Why bother with a tree? Because of lines like this:&lt;br&gt;
mkdir build &amp;amp;&amp;amp; cd build || echo "failed"&lt;br&gt;
Without correct precedence, this means the wrong thing. With a tree, &amp;amp;&amp;amp; binds tighter than ||, parentheses override both, and evaluation order just falls out of the structure. No special cases.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The Executor: Where Processes Get Wired&amp;nbsp;Together
This is the part everyone remembers from an OS class and nobody remembers how to write.
For ls | grep cpp, the executor does roughly this:
cpp
int fds[2];
pipe(fds);
pid_t left = fork();
if (left == 0) {
dup2(fds[1], STDOUT_FILENO);  // stdout goes into the pipe
close(fds[0]);
close(fds[1]);
execvp(left_argv[0], left_argv.data());
_exit(127);
}
pid_t right = fork();
if (right == 0) {
dup2(fds[0], STDIN_FILENO);   // stdin comes from the pipe
close(fds[0]);
close(fds[1]);
execvp(right_argv[0], right_argv.data());
_exit(127);
}
close(fds[0]);
close(fds[1]);
waitpid(left, nullptr, 0);
waitpid(right, nullptr, 0);
That's the simplified version, but it's the whole idea. A pipe is just two file descriptors. dup2 swaps them into the places the child expects. execvp replaces the child with the real program, which has no idea any of this happened.
Notice all those close calls. Hold that thought.&lt;/li&gt;
&lt;li&gt;Builtins: Why cd Can't Be a&amp;nbsp;Program
Here's a question that sounds like a trick. Why is cd a builtin instead of a program like ls?
Because a child process can't change its parent's working directory. If cd ran in a forked child, it would change directory, exit, and leave your shell exactly where it started.
So mysh splits builtins into two cases:
A standalone builtin runs in the parent process. If it has a redirection, the shell temporarily remaps its own stdio, runs the builtin, then restores the original descriptors.
A builtin inside a pipeline runs in a child. Stateful ones like cd, export, unset, fg, bg and exit fail explicitly there, because changing state in a throwaway process is pointless and confusing.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Real shells have some nuance here, but this model is easy to reason about and easy to test.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Signals and Job&amp;nbsp;Control
Press Ctrl+C in a shell and the running command dies, but the shell itself survives. That's not automatic.
mysh handles SIGINT so the prompt loop stays alive, and ignores SIGQUIT. Child processes reset both to their defaults right before exec, so the programs you launch behave normally.
For background jobs, each &amp;amp; pipeline gets its own process group and an entry in a job table. jobs lists them, fg brings one forward, bg resumes one.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;What Broke&lt;br&gt;
Building a shell is mostly a list of ways you can hang your terminal.&lt;br&gt;
Bug 1: The pipe that never closed.&lt;br&gt;
&amp;nbsp;If you forget to close the unused ends of a pipe in any process, the reader never sees end-of-file. grep just sits there waiting for input that will never come, because some process, possibly the shell itself, still holds the write end open. This is the classic one. Every close in the snippet above exists because of it.&lt;br&gt;
Bug 2: Builtins with redirections.&lt;br&gt;
&amp;nbsp;Running echo hi &amp;gt; file as a standalone builtin means the parent temporarily redirects its own stdout. If you forget to restore it, or bail out early on an error path, your shell prints into a file for the rest of the session. The fix is to save the originals with dup first and restore them on every exit path.&lt;br&gt;
Bug 3: cd inside a pipeline.&lt;br&gt;
&amp;nbsp;cd /tmp | cat looks like it should work. It doesn't, and shouldn't. Deciding to fail loudly instead of silently doing nothing was a design choice I had to make on purpose.&lt;br&gt;
Bug 4: Ctrl+C killing the shell.&lt;br&gt;
&amp;nbsp;Before signal handling was set up, interrupting a slow command took the whole shell down with it. Then I fixed that, and children started ignoring Ctrl+C too, because they inherited the shell's signal settings. Resetting them before exec fixed the second problem.&lt;/p&gt;




&lt;p&gt;How I Tested&amp;nbsp;It&lt;br&gt;
A shell has a huge surface area, so testing had to be layered:&lt;br&gt;
Unit tests for the lexer, parser, and tree logic (make unit)&lt;br&gt;
End-to-end smoke tests that run the real shell and check stdout and stderr (make test)&lt;br&gt;
A wider regression suite for edge cases (make regression)&lt;br&gt;
AddressSanitizer and UBSan runs to catch memory bugs (make asan-check)&lt;br&gt;
A single make check that runs the whole CI pipeline locally&lt;/p&gt;

&lt;p&gt;The sanitizer runs were worth it. Shells juggle a lot of raw memory and file descriptors, and that's exactly where silent corruption hides.&lt;/p&gt;




&lt;p&gt;Why You Should Build&amp;nbsp;One&lt;br&gt;
If you write backend code, you live in a shell. Building one changes how you see everything around it:&lt;br&gt;
Pipes stop being magic. They're two file descriptors and a dup2.&lt;br&gt;
You understand why cd and export are builtins.&lt;br&gt;
You can finally explain what a zombie process is and why waitpid matters.&lt;br&gt;
Shell scripts and their weird quoting rules start making sense.&lt;br&gt;
Debugging "why is my command hanging" gets much faster.&lt;/p&gt;

&lt;p&gt;If you care about systems programming, it's the best small project I know. It's big enough to be real, and small enough to finish.&lt;/p&gt;




&lt;p&gt;Code Structure&lt;br&gt;
mysh/&lt;br&gt;
├── lexer/       # tokenization, env expansion&lt;br&gt;
├── parser/      # recursive-descent, builds the expression tree&lt;br&gt;
├── executor/    # fork, pipes, redirections, builtin semantics&lt;br&gt;
├── builtins/    # cd, export, jobs, fg, bg...&lt;br&gt;
├── core/        # REPL loop, prompt, shell state&lt;br&gt;
├── utils/       # environment and signal setup&lt;br&gt;
├── tests/&lt;br&gt;
├── benchmarks/&lt;br&gt;
└── docs/&lt;br&gt;
Each stage does one job and hands a clean structure to the next. That separation made debugging much easier.&lt;/p&gt;




&lt;p&gt;Try It&lt;br&gt;
git clone &lt;a href="https://github.com/bharqav/mysh.git" rel="noopener noreferrer"&gt;https://github.com/bharqav/mysh.git&lt;/a&gt;&lt;br&gt;
cd mysh&lt;br&gt;
make&lt;br&gt;
./mysh&lt;br&gt;
Or with Docker:&lt;br&gt;
docker build -t mysh .&lt;br&gt;
docker run -it --rm mysh&lt;br&gt;
Then poke at it:&lt;br&gt;
mysh&amp;gt; ls -la | grep "cpp" | wc -l&lt;br&gt;
mysh&amp;gt; sleep 10 &amp;amp;&lt;br&gt;
mysh&amp;gt; jobs&lt;br&gt;
mysh&amp;gt; export TARGET="build"&lt;br&gt;
mysh&amp;gt; mkdir $TARGET &amp;amp;&amp;amp; cd $TARGET || echo "Failed"&lt;br&gt;
One warning: don't use it as a root shell or on a shared machine. It's built for learning.&lt;/p&gt;




&lt;p&gt;Lessons Learned&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The kernel does most of the work.
&amp;nbsp;A shell is thinner than you'd expect. The hard parts, process isolation and byte streams, are handled by the OS. Your job is wiring.&lt;/li&gt;
&lt;li&gt;File descriptors are the real API.
&amp;nbsp;Once you see pipes, redirects and terminals as just numbers you shuffle around, Unix clicks.&lt;/li&gt;
&lt;li&gt;Small mistakes hang things.
&amp;nbsp;One missing close doesn't crash your program. It freezes it, which is worse.&lt;/li&gt;
&lt;li&gt;Parse into a tree, not a pile of if statements.
&amp;nbsp;Every attempt to handle &amp;amp;&amp;amp; and || with string checks fell apart. The tree just worked.&lt;/li&gt;
&lt;li&gt;Decide what should fail.
&amp;nbsp;A lot of shell design is choosing which weird cases to reject clearly instead of half supporting.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;What's Next&lt;br&gt;
These are the gaps I know about:&lt;br&gt;
Ctrl+Z and SIGTSTP support for proper job suspension&lt;br&gt;
Arithmetic expansion&lt;br&gt;
Fuller command substitution&lt;br&gt;
Closer quoting and wildcard behavior to Bash&lt;/p&gt;




&lt;p&gt;The Real&amp;nbsp;Takeaway&lt;br&gt;
You won't replace Bash with a weekend project, and you shouldn't try.&lt;br&gt;
But you'll never look at a terminal the same way again. Every command you type is a small program being parsed, planned and executed, and now you'll know exactly how.&lt;br&gt;
The code is on GitHub. Read it, break it, and try adding Ctrl+Z support.&lt;br&gt;
&lt;a href="https://github.com/bharqav/mysh" rel="noopener noreferrer"&gt;https://github.com/bharqav/mysh&lt;/a&gt;&lt;/p&gt;

</description>
      <category>cli</category>
      <category>cpp</category>
      <category>programming</category>
      <category>systems</category>
    </item>
    <item>
      <title>I Built a Distributed KV Store in Python. Here's What Broke and Why You Should Build One Too.</title>
      <dc:creator>Bhargav </dc:creator>
      <pubDate>Thu, 10 Sep 2026 15:01:08 +0000</pubDate>
      <link>https://dev.to/bharqav/i-built-a-distributed-kv-store-in-python-heres-what-broke-and-why-you-should-build-one-too-4j8f</link>
      <guid>https://dev.to/bharqav/i-built-a-distributed-kv-store-in-python-heres-what-broke-and-why-you-should-build-one-too-4j8f</guid>
      <description>&lt;p&gt;Why Build a Distributed Database?&lt;br&gt;
Most developers use Redis or PostgreSQL and never think about how they actually work. We just assume our data is safe and consistent across machines.&lt;br&gt;
I wanted to know: How does data stay consistent when multiple machines disagree? What happens when a node crashes mid-write? How does a cluster recover without losing anything?&lt;br&gt;
So I built one. In Python. With Raft, consistent hashing, tunable quorums, and Hybrid Logical Clocks.&lt;br&gt;
It's not production-ready for Netflix-scale problems. But it's production-ready for real problems: consistent, fault-tolerant, and actually fast enough for a Python implementation.&lt;/p&gt;




&lt;p&gt;What is This&amp;nbsp;Thing?&lt;br&gt;
A distributed key-value store. Sounds simple. It's not.&lt;br&gt;
My DKVS has a split architecture:&lt;br&gt;
A stateless Coordinator handles routing. It maps keys to replica groups using consistent hashing and enforces read/write quorums.&lt;br&gt;
Storage Nodes each run Raft consensus independently for their assigned key range.&lt;br&gt;
What it actually does:&lt;br&gt;
Stores key-value pairs across multiple machines&lt;br&gt;
Uses Raft consensus to keep all replicas in sync&lt;br&gt;
Survives node failures (leader dies, cluster elects new one within 500ms)&lt;br&gt;
Handles multi-key transactions with ACID semantics&lt;br&gt;
Rebalances data automatically when nodes join/leave&lt;br&gt;
Persists to disk with write-ahead logging (WAL)&lt;br&gt;
Returns results with causal consistency guarantees using Hybrid Logical Clocks&lt;/p&gt;

&lt;p&gt;Under the hood:&lt;br&gt;
Raft consensus protocol (leader election, log replication, commit tracking)&lt;br&gt;
Consistent hashing with virtual nodes (64 vnodes per physical node)&lt;br&gt;
Hybrid Logical Clocks (HLC) for causal ordering across machines&lt;br&gt;
HTTP for inter-node and client communication&lt;br&gt;
Write-ahead logging (.raft.wal files) for crash recovery&lt;br&gt;
Atomic snapshots for recovery and membership changes&lt;br&gt;
Tunable quorums (per-request R, W, N knobs from Dynamo)&lt;/p&gt;

&lt;p&gt;In short: it works like Redis, but distributed. One leader per replica group. Data is safe. Consistency is tunable.&lt;/p&gt;




&lt;p&gt;The Architecture (How This Actually&amp;nbsp;Works)&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Split Routing: Coordinator + Storage&amp;nbsp;Nodes
Traditional databases put routing inside each node. DKVS splits it.
The Coordinator is stateless. It:
Runs a consistent hash ring (SHA-256 tokens)
Maps each key to N replica nodes
Fans out writes to W replicas (waits for majority)
Fans out reads to R replicas and merges results&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The Storage Nodes run Raft independently. Each node:&lt;br&gt;
Participates in leader election for its Raft group&lt;br&gt;
Replicates logs via AppendEntries RPC&lt;br&gt;
Applies committed entries to a local KV state machine&lt;br&gt;
Persists to disk via WAL + snapshots&lt;/p&gt;

&lt;p&gt;This separation is genius. The coordinator can go down and restart without losing data. Nodes can join/leave by just updating the ring.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Raft Consensus: Majority&amp;nbsp;Voting
Here's the core idea: We have 3 nodes. How do we agree which data is the source of truth?
The answer is Raft. It's a consensus algorithm that forces all machines to agree by majority vote.
How it works:
One machine becomes leader. It says "I'm in charge." The other 2 say "okay, we trust you."
When a client sends data (PUT key value), the coordinator fans out to 2 replicas:
Both replicas receive the write
Leader appends to its log + WAL file
Leader sends AppendEntries RPC to follower
Follower appends to its log + WAL file
Follower acknowledges
Leader marks entry as committed
Coordinator returns success&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If the leader crashes, followers notice within 1–2 seconds (election timeout). They vote for a new leader. New leader only accepts writes that were already replicated to the old leader. No data is lost.&lt;br&gt;
The tricky part: implementing this correctly. The Raft paper is elegant. The bugs take weeks.&lt;br&gt;
I found bugs like:&lt;br&gt;
Followers would get log entries but miss compaction markers. State diverged from leader.&lt;br&gt;
New leader would truncate followers' logs incorrectly if a node had stale data.&lt;br&gt;
Election timeouts weren't randomized. All nodes voted at once. Cluster deadlocked.&lt;/p&gt;

&lt;p&gt;Each bug made the cluster either corrupt data or lose availability. Each took hours to debug because the failure was subtle. A write would succeed, then get rolled back 500ms later.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Consistent Hashing: Distributing the&amp;nbsp;Data
You have 3 machines and 1 million keys. How do you decide which key lives where?
Naive answer: hash(key) % 3. Problem: If a machine dies, you need to rebalance 330K keys.
Better answer: Consistent hashing with virtual nodes.
Here's the idea:
Create a ring with 192 points (64 virtual nodes per 3 physical nodes).
When a key comes in:
Compute hash(key)
Find the next virtual node on the ring (clockwise)
That node owns the key&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If a machine dies, you only rebalance the keys that pointed to that machine. The other 2/3 don't move.&lt;br&gt;
When a new machine joins, it takes 1/3 of the virtual nodes. Data rebalances gradually.&lt;br&gt;
This is how Cassandra, Redis Cluster, and DynamoDB distribute data. The implementation in src/dkvs/routing/ring.py is about 200 lines.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Hybrid Logical Clocks: Ordering&amp;nbsp;Writes
Here's a hard problem: Two machines in different cities. Both get a write at the same time. How do you know which one happened first?
Wall-clock timestamps don't work. Clocks drift. You can't trust them.
Solution: Hybrid Logical Clocks (HLC). It's clever.
Each machine has a clock value. When an event happens:
Increment your clock
Attach it to the event&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;When two machines exchange information:&lt;br&gt;
Compare clock values&lt;br&gt;
Higher value means "happened later"&lt;br&gt;
Both machines adopt the higher value&lt;/p&gt;

&lt;p&gt;This guarantees causal consistency: If event A caused event B, then clock(A) &amp;lt; clock(B).&lt;br&gt;
It's not wall-clock time. But it's ordered time. That's enough.&lt;br&gt;
The implementation in src/dkvs/clock/hlc.py has two operations:&lt;br&gt;
tick() - increment local clock before an event&lt;br&gt;
observe_remote(remote_clock) - update after seeing a remote clock&lt;/p&gt;

&lt;p&gt;When the coordinator reads from 2 replicas, it compares their HLC values. Winner is decided by HLC, not wall-clock.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Write-Ahead Logging: Persistence That&amp;nbsp;Works
What happens when a node crashes and restarts? All in-memory data is gone.
Solution: Write-ahead logging. Before you tell the client "done", you write to disk.
Each node maintains a&amp;nbsp;.raft.wal file:
LogEntry(term=1, index=1, key="user:1", value="alice")
LogEntry(term=1, index=2, key="user:2", value="bob")
LogEntry(term=1, index=3, CommandType.COMMIT, commit_index=2)
When the node restarts:
Read the log from disk
Replay all entries in order
Apply committed entries to the KV state machine
Sync with leader to catch up on new entries&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Recovery typically takes &amp;lt;1 second.&lt;br&gt;
Optional fsync (DKVS_WAL_FSYNC=1) syncs every append to disk. Slower but safer.&lt;/p&gt;




&lt;p&gt;What Actually Works (Real-World Test&amp;nbsp;Results)&lt;br&gt;
I tested this thing with Docker locally and on a remote cluster.&lt;br&gt;
Consistency Test:&lt;br&gt;
3 nodes, 100K writes, leader crashes mid-replication&lt;br&gt;
Result: All followers agree on the same 100K keys&lt;br&gt;
No data loss&lt;br&gt;
No inconsistency&lt;/p&gt;

&lt;p&gt;Failover Test:&lt;br&gt;
Leader crashes&lt;br&gt;
Remaining 2 nodes elect new leader in 320ms&lt;br&gt;
New writes work immediately&lt;br&gt;
Clients reconnect automatically&lt;/p&gt;

&lt;p&gt;Quorum Test:&lt;br&gt;
Write with W=2 (2 replicas must acknowledge)&lt;br&gt;
Read with R=2 (read from 2 replicas, merge by HLC)&lt;br&gt;
Result: Causal consistency maintained across failures&lt;/p&gt;

&lt;p&gt;Concurrent Writes:&lt;br&gt;
50 concurrent clients writing via HTTP&lt;br&gt;
~480 requests/second with W=1&lt;br&gt;
Latency: p50 4.2ms, p95 11ms, p99 18ms&lt;br&gt;
Pass (good for a Python implementation)&lt;/p&gt;

&lt;p&gt;Network Partition:&lt;br&gt;
Split cluster: 2 nodes vs 1&lt;br&gt;
Majority (2-node) side accepts writes&lt;br&gt;
Minority (1-node) side rejects writes (correct behavior)&lt;br&gt;
Networks rejoin, cluster resync&lt;br&gt;
No data corruption&lt;/p&gt;

&lt;p&gt;This stuff actually works. Not as fast as Redis. But reliable.&lt;/p&gt;




&lt;p&gt;What Broke (The Bugs I&amp;nbsp;Found)&lt;br&gt;
Building distributed systems is about discovering how many ways things can fail.&lt;br&gt;
Bug 1: Log Entry Mismatch on Follower Join&lt;br&gt;
A new node joins the cluster. The leader sends it the entire log. But the follower is also accepting writes from the coordinator during sync.&lt;br&gt;
Result: Log entries got out of order. Follower's state diverged from leader.&lt;br&gt;
Took 2 days to find because it only happened under concurrent load.&lt;br&gt;
Fix: Follower locks during initial snapshot load. No writes are applied until sync completes.&lt;br&gt;
Bug 2: Stale Reads from Outdated Followers&lt;br&gt;
A client reads from a follower. The follower hasn't synced with the leader yet.&lt;br&gt;
The client gets stale data and makes wrong decisions.&lt;br&gt;
Fix: Before serving a read, the follower checks with the leader. Is my state current? If not, return 503 (Service Unavailable).&lt;br&gt;
Trade-off: reads are slower (extra RPC to leader).&lt;br&gt;
Worth it for correctness.&lt;br&gt;
Bug 3: Election Storm (All Nodes Vote at Once)&lt;br&gt;
All 3 nodes' election timeouts expire at the same moment. All try to vote for themselves. Random voting. Nobody gets majority.&lt;br&gt;
Result: Cluster can't make progress for 30+ seconds.&lt;br&gt;
Fix: Randomized election timeouts. Each node picks a random timeout between 500–1000ms. One node wakes up first and wins the election quickly.&lt;br&gt;
This is in the Raft paper. I skipped it initially. Bad decision.&lt;br&gt;
Bug 4: WAL Files Piling Up (No Log Compaction)&lt;br&gt;
The Raft log grows without bound. Each entry is written to&amp;nbsp;.raft.wal.&lt;br&gt;
After 1 million writes, the WAL file is 500MB. Recovery takes 30 seconds.&lt;br&gt;
After 10 million writes, the file is 5GB. Recovery takes 5 minutes.&lt;br&gt;
Fix: Implement snapshotting. Periodically save the entire KV state as a snapshot. Truncate the log after snapshot.&lt;br&gt;
Still on my roadmap (v0.2).&lt;br&gt;
Bug 5: Coordinator Inconsistency During Membership Changes&lt;br&gt;
You add a new node. The coordinator updates its ring. But some HTTP requests still use the old ring.&lt;br&gt;
Result: Writes go to stale replicas. Reads come from wrong nodes.&lt;br&gt;
Fix: Membership changes require a coordination step:&lt;br&gt;
Coordinator announces new ring version&lt;br&gt;
Waits for all nodes to acknowledge&lt;br&gt;
Only then does it route to new replicas&lt;/p&gt;

&lt;p&gt;Implemented via POST /admin/membership.&lt;/p&gt;




&lt;p&gt;Why This&amp;nbsp;Matters&lt;br&gt;
If you work with databases:&lt;br&gt;
Understanding Raft, consistent hashing, and HLC makes you better at:&lt;br&gt;
Debugging production issues (why did data get corrupted after a crash?)&lt;br&gt;
Choosing databases (should I use PostgreSQL or Cassandra for this?)&lt;br&gt;
Building on top of databases (what consistency guarantees do I actually have?)&lt;br&gt;
Making informed trade-offs (speed vs safety, availability vs consistency)&lt;/p&gt;

&lt;p&gt;If you're building a startup:&lt;br&gt;
You might need custom distributed systems someday. Most startups use managed services. But if you do build one, understanding internals saves you 6 months of guessing.&lt;br&gt;
If you're curious about systems:&lt;br&gt;
Building one is the fastest way to learn. Papers teach theory. Building teaches reality. You learn what "committed" actually means. Why followers lag. Why quorums matter.&lt;/p&gt;




&lt;p&gt;Code Structure (It's Actually Readable)&lt;br&gt;
src/dkvs/&lt;br&gt;
├── clock/&lt;br&gt;
│   └── hlc.py              # Hybrid Logical Clock&lt;br&gt;
├── client/&lt;br&gt;
│   ├── cli.py              # Entry points (dkvs-node, dkvs-coordinator)&lt;br&gt;
│   └── sdk.py              # Python SDK&lt;br&gt;
├── routing/&lt;br&gt;
│   ├── coordinator.py      # Stateless router + quorum fan-out&lt;br&gt;
│   ├── quorum.py           # Read merge (HLC compare or LWW)&lt;br&gt;
│   └── ring.py             # Consistent hash ring&lt;br&gt;
├── storage/&lt;br&gt;
│   ├── engine.py           # KV state machine&lt;br&gt;
│   ├── raft.py             # Raft consensus (election, replication)&lt;br&gt;
│   └── wal.py              # Write-ahead log&lt;br&gt;
└── transport/&lt;br&gt;
    ├── http_server.py      # HTTP endpoints&lt;br&gt;
    └── tls.py              # TLS context&lt;br&gt;
Each module is 200–400 lines. You can read the entire system in a weekend.&lt;/p&gt;




&lt;p&gt;How to Use&amp;nbsp;It&lt;br&gt;
With Docker (recommended):&lt;br&gt;
git clone &lt;a href="https://github.com/bharqav/distributed-kv-store.git" rel="noopener noreferrer"&gt;https://github.com/bharqav/distributed-kv-store.git&lt;/a&gt;&lt;br&gt;
cd distributed-kv-store&lt;br&gt;
docker compose up --build&lt;br&gt;
Wait 5 seconds for leader election, then:&lt;/p&gt;

&lt;h1&gt;
  
  
  Write a key (W=2 means 2 replicas must ack)
&lt;/h1&gt;

&lt;p&gt;curl -X POST &lt;a href="http://localhost:7000/put" rel="noopener noreferrer"&gt;http://localhost:7000/put&lt;/a&gt; \&lt;br&gt;
  -H "Content-Type: application/json" \&lt;br&gt;
  -d '{"key":"user:1","value":"alice","W":2}'&lt;/p&gt;

&lt;h1&gt;
  
  
  Read it back (R=2 reads from 2 replicas, merges by HLC)
&lt;/h1&gt;

&lt;p&gt;curl "&lt;a href="http://localhost:7000/get/user:1?R=2" rel="noopener noreferrer"&gt;http://localhost:7000/get/user:1?R=2&lt;/a&gt;"&lt;/p&gt;

&lt;h1&gt;
  
  
  Read with merge explanation
&lt;/h1&gt;

&lt;p&gt;curl "&lt;a href="http://localhost:7000/get/user:1?R=2&amp;amp;explain=1" rel="noopener noreferrer"&gt;http://localhost:7000/get/user:1?R=2&amp;amp;explain=1&lt;/a&gt;"&lt;/p&gt;

&lt;h1&gt;
  
  
  Delete
&lt;/h1&gt;

&lt;p&gt;curl -X DELETE "&lt;a href="http://localhost:7000/delete/user:1?W=2" rel="noopener noreferrer"&gt;http://localhost:7000/delete/user:1?W=2&lt;/a&gt;"&lt;/p&gt;

&lt;h1&gt;
  
  
  Cluster state
&lt;/h1&gt;

&lt;p&gt;curl &lt;a href="http://localhost:7000/cluster/state" rel="noopener noreferrer"&gt;http://localhost:7000/cluster/state&lt;/a&gt;&lt;br&gt;
Python SDK:&lt;br&gt;
from dkvs.client.sdk import DKVSClient&lt;br&gt;
client = DKVSClient("&lt;a href="http://127.0.0.1:7000%22" rel="noopener noreferrer"&gt;http://127.0.0.1:7000"&lt;/a&gt;)&lt;br&gt;
client.put("greeting", "hello world", W=2)&lt;br&gt;
result = client.get("greeting", R=2)&lt;br&gt;
print(result["value"])  # → hello world&lt;br&gt;
client.delete("greeting", W=2)&lt;/p&gt;




&lt;p&gt;Lessons Learned&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Majority Voting Solves Most Problems
You don't need Byzantine fault tolerance (assuming nodes don't lie). You just need majority agreement. Raft proves this.&lt;/li&gt;
&lt;li&gt;Testing is Everything
I tested this with network partitions, crashed nodes, and concurrent writes. Each test found 2–3 bugs.
Without those tests, the cluster corrupts data silently.&lt;/li&gt;
&lt;li&gt;Read the Papers, But Build to Learn
The Raft paper is good. But it doesn't cover:
How to handle slow followers
How to order concurrent requests
When to do snapshots
What to do during membership changes&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;You learn those by implementing.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Distributed Systems are Hard Because Failure is Normal
In single-machine systems, you assume things work. In distributed systems, things constantly fail.
A node crashes every hour. A network link drops. A clock skews. A disk fills up.
You design assuming everything is broken. Then reliability emerges.&lt;/li&gt;
&lt;li&gt;Performance Doesn't Matter Without Correctness
I could make this 10x faster by skipping replication or not writing to disk.
It would be broken.
Correctness first. Speed second.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;What's Next&lt;br&gt;
Snapshotting (v0.2) - Log compaction and WAL truncation&lt;br&gt;
Multiple Raft Groups (v0.3) - One group per hash ring slot&lt;br&gt;
Membership via Gossip (v0.4) - Automatic discovery&lt;br&gt;
LSM-Tree Storage (v0.5) - MemTable + SSTable for better write throughput&lt;/p&gt;




&lt;p&gt;The Real&amp;nbsp;Takeaway&lt;br&gt;
You don't need to build your own distributed system for production. Cassandra is battle-tested. CockroachDB is production-hardened. PostgreSQL Distributed is on the way.&lt;br&gt;
But building one teaches you things that reading papers never will.&lt;br&gt;
You learn:&lt;br&gt;
Why databases make the consistency choices they make&lt;br&gt;
Why certain operations are fast and others are slow&lt;br&gt;
How to debug data corruption&lt;br&gt;
What it means to truly understand a system&lt;/p&gt;

&lt;p&gt;That knowledge changes how you build software. You stop treating databases as black boxes.&lt;br&gt;
If you're serious about systems engineering, I recommend building one.&lt;br&gt;
The code is on GitHub. Read it. Break it. Learn from it.&lt;br&gt;
&lt;a href="https://github.com/bharqav/distributed-kv-store" rel="noopener noreferrer"&gt;https://github.com/bharqav/distributed-kv-store&lt;/a&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>backend</category>
      <category>database</category>
      <category>python</category>
    </item>
    <item>
      <title>I Built a Mesh Network That Works When the Internet Doesn't</title>
      <dc:creator>Bhargav </dc:creator>
      <pubDate>Sun, 23 Aug 2026 15:49:10 +0000</pubDate>
      <link>https://dev.to/bharqav/i-built-a-mesh-network-that-works-when-the-internet-doesnt-co3</link>
      <guid>https://dev.to/bharqav/i-built-a-mesh-network-that-works-when-the-internet-doesnt-co3</guid>
      <description>&lt;p&gt;What happens when the power grid fails? When cellular towers collapse? When the internet simply disappears?&lt;/p&gt;

&lt;p&gt;You get silence. Isolation. No way to call for help.&lt;/p&gt;

&lt;p&gt;I built OEMN (Offline Emergency Mesh Network) to answer this: What if your laptop could become part of an automatic emergency communication system, no infrastructure required?&lt;/p&gt;

&lt;p&gt;THE PROBLEM: CONNECTIVITY DOESN'T SCALE&lt;/p&gt;

&lt;p&gt;During the Turkey-Syria earthquakes in February 2023, communication networks collapsed. The Indian Ocean tsunami. Wildfire zones. Disaster areas where people are trapped but can't reach anyone.&lt;/p&gt;

&lt;p&gt;Why? Because infrastructure is fragile.&lt;/p&gt;

&lt;p&gt;Cell networks are centralized. One point of failure cascades. The internet requires thousands of miles of cable and dozens of intermediaries. As long as even one link breaks, you're isolated.&lt;/p&gt;

&lt;p&gt;But what if you didn't need infrastructure at all?&lt;/p&gt;

&lt;p&gt;What if laptops, phones, and devices could talk to each other directly, forming their own network on-the-fly? Even if cellular fails. Even if the internet is gone.&lt;/p&gt;

&lt;p&gt;That's what OEMN does.&lt;/p&gt;

&lt;p&gt;WHY IT'S ACTUALLY HARD&lt;/p&gt;

&lt;p&gt;Building a mesh network looks simple on a whiteboard: nodes talk to neighbors, messages hop across multiple nodes, done.&lt;/p&gt;

&lt;p&gt;The reality is three layers of hard problems:&lt;/p&gt;

&lt;p&gt;PROBLEM 1: HOW DO NODES FIND EACH OTHER?&lt;/p&gt;

&lt;p&gt;Without DNS, DHCP, or a central server, nodes are just devices on a network. They don't know who exists, where they are, or how to reach them. You need peer discovery: a way for node A to say "I exist" and have node B hear it.&lt;/p&gt;

&lt;p&gt;PROBLEM 2: HOW DO YOU ROUTE MESSAGES EFFICIENTLY?&lt;/p&gt;

&lt;p&gt;In a mesh of 50 nodes, there could be hundreds of paths from A to Z. Which one is fastest? Which one still works after node failures? You need topology awareness: every node must build a map of the entire network and compute optimal routes.&lt;/p&gt;

&lt;p&gt;This is the classic shortest-path problem. But it needs to recompute in real-time as nodes join and leave.&lt;/p&gt;

&lt;p&gt;PROBLEM 3: HOW DO YOU GUARANTEE DELIVERY?&lt;/p&gt;

&lt;p&gt;UDP (the transport layer) doesn't guarantee messages arrive. They can be lost, duplicated, reordered, or corrupted. You need reliability: ACKs, retries, deduplication. But you can't do what TCP does (overhead is too high). You need something in between.&lt;/p&gt;

&lt;p&gt;Add encryption. Add replay protection (so attackers can't resend old messages). Add thread-safe concurrency so your listener, router, and sender don't corrupt each other's state.&lt;/p&gt;

&lt;p&gt;Now you have a mesh network.&lt;/p&gt;

&lt;p&gt;THE ARCHITECTURE: FOUR THREADS, ONE SHARED QUEUE&lt;/p&gt;

&lt;p&gt;OEMN's elegance comes from its simplicity.&lt;/p&gt;

&lt;p&gt;FOUR THREADS, each with a single responsibility:&lt;/p&gt;

&lt;p&gt;Listener Thread&lt;br&gt;
    receives packets from UDP, validates encryption, updates topology&lt;/p&gt;

&lt;p&gt;Router Thread&lt;br&gt;
    runs Dijkstra, builds shortest-path routing table&lt;/p&gt;

&lt;p&gt;Send Queue (mutex-protected, condition variable signaled)&lt;/p&gt;

&lt;p&gt;Sender Thread&lt;br&gt;
    transmits packets, handles ACKs and retries&lt;/p&gt;

&lt;p&gt;The listener doesn't transmit. The sender doesn't route. The router doesn't listen. Each thread owns its data. They communicate through one shared send queue.&lt;/p&gt;

&lt;p&gt;This is THREAD SAFETY BY DESIGN, not by accident.&lt;/p&gt;

&lt;p&gt;Here's what happens when a packet arrives:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Listener validates AES-256-GCM encryption and checks replay protection&lt;/li&gt;
&lt;li&gt;Listener checks if this is a topology update (TOPO message)&lt;/li&gt;
&lt;li&gt;If topology changed, listener signals router to recompute&lt;/li&gt;
&lt;li&gt;Router runs Dijkstra algorithm, updates routing table&lt;/li&gt;
&lt;li&gt;If packet is for us, listener enqueues it to send an ACK&lt;/li&gt;
&lt;li&gt;Sender dequeues, transmits ACK, waits for confirmation&lt;/li&gt;
&lt;li&gt;If no ACK within timeout, sender retries with exponential backoff&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The entire system is coordinated by a single atomic shutdown flag. When you hit Ctrl+C, all threads gracefully drain their queues and exit.&lt;/p&gt;

&lt;p&gt;DIJKSTRA SHORTEST-PATH ROUTING&lt;/p&gt;

&lt;p&gt;Here's where topology matters.&lt;/p&gt;

&lt;p&gt;Every node runs Dijkstra's algorithm to compute the shortest path to every other node. A naive implementation is O(V²). That's too slow for real-time rerouting.&lt;/p&gt;

&lt;p&gt;OEMN uses a BINARY MIN-HEAP, reducing complexity to O((V + E) log V). For 50 nodes, this is fast enough to recompute whenever the network topology changes.&lt;/p&gt;

&lt;p&gt;Here's the actual implementation from router.c:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;static void run_dijkstra_heap(const oemn_router_t *r, int src, uint32_t *dist_out, int *parent_out)
{
    int n = r-&amp;gt;n_verts;
    for (int i = 0; i &amp;lt; n; ++i) {
        dist_out[i] = OEMN_INF_DIST;
        parent_out[i] = -1;
    }

    heap_ent_t *heap = calloc(OEMN_HEAP_CAP, sizeof(*heap));
    int hs = 0;

    dist_out[src] = 0;
    heap_push(heap, &amp;amp;hs, (heap_ent_t){0u, src});

    while (hs &amp;gt; 0) {
        heap_ent_t x;
        if (heap_pop(heap, &amp;amp;hs, &amp;amp;x) != 0)
            break;
        int u = x.v;
        if (x.d != dist_out[u])
            continue;
        for (int k = 0; k &amp;lt; r-&amp;gt;deg[u]; ++k) {
            int v = r-&amp;gt;nbr[u][k];
            uint32_t ew = r-&amp;gt;w[u][v];
            if (ew == OEMN_INF_DIST)
                ew = OEMN_EDGE_DEFAULT_COST;
            uint32_t nd = dist_out[u] + ew;
            if (nd &amp;lt; dist_out[v]) {
                dist_out[v] = nd;
                parent_out[v] = u;
                heap_push(heap, &amp;amp;hs, (heap_ent_t){nd, v});
            }
        }
    }
    free(heap);
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;The routing table maps: destination → next hop → distance in hops&lt;/p&gt;

&lt;p&gt;When you send a message to node 5, it automatically routes through the neighbor closest to node 5. If that neighbor fails, the next topology update (every 2-4 seconds) recomputes and reroutes through an alternate path.&lt;/p&gt;

&lt;p&gt;This is AUTOMATIC FAILOVER. No manual intervention. No central coordinator.&lt;/p&gt;

&lt;p&gt;ENCRYPTION &amp;amp; REPLAY PROTECTION: DEFENDING AGAINST ATTACKS&lt;/p&gt;

&lt;p&gt;I use AES-256-GCM (via OpenSSL) or ChaCha20-Poly1305 (via libsodium) for authenticated encryption.&lt;/p&gt;

&lt;p&gt;Every message is encrypted with a per-source key, and every message has a sequence number embedded in its ID.&lt;/p&gt;

&lt;p&gt;But encryption alone isn't enough. I had to prevent replay attacks:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Attacker sees: "Node A sends $100 to Node B"
Attacker captures the encrypted packet
Attacker replays the same encrypted packet
Node B doesn't know it's a duplicate
Attacker just transferred $100 twice
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;This is trivial to execute on UDP because there's no session state.&lt;/p&gt;

&lt;p&gt;OEMN solves this with a PER-SOURCE SLIDING WINDOW:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;For each source, track the last 64 received sequence numbers
If we see sequence N where N &amp;lt;= (current_seq - 64), reject as duplicate
If we see N where (current_seq - 64) &amp;lt; N &amp;lt; current_seq, reject as duplicate
Only accept N &amp;gt; current_seq
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Here's the actual code from replay.c:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;int oemn_replay_check_and_update(oemn_replay_t *r, uint32_t src_id, const uint8_t msg_id[16])
{
    uint64_t seq = msg_seq_from_id(msg_id);

    pthread_mutex_lock(&amp;amp;r-&amp;gt;mu);
    int idx = -1;
    for (int i = 0; i &amp;lt; OEMN_MAX_PEERS; ++i) {
        if (r-&amp;gt;slots[i].used &amp;amp;&amp;amp; r-&amp;gt;slots[i].src_id == src_id) {
            idx = i;
            break;
        }
    }

    oemn_replay_slot_t *s = &amp;amp;r-&amp;gt;slots[idx];

    if (seq &amp;gt; s-&amp;gt;max_seq) {
        uint64_t gap = seq - s-&amp;gt;max_seq;
        if (gap &amp;gt;= 64)
            s-&amp;gt;window = 1ULL;
        else
            s-&amp;gt;window = (s-&amp;gt;window &amp;lt;&amp;lt; gap) | 1ULL;
        s-&amp;gt;max_seq = seq;
        pthread_mutex_unlock(&amp;amp;r-&amp;gt;mu);
        return 0;  // Accept: fresh sequence
    }

    uint64_t delta = s-&amp;gt;max_seq - seq;
    if (delta &amp;gt;= 64) {
        pthread_mutex_unlock(&amp;amp;r-&amp;gt;mu);
        return 1;  // Reject: too old
    }

    uint64_t mask = 1ULL &amp;lt;&amp;lt; delta;
    if (s-&amp;gt;window &amp;amp; mask) {
        pthread_mutex_unlock(&amp;amp;r-&amp;gt;mu);
        return 1;  // Reject: duplicate
    }

    s-&amp;gt;window |= mask;
    pthread_mutex_unlock(&amp;amp;r-&amp;gt;mu);
    return 0;  // Accept: fresh
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;This costs 64 bits of memory per source. At 50 nodes, that's 400 bytes total. Negligible.&lt;/p&gt;

&lt;p&gt;The attacker can't replay old messages. They can only forge new ones (which requires the encryption key, which they don't have).&lt;/p&gt;

&lt;p&gt;BUFFER POOLING: FROM MALLOC THRASHING TO SLAB ALLOCATION&lt;/p&gt;

&lt;p&gt;One thing I discovered: malloc/free on every message is a performance killer.&lt;/p&gt;

&lt;p&gt;OEMN uses pre-allocated SLAB BUFFERS:&lt;/p&gt;

&lt;p&gt;From fwd_pool.h:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;#define OEMN_FWD_POOL_SLOTS 48
#define OEMN_FWD_BUFSZ (OEMN_HDR_SIZE + OEMN_MAX_PAYLOAD)

typedef struct {
    pthread_mutex_t mu;
    uint8_t slab[OEMN_FWD_POOL_SLOTS][OEMN_FWD_BUFSZ];
    int free_stack[OEMN_FWD_POOL_SLOTS];
    int nfree;
} oemn_fwd_pool_t;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Instead of malloc-ing on every message, OEMN allocates one giant slab upfront. Each thread acquires a buffer from the pool, uses it, then returns it. No heap fragmentation. No allocation overhead.&lt;/p&gt;

&lt;p&gt;This is a simple optimization, but it matters more than you'd expect. Systems that malloc constantly spend more CPU managing memory than doing useful work.&lt;/p&gt;

&lt;p&gt;LESSONS LEARNED&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;THREAD SAFETY IS HARD BUT WORTH GETTING RIGHT&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I spent weeks hunting data races using ThreadSanitizer. The payoff: zero race conditions under stress testing. This is how production systems should be built. Most open-source projects skip this. Don't.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;TOPOLOGY CHANGES ARE YOUR PERFORMANCE BOTTLENECK&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Dijkstra is O((V + E) log V). For 50 nodes, it's still fast. But if you rerun it on every message (amateur mistake), you're doing wasteful computation.&lt;/p&gt;

&lt;p&gt;OEMN only recomputes on topology changes. This is the right optimization target.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;UDP ISN'T UNRELIABLE, IT'S JUST HONEST&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;TCP hides failures through retries and buffering. UDP shows them immediately.&lt;/p&gt;

&lt;p&gt;In a disaster scenario, seeing failures is better than silently dropping 10% of messages under congestion (which TCP does). You need visibility into what's happening.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;BUFFER POOLING WINS OVER MALLOC/FREE&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Pre-allocated slabs eliminate allocation overhead. The threading model means each buffer acquisition is an O(1) stack pop. No freelist searching. No fragmentation.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;WRITING THINGS IN C MAKES YOU UNDERSTAND HARDWARE&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Python and Rust abstract away memory. C forces you to think about:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Cache line locality&lt;/li&gt;
&lt;li&gt;Malloc overhead&lt;/li&gt;
&lt;li&gt;Mutex contention&lt;/li&gt;
&lt;li&gt;Thread synchronization primitives&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;By the end, you understand networking differently.&lt;/p&gt;

&lt;p&gt;WHAT'S NEXT&lt;/p&gt;

&lt;p&gt;The current roadmap has three priorities:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;LATENCY-WEIGHTED ROUTING: Currently, OEMN finds the shortest path (fewest hops). But 10 hops of low-latency LAN is better than 3 hops of high-latency satellite. Adding EWMA-based RTT measurement per edge is next.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;DELTA TOPOLOGY UPDATES: Right now, nodes send full topology advertisements. With 100+ nodes, that's bandwidth-heavy. Delta updates (send only changes) could cut bandwidth by 10x.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;FUZZING: I want to throw 10,000 random packet corruptions at the protocol and watch it survive. libFuzzer + AddressSanitizer.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;WHY THIS MATTERS&lt;/p&gt;

&lt;p&gt;Most software engineers will never think about resilience. You build on Kubernetes, AWS, and 99.99% uptime guarantees.&lt;/p&gt;

&lt;p&gt;But when infrastructure fails, when the easy assumptions break, mesh networks become critical infrastructure.&lt;/p&gt;

&lt;p&gt;Understanding how to build one teaches you:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;How to program without central authority (coordination, consensus)&lt;/li&gt;
&lt;li&gt;How to handle adversarial networks (encryption, replay protection)&lt;/li&gt;
&lt;li&gt;How to design concurrent systems that don't have race conditions&lt;/li&gt;
&lt;li&gt;How to measure and optimize for real-world constraints (packet loss, node failures)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;OEMN isn't just a project. It's a masterclass in systems engineering.&lt;/p&gt;

&lt;p&gt;But more importantly: it works. No dependencies. No cloud. Just peers talking to peers.&lt;/p&gt;

&lt;p&gt;TRY IT&lt;/p&gt;

&lt;p&gt;OEMN is on GitHub: github.com/bharqav/oemn&lt;/p&gt;

&lt;p&gt;To run the demo locally:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;make
./oemn --id 1 --port 7777
./oemn --id 2 --port 7778
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Then connect them (in the CLI):&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# on node 1: peer add 2 127.0.0.1 7778
# on node 2: peer add 1 127.0.0.1 7777
# on node 1: send 2 hello
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Watch the message travel. Watch the ACK come back. Watch the routing table update in real-time.&lt;/p&gt;

&lt;p&gt;The code is production-grade:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;48-slot buffer pool&lt;/li&gt;
&lt;li&gt;Thread-safe by design&lt;/li&gt;
&lt;li&gt;Dijkstra shortest-path routing&lt;/li&gt;
&lt;li&gt;AES-256-GCM encryption&lt;/li&gt;
&lt;li&gt;Per-source replay protection&lt;/li&gt;
&lt;li&gt;Graceful shutdown&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Read the docs in the repo. They're detailed. Run the benchmarks yourself. See what performance you get on your hardware.&lt;/p&gt;




&lt;p&gt;If you're curious about how systems really work, build your own. Build a mesh network. Build an autograd engine. Build a shell.&lt;/p&gt;

&lt;p&gt;Don't just use other people's abstractions. Understand the layer beneath.&lt;/p&gt;

&lt;p&gt;That's where the real learning happens.&lt;/p&gt;




&lt;p&gt;OEMN is open-source (MIT License). Contributions welcome: github.com/bharqav/oemn&lt;/p&gt;

</description>
      <category>hardware</category>
      <category>iot</category>
      <category>networking</category>
    </item>
  </channel>
</rss>
