<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Edip</title>
    <description>The latest articles on DEV Community by Edip (@edipcmhash).</description>
    <link>https://dev.to/edipcmhash</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254507%2F127cc852-6557-4650-9606-60c2752028d8.jpg</url>
      <title>DEV Community: Edip</title>
      <link>https://dev.to/edipcmhash</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/edipcmhash"/>
    <language>en</language>
    <item>
      <title>A language model with zero dependencies - and three bugs that proved it worked</title>
      <dc:creator>Edip</dc:creator>
      <pubDate>Thu, 19 Nov 2026 21:30:03 +0000</pubDate>
      <link>https://dev.to/edipcmhash/a-language-model-with-zero-dependencies-and-three-bugs-that-proved-it-worked-141</link>
      <guid>https://dev.to/edipcmhash/a-language-model-with-zero-dependencies-and-three-bugs-that-proved-it-worked-141</guid>
      <description>&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;$ cargo tree
stonefish_ai v1.0.0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the whole dependency tree. No PyTorch, no &lt;code&gt;candle&lt;/code&gt;, no &lt;code&gt;cudarc&lt;/code&gt; — I wrote&lt;br&gt;
the autograd, the byte-level BPE tokenizer, the transformer and the CUDA kernels by&lt;br&gt;
hand. The GPU driver, cuBLAS and NVRTC are loaded at runtime through the operating&lt;br&gt;
system's own loader, so even the GPU path has no crates.&lt;/p&gt;

&lt;p&gt;It works: 222,331,904 parameters, 2.05 billion tokens, held-out bits-per-character&lt;br&gt;
of &lt;strong&gt;0.6064&lt;/strong&gt; against a 2.909 random baseline.&lt;/p&gt;

&lt;p&gt;But the interesting part was the bugs, because &lt;strong&gt;none of them crashed&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A web handler that throws gets fixed in an hour. A gradient that is subtly wrong&lt;br&gt;
produces a slightly worse model — and a slightly worse model looks exactly like&lt;br&gt;
"the model is just small".&lt;/p&gt;

&lt;p&gt;Here are three.&lt;/p&gt;


&lt;h2&gt;
  
  
  1. The gradient that looked reasonable
&lt;/h2&gt;

&lt;p&gt;The attention backward pass used &lt;code&gt;q&lt;/code&gt; where it needed &lt;code&gt;dOut&lt;/code&gt;. It produced a&lt;br&gt;
correctly-shaped matrix of the right magnitude, and it was wrong.&lt;/p&gt;

&lt;p&gt;The loss did not move for days. I blamed the learning rate, the initialisation,&lt;br&gt;
the data.&lt;/p&gt;

&lt;p&gt;The fix took one line. Finding it took a numerical gradient check: compare the&lt;br&gt;
analytic gradient against a central-difference estimate of the same gradient, and&lt;br&gt;
fail if the relative error is above a threshold. I now have 32 of those tests. The&lt;br&gt;
failure mode I was accepting — "close enough" — is exactly what the test forbids.&lt;/p&gt;
&lt;h2&gt;
  
  
  2. &lt;code&gt;beta = 1.0&lt;/code&gt; when I meant "overwrite"
&lt;/h2&gt;

&lt;p&gt;cuBLAS computes &lt;code&gt;C = alpha·A·B + beta·C&lt;/code&gt;. I passed &lt;code&gt;beta = 1.0&lt;/code&gt;, which means "add&lt;br&gt;
to whatever was in C". Since C was a reused buffer, every step added to the&lt;br&gt;
previous step's output.&lt;/p&gt;

&lt;p&gt;The numbers were not garbage. They were plausible and drifting. A four-line test —&lt;br&gt;
run the same input twice, compare — found it immediately.&lt;/p&gt;
&lt;h2&gt;
  
  
  3. The profiler that reported 0 ms
&lt;/h2&gt;

&lt;p&gt;My profiler showed 0 ms for the two biggest line items. That is not speed, that is&lt;br&gt;
a broken hook attached to a code path that had been refactored away.&lt;/p&gt;

&lt;p&gt;The lesson: &lt;strong&gt;a profiler that reports zero is broken, not fast.&lt;/strong&gt;&lt;/p&gt;


&lt;h2&gt;
  
  
  What actually helped
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Read the output, not the status.&lt;/strong&gt; A green build means it compiled.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Compare against a second implementation.&lt;/strong&gt; I rebuilt the same architecture in&lt;br&gt;
PyTorch and compared the loss on the same weights and batches. They agree to&lt;br&gt;
1.8e-05. Without that number, "my model is fine" is a belief.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Write the failures down.&lt;/strong&gt; The full list is 23 items, and it is longer than the&lt;br&gt;
feature list.&lt;/p&gt;



&lt;p&gt;Because there are no dependencies, the whole thing is &lt;strong&gt;839 KB of code&lt;/strong&gt; — against&lt;br&gt;
4.11 GB for a PyTorch install. That is the trade I was after: not speed, not&lt;br&gt;
quality, but being able to read every line that makes it run.&lt;/p&gt;

&lt;p&gt;The code: &lt;a href="https://github.com/Edipcm-hash/stonefish" rel="noopener noreferrer"&gt;https://github.com/Edipcm-hash/stonefish&lt;/a&gt;&lt;br&gt;
The weights: &lt;a href="https://huggingface.co/StonefishAI/stonefish-222m" rel="noopener noreferrer"&gt;https://huggingface.co/StonefishAI/stonefish-222m&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cargo run --release -- demo 90
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Trains a small model from scratch and writes Shakespeare in 90 seconds.&lt;/p&gt;

</description>
      <category>rust</category>
      <category>debugging</category>
      <category>machinelearning</category>
    </item>
  </channel>
</rss>
