<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Logit Solutions AI</title>
    <description>The latest articles on DEV Community by Logit Solutions AI (logitsolutionsai).</description>
    <link>https://dev.to/logitsolutionsai</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Forganization%2Fprofile_image%2F15583%2Ff8b421eb-e1a4-4eaa-a081-400de38046bb.png</url>
      <title>DEV Community: Logit Solutions AI</title>
      <link>https://dev.to/logitsolutionsai</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/logitsolutionsai"/>
    <language>en</language>
    <item>
      <title>Clone yourself as a character, then let Claude Code make the explainer videos</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Tue, 17 Nov 2026 05:53:58 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/clone-yourself-as-a-character-then-let-claude-code-make-the-explainer-videos-g1e</link>
      <guid>https://dev.to/logitsolutionsai/clone-yourself-as-a-character-then-let-claude-code-make-the-explainer-videos-g1e</guid>
      <description>&lt;p&gt;The host in this clip was drawn from one photo.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxli4o3hwrxaqtdjnp02l.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxli4o3hwrxaqtdjnp02l.gif" alt="A photo of the Mona Lisa, an arrow, and the cartoon host drawn from it, waving and saying: I was drawn from one photo, and now I can host your explainer." width="640" height="360"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It blinks, waves, points and moves its lips to the words, and it can present an explainer on any topic you ask Claude for. The photo here is the Mona Lisa, a public domain painting, standing in for yours.&lt;/p&gt;

&lt;p&gt;We built this as &lt;strong&gt;selfclone&lt;/strong&gt;, a free, open-source Claude Code skill. Version 0.2.0 is out now. Here is what it does, how it works and where it falls short.&lt;/p&gt;

&lt;h2&gt;
  
  
  One photo in, a host out
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;selfclone character --photo me.jpg --consent &lt;span class="s2"&gt;"This is my own face. Your Name"&lt;/span&gt; --name &lt;span class="s2"&gt;"Your Name"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The skill finds the landmarks of your face with MediaPipe, takes the proportions from them and samples the colours from the photo. Out comes a drawn SVG character: a rig with jointed arms, a face with six moods, and mouth shapes for the sounds it speaks. You can pick the hair (twelve styles), facial hair and outfit. It also writes a preview sheet like this one:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frlhlkw5uw940sgnwljhr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frlhlkw5uw940sgnwljhr.png" alt="The cartoon character drawn from the Mona Lisa in eight poses: rest, smile, open mouth, rounded mouth, serious, pointing, waving and explaining." width="800" height="507"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It is a cartoon on purpose, not a deepfake of you. It refuses to run without a consent statement, and it stores that statement inside the character file.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then ask for a video
&lt;/h2&gt;

&lt;p&gt;Ask Claude for "an explainer about compound interest, hosted by my character". Claude checks the facts, writes short sentences with one idea per scene, and lays the video out as a scene file. This is the first scene of one of our examples:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"How does money grow?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"cast"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"host"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"character"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"random:money-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"voice"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"kokoro:af_sarah"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"music"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ambient"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"scenes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"say"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s2"&gt;"Put one thousand dollars away at seven percent a year."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"After one year, you have one thousand and seventy."&lt;/span&gt;
      &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"background"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"#8fc6a0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"backdrop"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"dots"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Compound interest"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"character"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"x"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"y"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"scale"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.85&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"enter"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"rise"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"items"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"prop"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"coin"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"x"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.58&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"y"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.45&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"loop"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"bob"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"${n}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"from"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"to"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1070&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"dur"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.6&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"x"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.78&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"y"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.45&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"size"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"line"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before rendering, &lt;code&gt;selfclone explainer scenes.json --preview&lt;/code&gt; writes three stills per scene and prints storyboard warnings: items that overlap, run off the frame, sit under the captions or cover a character, and scenes that are empty or rushed. Claude fixes the file, then renders:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;selfclone explainer scenes.json --character .selfclone/character.json
selfclone explainer scenes.json --aspect 9:16    &lt;span class="c"&gt;# vertical, for Shorts, Reels and TikTok&lt;/span&gt;
selfclone explainer scenes.json --html           &lt;span class="c"&gt;# an interactive player with chapters and captions&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each render writes the MP4, an &lt;code&gt;.srt&lt;/code&gt; caption file and a provenance file that records how the video was made.&lt;/p&gt;

&lt;h2&gt;
  
  
  What happens during a render
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Voice.&lt;/strong&gt; Kokoro, an 82M-parameter text-to-speech model, runs on your CPU. It is Apache-2.0, pinned and hash-checked. ElevenLabs, OpenAI or your own recording also work.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mouth.&lt;/strong&gt; The lip sync is read from the text: closed lips on M, B and P, teeth on the lip for F and V, rounded O and U, timed to the voice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Body.&lt;/strong&gt; Seven gestures (point, wave, explain, present, shrug, think, rest), chosen from the words when the script gives no cues: a wave on "hello", a shrug at a question.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scene.&lt;/strong&gt; 60 hand-drawn props, bar, line and pie charts that grow, handwriting that writes itself, and Manim for maths.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Frames.&lt;/strong&gt; Headless Chrome draws every frame, with the network blocked, across parallel workers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There is no GPU and no image model in the loop. One of the examples below, a 37-second explainer about the seasons, rendered its 900 frames in 27 seconds on six workers, on a laptop.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ten explainers, ten fields
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb6o7dqgddt75wydunxdf.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb6o7dqgddt75wydunxdf.gif" alt="A showreel of ten animated explainers made with selfclone: neural networks, how planes fly, compound interest, vaccines, the Pythagorean theorem, who or whom, neurons, the Ship of Theseus, why the sky is blue and the seasons." width="704" height="396"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Every one of these came out of the same skill, with a different cast each time: one host, two characters in dialogue, or nobody on screen and a narrator. They use random fictional people (&lt;code&gt;random:&amp;lt;seed&amp;gt;&lt;/code&gt;), so no real face appears. The topics are neural networks, how planes fly, compound interest, vaccines, the Pythagorean theorem, who or whom, neurons, the Ship of Theseus, why the sky is blue (as a vertical short) and the seasons.&lt;/p&gt;

&lt;p&gt;The scene files are in the repo under &lt;code&gt;examples/showcase/&lt;/code&gt;, so you can read exactly how each one was written. The &lt;a href="https://github.com/logitsolutionsai/selfclone/releases/tag/v0.2.0" rel="noopener noreferrer"&gt;v0.2.0 release&lt;/a&gt; has the showreel with sound.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it is not
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Not hand animation.&lt;/strong&gt; Bodies move in 2D, a head turn is a shift of the face, and walking is a simple cycle.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not photoreal.&lt;/strong&gt; It does not try to be.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not a fact checker.&lt;/strong&gt; A video is only as good as its scene file. Read the script before you render; the preview stills are there to catch layout mistakes.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  It also clones how you write
&lt;/h2&gt;

&lt;p&gt;The other half of selfclone is a writing clone. Give it your own chat exports and it builds a persona skill that replies the way you write. Then it scores the clone on replies it never saw, by counting style features instead of asking a model to grade itself. On the fictional example in the repo, the clone scores an 89% style match, against 41% for a bare model. The &lt;a href="https://github.com/logitsolutionsai/selfclone" rel="noopener noreferrer"&gt;README&lt;/a&gt; has the method and the limits.&lt;/p&gt;

&lt;h2&gt;
  
  
  Consent and labels
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A real face needs its owner's consent, and the consent record travels inside the character file.&lt;/li&gt;
&lt;li&gt;The photo avatar, selfclone's other face feature, animates the photo itself, so it always carries an "AI CLONE" label that cannot be switched off.&lt;/li&gt;
&lt;li&gt;Explainer videos carry their provenance in the file's metadata and in a &lt;code&gt;.provenance.json&lt;/code&gt; next to the video.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;In Claude Code:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/plugin marketplace add logitsolutionsai/selfclone
/plugin install selfclone@selfclone
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then ask: "make me a character from me.jpg and an explainer video about how vaccines work". You need Node 20 or newer and ffmpeg. Manim is optional, for maths. The skill also works in Codex and Cursor.&lt;/p&gt;

&lt;p&gt;It is MIT-licensed and on GitHub: &lt;a href="https://github.com/logitsolutionsai/selfclone" rel="noopener noreferrer"&gt;logitsolutionsai/selfclone&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If you had a cartoon version of yourself, what would you have it explain first?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>claude</category>
      <category>showdev</category>
    </item>
    <item>
      <title>Overfitting, visually: zero training errors, and accuracy fell from 98.6% to 85%</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Mon, 16 Nov 2026 14:14:32 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/overfitting-visually-zero-training-errors-and-accuracy-fell-from-986-to-85-14c5</link>
      <guid>https://dev.to/logitsolutionsai/overfitting-visually-zero-training-errors-and-accuracy-fell-from-986-to-85-14c5</guid>
      <description>&lt;p&gt;&lt;em&gt;Deep Learning, Visually #21&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watch the Short:&lt;/strong&gt; &lt;a href="https://youtube.com/shorts/K_XwgYYtXSM" rel="noopener noreferrer"&gt;youtube.com/shorts/K_XwgYYtXSM&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A network that gets every training example right sounds like a success. It can be the opposite. In this article we train a real network, stop it at two moments, and compare them on data it has never seen.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The data is two classes shaped like crescents: 110 training points on a plane. We flip the labels of ten of them on purpose. Those ten are noise, the kind of error any real dataset has.&lt;/p&gt;

&lt;p&gt;To judge the network, we keep 24 new points aside. It never trains on them. Their labels are clean.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fojrr8ap4kwq0o5hvrncy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fojrr8ap4kwq0o5hvrncy.png" alt="The training data: filled and hollow dots are the two classes, and the ten rings mark the wrong labels." width="515" height="343"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  One neuron is too simple
&lt;/h2&gt;

&lt;p&gt;One neuron can only draw a straight line. Trained for 3,000 steps, it ends with 19 training errors, and it gets 3 of the 24 new points wrong. The crescents curl around each other, so no straight line can separate them.&lt;/p&gt;

&lt;p&gt;That model isn't memorizing anything. It just can't follow the shape.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1bqkpugnn28k6dyx87k9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1bqkpugnn28k6dyx87k9.png" alt="One neuron: a straight line, 19 training errors, 3 new points wrong." width="516" height="590"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  A big network learns the real shape first
&lt;/h2&gt;

&lt;p&gt;Now take a network with three hidden layers of 128 neurons each. That is 33,152 weights to fit 110 points, far more than it needs. We train it with Adam for 2,500 steps, and we look at it along the way.&lt;/p&gt;

&lt;p&gt;By step 70, its line follows the two crescents. It has 9 training errors, and all 9 are wrong labels. It gets all 24 new points right. On a larger check, 5,000 more new points, its accuracy is 98.6%. This is as good as this run ever gets.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwqdisycrpag97t7xjko4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwqdisycrpag97t7xjko4.png" alt="Step 70: the line follows the crescents. The nine training errors are all wrong labels, and no new point is wrong." width="516" height="653"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep training, and it memorizes the noise
&lt;/h2&gt;

&lt;p&gt;Nothing stops the training at step 70, so gradient descent keeps working on those last nine points. To get them right, the line has to curl around each one.&lt;/p&gt;

&lt;p&gt;By step 445 there are no training errors left: a perfect training score. By step 628, 5 of the 24 new points are wrong. At step 2,500, the same 5,000 points score 85.0%.&lt;/p&gt;

&lt;p&gt;The network didn't forget the crescents. It added a detail for every wrong label, and those details are wrong everywhere else. This is overfitting: the network has memorized the training data, noise included, and does worse on new data because of it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fess14svqvp7n83lki3xq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fess14svqvp7n83lki3xq.png" alt="Step 2,500: no training errors, and five new points wrong. The curve snakes around every wrong label." width="516" height="638"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Seeing where it fails
&lt;/h2&gt;

&lt;p&gt;The 5,000 points show this directly. In the next picture, the white dots are the ones the network gets wrong. At step 70 there are about 70 of them, close to the line. At step 2,500 there are about 750, in whole patches around the detours it took for the wrong labels.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F69xud57gfaq8xxftknzd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F69xud57gfaq8xxftknzd.png" alt="The 5,000 new points at step 2,500: the white ones are wrong, 15% of them." width="516" height="647"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A note on the 24 points. Any sample that small is noisy, and we picked one whose counts sit near each model's true error rate, so the picture reads clearly. The 5,000 points are the honest measure.&lt;/p&gt;

&lt;h2&gt;
  
  
  Real networks do it too
&lt;/h2&gt;

&lt;p&gt;The crescents are a toy, so here is a real run. Take 1,000 images of handwritten digits and a network with 26,432 weights, about 26 weights per image. Train it for 200 epochs with no protection against overfitting, and measure it on 10,000 images it never trains on.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Training accuracy&lt;/strong&gt; reaches 100% at epoch 26 and stays there. Training loss falls to 0.0002.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Accuracy on new images&lt;/strong&gt; reaches about 88% and stalls.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Loss on new images&lt;/strong&gt; hits its lowest point, 0.42, at epoch 14, and then climbs to 0.81 while the training loss keeps falling.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The two loss curves tell you when it starts: from epoch 14 on, more training makes the training loss smaller and the loss on new images bigger.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo908nmkw8ir9gagib5z0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo908nmkw8ir9gagib5z0.png" alt="Run on 1,000 digit images: the training loss falls to almost zero while the loss on new images stops falling and climbs." width="502" height="638"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What to take from this
&lt;/h2&gt;

&lt;p&gt;A training score only tells you how well the network fits data it has already seen. With enough weights, it can get that score to perfect by memorizing, wrong labels included. So a perfect training score proves nothing on its own. Only new data can tell.&lt;/p&gt;

&lt;p&gt;That raises the practical questions: how to measure on new data honestly, and how to stop a network from memorizing. Those are the next few topics in this series.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;The same effect fits in six lines of NumPy. Eight noisy points lie along the line y = x, and a degree-7 polynomial has just enough room to pass through all of them, so its training error is zero.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;default_rng&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;linspace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;normal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;poly1d&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;polyfit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;f&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;95&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It prints &lt;code&gt;0.48&lt;/code&gt;. At x = 0.95, the real pattern says about 0.95. The curve fit every point it was given and missed the one it wasn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  Key takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Overfitting means a network does better and better on its training data while doing worse on new data.&lt;/li&gt;
&lt;li&gt;It happens when a network has far more weights than it needs, and it gets worse with longer training.&lt;/li&gt;
&lt;li&gt;The network often learns the real pattern first, then memorizes the noise.&lt;/li&gt;
&lt;li&gt;A perfect training score can come with a worse network: here, 98.6% on new points fell to 85.0% as training errors fell to zero.&lt;/li&gt;
&lt;li&gt;Always measure on data the network never trains on.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;CS878 Deep Learning (Fall 2023), Lecture 12–13: Regularization &amp;amp; Transfer Learning.&lt;/li&gt;
&lt;li&gt;Logit Solutions AI, Neural Networks From Scratch (full lecture), Chapter 14: Generalization. &lt;a href="https://youtu.be/uYVk4iFb4uQ" rel="noopener noreferrer"&gt;https://youtu.be/uYVk4iFb4uQ&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>deeplearning</category>
      <category>machinelearning</category>
      <category>ai</category>
      <category>python</category>
    </item>
    <item>
      <title>Google says preorder, OpenAI says pre_order: product feeds for AI shopping</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Sun, 15 Nov 2026 08:39:41 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/google-says-preorder-openai-says-preorder-product-feeds-for-ai-shopping-3g42</link>
      <guid>https://dev.to/logitsolutionsai/google-says-preorder-openai-says-preorder-product-feeds-for-ai-shopping-3g42</guid>
      <description>&lt;p&gt;Take a product you can order before it ships. Google Merchant Center wants its availability written as &lt;code&gt;preorder&lt;/code&gt;. The OpenAI product feed, the one ChatGPT shopping reads, wants &lt;code&gt;pre_order&lt;/code&gt;, and its spec is blunt about anything else: "Omitted, empty, or unrecognized values reject the row."&lt;/p&gt;

&lt;p&gt;Same product, same store, one underscore. One feed is accepted and the other silently loses the row.&lt;/p&gt;

&lt;p&gt;That is the new chore in e-commerce. AI shopping assistants read your catalog through product feeds, and each one has its own format:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the &lt;strong&gt;OpenAI product feed&lt;/strong&gt;: nine required fields in basic product data, plus &lt;code&gt;target_countries&lt;/code&gt; and &lt;code&gt;store_country&lt;/code&gt;;&lt;/li&gt;
&lt;li&gt;the &lt;strong&gt;Agentic Commerce Protocol (ACP)&lt;/strong&gt; feed schema, version 2026-04-17, still in beta;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google Merchant Center&lt;/strong&gt; product data, including its conversational attributes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;They mostly agree, which is exactly why the differences slip through.&lt;/p&gt;

&lt;h2&gt;
  
  
  Small differences, real rejections
&lt;/h2&gt;

&lt;p&gt;A few we ran into while reading the specs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Availability spelling.&lt;/strong&gt; &lt;code&gt;preorder&lt;/code&gt; for Google, &lt;code&gt;pre_order&lt;/code&gt; for OpenAI.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Barcodes.&lt;/strong&gt; OpenAI asks for a GTIN with "no spaces or dashes". Google accepts a barcode written with spaces and reads it without them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Required fields.&lt;/strong&gt; &lt;code&gt;target_countries&lt;/code&gt; is required for the OpenAI feed. Lists that say "nine required fields" leave it out, along with &lt;code&gt;store_country&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check digits.&lt;/strong&gt; A GTIN with a wrong final digit fails everywhere, and nobody notices until a channel rejects it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this is hard. It is just a lot of small rules spread over several spec pages, all changing at their own pace.&lt;/p&gt;

&lt;h2&gt;
  
  
  So we built a checker
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/logitsolutionsai/shelfcheck" rel="noopener noreferrer"&gt;shelfcheck&lt;/a&gt; is a free, open-source command-line tool written in Rust. You point it at your feed and it checks every product against all three formats, plus GS1 barcode rules, in one run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ shelfcheck check fixtures/store-a/products.csv
shelfcheck 0.1.0 · rules OpenAI 2026-11-14 · ACP 2026-04-17 · Google 2026-11-14 · GS1 26.0

read      OpenAI CSV · 1,400 products, 3,912 variants                     0.30 s

          ready            errors   warnings
OpenAI    1,228    87.7%   172      27
ACP       1,310    93.6%   90       0
Google    1,293    92.4%   107      0

✗   82  Barcodes with a wrong check digit       OpenAI, ACP, Google · e.g. PHO0071-3, line 157
✗   61  target_countries is missing             OpenAI · e.g. LAM0015-3, line 36
✗   23  availability values the format rejects  OpenAI · e.g. MUG0033-1, line 79
    + 4 more fixes, 43 products. All of them are in the report.

182 of 1,400 products need fixes before every format accepts them.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That run is on the synthetic 1,400-product demo store that ships with the repo, so the numbers show what the tool does, not how real stores score.&lt;/p&gt;

&lt;p&gt;The fixes come ranked by how many products each one affects, so the first fix you make is the one that clears the most rows. Every problem names the product, the line and the column, and cites the rule it broke.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftl03vpmfbdtdrxbo75qa.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftl03vpmfbdtdrxbo75qa.png" alt="The shelfcheck report for the demo feed: 182 of 1,400 products need fixes, readiness for the OpenAI product feed, the ACP feed schema and Google Merchant Center, and the fixes ranked by products affected" width="800" height="1000"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The second problem: feeds change
&lt;/h2&gt;

&lt;p&gt;A feed that passed last month can break this month without a single error. So shelfcheck also compares two feeds and flags risky changes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;an id reused for a different product (in the demo, &lt;code&gt;PHN-X2-BLK&lt;/code&gt; was a phone and is now a phone case);&lt;/li&gt;
&lt;li&gt;the same product under a new id;&lt;/li&gt;
&lt;li&gt;a variant whose storage, size, colour, material or pack quantity changed under the same id and barcode;&lt;/li&gt;
&lt;li&gt;store titles that still describe the old variant, like a tablet that is now 256 GB while the title still says 128 GB.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You record a decision for each change once (&lt;code&gt;shelfcheck decide&lt;/code&gt;), and it stays quiet after that.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fglcuk9158cmbnsrlqzzc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fglcuk9158cmbnsrlqzzc.png" alt="The changes section of the shelfcheck diff report: an id reused for a different product, with old and new values side by side" width="800" height="525"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How it works
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rules are code, not a language model.&lt;/strong&gt; The same feed always gets the same result.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every rule cites its source&lt;/strong&gt; and the date it was last checked against that page, because these specs change often.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;It reads CSV, TSV, JSONL, Google XML, ACP feeds and Shopify product exports.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It fits into CI.&lt;/strong&gt; Each run writes an offline &lt;code&gt;report.html&lt;/code&gt;, &lt;code&gt;results.json&lt;/code&gt;, SARIF 2.1.0 for GitHub code scanning, &lt;code&gt;junit.xml&lt;/code&gt; and a &lt;code&gt;failing-products.csv&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It never touches your store&lt;/strong&gt; and never connects to the internet. Links in a feed are checked for form, never fetched.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is fast.&lt;/strong&gt; On a Windows laptop (Intel Core i7-1280P), one million generated rows were checked against all four rule packs in about 8.3 seconds (warm runs; the first, cold run took 12.3).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One honest limit: passing shelfcheck means your feed matches the formats it checks. It does not mean an AI assistant will show your products.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx shelfcheck check products.csv                 &lt;span class="c"&gt;# no install&lt;/span&gt;
brew install logitsolutionsai/tap/shelfcheck      &lt;span class="c"&gt;# Homebrew&lt;/span&gt;
cargo install shelfcheck                          &lt;span class="c"&gt;# crates.io&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The code, the rule packs and their sources are on GitHub under Apache-2.0: &lt;a href="https://github.com/logitsolutionsai/shelfcheck" rel="noopener noreferrer"&gt;github.com/logitsolutionsai/shelfcheck&lt;/a&gt;. If a rule is out of date, open an issue with the rule id and a link to the page that changed.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Sources&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;OpenAI product feed spec (file upload): &lt;a href="https://developers.openai.com/commerce/specs/file-upload/products" rel="noopener noreferrer"&gt;https://developers.openai.com/commerce/specs/file-upload/products&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Agentic Commerce Protocol: &lt;a href="https://github.com/agentic-commerce-protocol/agentic-commerce-protocol" rel="noopener noreferrer"&gt;https://github.com/agentic-commerce-protocol/agentic-commerce-protocol&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Google Merchant Center product data specification: &lt;a href="https://support.google.com/merchants/answer/7052112" rel="noopener noreferrer"&gt;https://support.google.com/merchants/answer/7052112&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Google Merchant Center, availability attribute: &lt;a href="https://support.google.com/merchants/answer/6324448" rel="noopener noreferrer"&gt;https://support.google.com/merchants/answer/6324448&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GS1 General Specifications: &lt;a href="https://www.gs1.org/standards/barcodes-epcrfid-id-keys/gs1-general-specifications" rel="noopener noreferrer"&gt;https://www.gs1.org/standards/barcodes-epcrfid-id-keys/gs1-general-specifications&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Logit Solutions AI is an AI product studio. We design, build and support production-ready AI products for startups and growing businesses.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ecommerce</category>
      <category>ai</category>
      <category>rust</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Karpathy's free ChatGPT video has 10M views. Here's why he gives it all away</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Sun, 15 Nov 2026 04:20:10 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/karpathys-free-chatgpt-video-has-10m-views-heres-why-he-gives-it-all-away-55kb</link>
      <guid>https://dev.to/logitsolutionsai/karpathys-free-chatgpt-video-has-10m-views-heres-why-he-gives-it-all-away-55kb</guid>
      <description>&lt;p&gt;In February 2025, Andrej Karpathy uploaded a three-and-a-half-hour video called "Deep Dive into LLMs like ChatGPT". It explains, from the ground up, how a chatbot like ChatGPT is built. It has now been watched more than 10 million times. It costs nothing.&lt;/p&gt;

&lt;p&gt;It isn't a one-off. His "Let's build GPT" lecture has over 8 million views, and his code for training small GPT models has tens of thousands of stars on GitHub. All of it is free.&lt;/p&gt;

&lt;p&gt;So why does one of the most sought-after AI researchers alive keep giving his best work away? He has answered that himself. The answer is less about generosity than about a fear he has for all of us.&lt;/p&gt;

&lt;h2&gt;
  
  
  Who he is
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Born in Bratislava&lt;/strong&gt;, then part of Czechoslovakia. In his words: "my family moved to Toronto when I was 15."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;University of Toronto:&lt;/strong&gt; "This is where I first got into deep learning, attending Geoff Hinton's class and reading groups."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stanford PhD&lt;/strong&gt; with Fei-Fei Li. There he "designed and was the primary instructor for the first deep learning class Stanford - CS 231n". It grew from 150 students in 2015 to 750 in 2017.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;December 2015:&lt;/strong&gt; named as one of OpenAI's founding members in its launch post.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;2017 to 2022:&lt;/strong&gt; Director of AI at Tesla, leading Autopilot's computer vision. When he left, he wrote that Autopilot had "graduated from lane keeping to city streets" in those five years.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;February 2023:&lt;/strong&gt; back at OpenAI ("I am joining OpenAI (again :))"). He left again in February 2024.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;July 2024:&lt;/strong&gt; started Eureka Labs, "an AI+Education company".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;February 2025:&lt;/strong&gt; coined "vibe coding". Collins Dictionary made it its Word of the Year 2025.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;May 2026:&lt;/strong&gt; joined Anthropic to work on pretraining research.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;His second spell at OpenAI included its strangest week. In November 2023 the board fired Sam Altman. Two days later, Karpathy posted:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"I like and respect Sam and I think so does the majority of OpenAI. The board had a chance to explain their drastic actions and they did not take it, so there is nothing to go on except exactly what it looks like."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Within days, Altman was back as CEO.&lt;/p&gt;

&lt;h2&gt;
  
  
  The free teacher
&lt;/h2&gt;

&lt;p&gt;Karpathy's YouTube channel has 1.76 million subscribers. His "Neural Networks: Zero to Hero" series is eight lectures. It starts with "building micrograd" (4.5 million views), a tiny engine that computes gradients, and works up to building a GPT yourself.&lt;/p&gt;

&lt;p&gt;His code follows the same idea: small enough to read in one sitting. Stars as of November 13, 2026:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Repo&lt;/th&gt;
&lt;th&gt;What it is&lt;/th&gt;
&lt;th&gt;Stars&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;nanoGPT&lt;/td&gt;
&lt;td&gt;The simplest code to train a mid-sized GPT&lt;/td&gt;
&lt;td&gt;64,700&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;nanochat&lt;/td&gt;
&lt;td&gt;A "little ChatGPT clone that you can kind of talk to", trained for about $100&lt;/td&gt;
&lt;td&gt;60,500&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;autoresearch&lt;/td&gt;
&lt;td&gt;AI agents that run training experiments overnight&lt;/td&gt;
&lt;td&gt;100,000&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In February 2026 he went further: microgpt, "a single file of 200 lines of pure Python with no dependencies that trains and inferences a GPT". His note on it: "I cannot simplify this any further."&lt;/p&gt;

&lt;h2&gt;
  
  
  He's not a cheerleader
&lt;/h2&gt;

&lt;p&gt;The person teaching millions how these models work is also one of the most quoted sober voices on what they can't do. Dates matter here, because his view has moved.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;October 2025&lt;/strong&gt;, on the Dwarkesh Podcast, asked why you wouldn't hire an AI agent as an intern today:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"They're cognitively lacking and it's just not working."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;His forecast: "this is more accurately described as the decade of agents." Not the year. On how the industry was presenting coding agents, he was blunter:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it's not. It's slop."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In the same answer: "The models are amazing. They still need a lot of work."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;January 2026&lt;/strong&gt;, after weeks of coding with agents, he described going "from about 80% manual+autocomplete coding and 20% agents in November to 80% agent coding and 20% edits+touchups in December". Then he admitted something few people at his level would:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"I've already noticed that I am slowly starting to atrophy my ability to write code manually."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;His advice in the same post: "if you have any code you actually care about I would watch them like a hawk".&lt;/p&gt;

&lt;p&gt;The man who coined "vibe coding" also wrote, in that very first post, that it's "not too bad for throwaway weekend projects". The limit was there from day one.&lt;/p&gt;

&lt;h2&gt;
  
  
  His biggest fear
&lt;/h2&gt;

&lt;p&gt;On the podcast, he was asked why he chose education over AI research at a lab. His answer:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"My personal big fear is that a lot of this stuff happens on the side of humanity, and that humanity gets disempowered by it."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And then, more concretely:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"I'm most afraid of something depicted in movies like WALL-E or Idiocracy or something like that, where humanity is on the side of this stuff."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In WALL-E, people drift through life in hover chairs while machines run everything. In Idiocracy, the future simply got dumber. Karpathy's fear isn't killer robots. It's a world where AI keeps getting better and people stop understanding any of it.&lt;/p&gt;

&lt;p&gt;He doesn't soften it: "If this is false and I'm wrong and we end up in a WALL-E or Idiocracy future, then I don't even care if there are Dyson spheres. This is a terrible outcome."&lt;/p&gt;

&lt;p&gt;So the free lectures aren't charity. They're his plan. People who understand the machines don't get sidelined by them. Even when he joined Anthropic in May 2026, he wrote: "I remain deeply passionate about education and plan to resume my work on it in time."&lt;/p&gt;

&lt;h2&gt;
  
  
  What builders can take from it
&lt;/h2&gt;

&lt;p&gt;We use coding agents every day, so three things from him hit home:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Learn it by building it small.&lt;/strong&gt; micrograd, nanoGPT, microgpt. If you can read the whole thing, you understand it, and you can debug what an agent writes for you.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch the agent on code you care about.&lt;/strong&gt; His words, not ours: "like a hawk".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep some skills manual on purpose.&lt;/strong&gt; If the person who named vibe coding noticed his own skills fading, the rest of us will too.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;In his words, the future he wants:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"I want humans to be much, much better in this future. To me, this is through education that you can achieve this."&lt;/p&gt;
&lt;/blockquote&gt;




&lt;p&gt;We made a short video on his story:&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/F4RYsQHAPkA" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sources&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Andrej Karpathy, "Deep Dive into LLMs like ChatGPT" (10,390,473 views on November 13, 2026): &lt;a href="https://www.youtube.com/watch?v=7xTGNNLPyMI" rel="noopener noreferrer"&gt;https://www.youtube.com/watch?v=7xTGNNLPyMI&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;karpathy.ai (education, CS231n, Hinton's class): &lt;a href="https://karpathy.ai/" rel="noopener noreferrer"&gt;https://karpathy.ai/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Data Science Weekly interview ("moved to Toronto when I was 15"): &lt;a href="https://www.datascienceweekly.org/data-scientist-interviews/training-deep-learning-models-browser-andrej-karpathy-interview" rel="noopener noreferrer"&gt;https://www.datascienceweekly.org/data-scientist-interviews/training-deep-learning-models-browser-andrej-karpathy-interview&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenAI, "Introducing OpenAI", December 11, 2015: &lt;a href="https://openai.com/index/introducing-openai/" rel="noopener noreferrer"&gt;https://openai.com/index/introducing-openai/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;TechCrunch, June 20, 2017 (Director of AI and Autopilot Vision): &lt;a href="https://techcrunch.com/2017/06/20/tesla-hires-deep-learning-expert-andrej-karpathy-to-lead-autopilot-vision/" rel="noopener noreferrer"&gt;https://techcrunch.com/2017/06/20/tesla-hires-deep-learning-expert-andrej-karpathy-to-lead-autopilot-vision/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Karpathy on X, July 13, 2022 (leaving Tesla): &lt;a href="https://x.com/karpathy/status/1547332300186066944" rel="noopener noreferrer"&gt;https://x.com/karpathy/status/1547332300186066944&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Karpathy on X, February 9, 2023 (rejoining OpenAI): &lt;a href="https://x.com/karpathy/status/1623476659369443328" rel="noopener noreferrer"&gt;https://x.com/karpathy/status/1623476659369443328&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Karpathy on X, November 19, 2023 ("I like and respect Sam"): &lt;a href="https://x.com/karpathy/status/1726289070345855126" rel="noopener noreferrer"&gt;https://x.com/karpathy/status/1726289070345855126&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenAI, "Sam Altman returns as CEO, OpenAI has a new initial board", November 29, 2023: &lt;a href="https://openai.com/index/sam-altman-returns-as-ceo-openai-has-a-new-initial-board/" rel="noopener noreferrer"&gt;https://openai.com/index/sam-altman-returns-as-ceo-openai-has-a-new-initial-board/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Karpathy on X, July 16, 2024 (Eureka Labs): &lt;a href="https://x.com/karpathy/status/1813263734707790301" rel="noopener noreferrer"&gt;https://x.com/karpathy/status/1813263734707790301&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Karpathy on X, February 2, 2025 ("vibe coding"): &lt;a href="https://x.com/karpathy/status/1886192184808149383" rel="noopener noreferrer"&gt;https://x.com/karpathy/status/1886192184808149383&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Collins Dictionary, Word of the Year 2025: &lt;a href="https://blog.collinsdictionary.com/language-lovers/collins-word-of-the-year-2025-ai-meets-authenticity-as-society-shifts/" rel="noopener noreferrer"&gt;https://blog.collinsdictionary.com/language-lovers/collins-word-of-the-year-2025-ai-meets-authenticity-as-society-shifts/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Dwarkesh Podcast, October 17, 2025 (transcript): &lt;a href="https://www.dwarkesh.com/p/andrej-karpathy" rel="noopener noreferrer"&gt;https://www.dwarkesh.com/p/andrej-karpathy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Karpathy on X, January 26, 2026 (notes from coding with agents): &lt;a href="https://x.com/karpathy/status/2015883857489522876" rel="noopener noreferrer"&gt;https://x.com/karpathy/status/2015883857489522876&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Karpathy, "microgpt", February 12, 2026: &lt;a href="http://karpathy.github.io/2026/02/12/microgpt/" rel="noopener noreferrer"&gt;http://karpathy.github.io/2026/02/12/microgpt/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GitHub: &lt;a href="https://github.com/karpathy/nanoGPT" rel="noopener noreferrer"&gt;https://github.com/karpathy/nanoGPT&lt;/a&gt;, &lt;a href="https://github.com/karpathy/nanochat" rel="noopener noreferrer"&gt;https://github.com/karpathy/nanochat&lt;/a&gt;, &lt;a href="https://github.com/karpathy/autoresearch" rel="noopener noreferrer"&gt;https://github.com/karpathy/autoresearch&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Karpathy on X, May 19, 2026 (joining Anthropic): &lt;a href="https://x.com/karpathy/status/2056753169888334312" rel="noopener noreferrer"&gt;https://x.com/karpathy/status/2056753169888334312&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Cover photo: Gladwin Analytics, CC BY 3.0, via Wikimedia Commons.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Logit Solutions AI is an AI product studio. We design, build and support production-ready AI products for startups and growing businesses.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>learning</category>
    </item>
    <item>
      <title>Your user said yes. United said no.</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Thu, 12 Nov 2026 11:04:08 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/your-user-said-yes-united-said-no-b4p</link>
      <guid>https://dev.to/logitsolutionsai/your-user-said-yes-united-said-no-b4p</guid>
      <description>&lt;p&gt;Imagine you built the perfect travel agent. It finds the cheapest flight, asks your user to confirm, waits for the "yes", and books it. No hallucinations, no surprise upgrades, a human in the loop at every step.&lt;/p&gt;

&lt;p&gt;As of November 11, United Airlines says that booking is against its rules. And it can cancel it.&lt;/p&gt;

&lt;p&gt;Not because your agent did anything wrong. Because it's an agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually changed
&lt;/h2&gt;

&lt;p&gt;On November 11, 2026, United revised its Contract of Carriage, the legal document every passenger agrees to when they buy a ticket. It now defines an "AI Agent" as:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"any software or service, including but not limited to an AI assistant, AI agent or AI-enabled browser that takes autonomous or semi-autonomous action, whether or not Passenger or another individual supervises or approves its actions…"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Read that middle part again: &lt;strong&gt;whether or not Passenger or another individual supervises or approves its actions.&lt;/strong&gt; Your carefully designed approval step doesn't count.&lt;/p&gt;

&lt;p&gt;The definition then names who it covers: AI assistants built into social media, messaging, search or other consumer platforms; AI-enabled browsers; and standalone AI agents.&lt;/p&gt;

&lt;p&gt;The contract's list of prohibited practices now includes:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"The use of AI Agents to make, modify or service a Ticket, and/or to purchase any service or product."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And the list of bookings United may cancel now ends with "reservations made, modified or serviced by AI Agents."&lt;/p&gt;

&lt;p&gt;The press release adds the scope: United is restricting access "directly within our digital platforms and indirectly via travel agencies and aggregators."&lt;/p&gt;

&lt;h2&gt;
  
  
  United's three reasons
&lt;/h2&gt;

&lt;p&gt;United's announcement gives three:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Sharing account credentials with personal, third-party AI assistants "can expose sensitive information".&lt;/li&gt;
&lt;li&gt;Actions taken by these tools "may result in unintended purchases, account changes, or use of MileagePlus miles".&lt;/li&gt;
&lt;li&gt;Automated actions could violate FAA and DOT rules. Their example: flight check-in "requires human acknowledgment that bags do not contain banned substances like explosives".&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It also says leaders in banking and retail "have taken similar actions". So this is unlikely to be the last "no" your agent hears.&lt;/p&gt;

&lt;h2&gt;
  
  
  The uncomfortable questions
&lt;/h2&gt;

&lt;p&gt;Most of the agent demos we see in 2026 skip these. United just put them in a contract.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Whose permission does your agent actually have?&lt;/strong&gt;&lt;br&gt;
Your user said yes. But the user doesn't own the airline's website, its fares or its checkout. Consent from one party was never authorisation from the other. We just got used to treating the web as if it were.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Is "human in the loop" a real safeguard, or a UX pattern?&lt;/strong&gt;&lt;br&gt;
United's wording says supervision doesn't change anything. That should make every builder ask what their approval step really proves. Did the human read the fare rules? The change fees? Or did they tap "Approve" on a summary the agent wrote?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Are you building on borrowed logins?&lt;/strong&gt;&lt;br&gt;
Many consumer agents work by logging in as the user and driving the website like a person would. That means holding someone's password, session or loyalty account. It's exactly the first risk United names. If your product only works because a user handed over their credentials, you're one policy update away from not working at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Who pays when the agent is wrong?&lt;/strong&gt;&lt;br&gt;
"Unintended purchases" is a polite phrase for a support nightmare: a wrong date, a non-refundable fare, miles spent that shouldn't have been. Today, nobody has clearly agreed who absorbs that cost. One of the biggest airlines just told us it won't be them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Should an agent ever say "I confirm" on a legal declaration?&lt;/strong&gt;&lt;br&gt;
The check-in example is the sharpest one. "These bags contain no banned substances" is a statement a person makes, under rules with real consequences. An agent clicking that box is not a convenience feature. It's a person's declaration made by software.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. What happens when everyone does this?&lt;/strong&gt;&lt;br&gt;
If airlines, banks and retailers all close the side door, agents that depend on scraping and auto-clicking will hit the same wall everywhere. The question isn't whether your agent works today. It's whether it's built on anything the other side agreed to.&lt;/p&gt;
&lt;h2&gt;
  
  
  What we'd build instead
&lt;/h2&gt;

&lt;p&gt;We build AI agents for clients, so this isn't abstract for us. Here's where we'd draw the lines:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Use the seller's door, not the user's keys.&lt;/strong&gt; Official APIs, partner programmes and affiliate deals exist for a reason. They come with terms, rate limits and liability that someone agreed to.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never store a user's password.&lt;/strong&gt; If a service offers scoped tokens or OAuth, use them. If it offers nothing, that's an answer too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Put a human tap before money and before declarations.&lt;/strong&gt; Not "approve this plan", but "you are paying $412, non-refundable, from this card". Let the person make legal statements themselves.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Design for "no".&lt;/strong&gt; When a merchant blocks agents, hand the user a ready-to-finish link instead of fighting the wall. A graceful handoff beats a cancelled booking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Be honest about what you are.&lt;/strong&gt; Agents that identify themselves can be given permissions. Agents that pretend to be people get banned.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  The real question
&lt;/h2&gt;

&lt;p&gt;United didn't ban AI. It banned AI that acts on its systems without its say-so. That's a line many more companies will draw, and it's a fair one to argue about.&lt;/p&gt;

&lt;p&gt;So two questions, one for everyone and one for builders:&lt;/p&gt;

&lt;p&gt;Would you let an AI book your flights?&lt;/p&gt;

&lt;p&gt;And if you're building that AI: &lt;strong&gt;did the seller ever say yes?&lt;/strong&gt;&lt;/p&gt;



&lt;p&gt;We made a 60-second video on this story:&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/YXGWDAjG_T4" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sources&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;United Airlines, "United Restricts Access for Third-Party AI Agents", November 11, 2026: &lt;a href="https://united.mediaroom.com/news-releases?item=125496" rel="noopener noreferrer"&gt;https://united.mediaroom.com/news-releases?item=125496&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;United Contract of Carriage (revised November 11, 2026): &lt;a href="https://www.united.com/en/us/fly/contract-of-carriage.html" rel="noopener noreferrer"&gt;https://www.united.com/en/us/fly/contract-of-carriage.html&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Cover photo: Oleg Yunakov, CC BY-SA 4.0, via Wikimedia Commons.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Logit Solutions AI is an AI product studio. We design, build and support production-ready AI products for startups and growing businesses.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>discuss</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Speech-to-text can't tell you if an AI voice said a name right</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Tue, 10 Nov 2026 23:24:36 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/speech-to-text-cant-tell-you-if-an-ai-voice-said-a-name-right-5bge</link>
      <guid>https://dev.to/logitsolutionsai/speech-to-text-cant-tell-you-if-an-ai-voice-said-a-name-right-5bge</guid>
      <description>&lt;p&gt;We make short, AI-voiced lessons. Before a video ships, every sentence goes through automatic checks, and then a person listens. The automatic checks kept failing in one place: names.&lt;/p&gt;

&lt;p&gt;This post explains why a transcript can't catch a mispronounced name, and the three-part check we built instead. All numbers are our own measurements. The code is open source and linked at the end.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem: a transcript hears what it expects
&lt;/h2&gt;

&lt;p&gt;The usual way to check a voice-over is to transcribe it and compare the transcript with the script. That works for missing or swapped words. For names it fails in both directions.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A wrong read can pass.&lt;/strong&gt; A speech-to-text model is trained to write likely text. When a voice gets a name almost right, the model often writes the correct spelling anyway, and the mistake disappears from the transcript.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A right read can fail.&lt;/strong&gt; When the model doesn't know a name, it spells it some other way, even when the voice said it perfectly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Our old check did exactly this. On 32 test reads of 8 hard names, it passed only 8 of the 22 correct reads. It did fail all 10 wrong reads, but only because it failed almost every name. It couldn't tell a right read from a wrong one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three ears instead of one
&lt;/h2&gt;

&lt;p&gt;Every take now goes through three listeners, each built differently:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Words.&lt;/strong&gt; A speech-to-text model transcribes the take, and we compare the transcript with the script. This catches missing, extra or wrong words.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Letters, with no language model.&lt;/strong&gt; A second recognizer writes what it hears letter by letter, with no idea which words are likely. It catches the slips the first one smooths over, small dropped words ("from zero [to] two hundred") and takes that cut off early.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sounds.&lt;/strong&gt; A third model writes the phones it hears, in IPA. For every hard word we keep a researched pronunciation, also in IPA, and compare the two sound by sound.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Here is one real failing take. The name should sound like &lt;strong&gt;/ʃ ɪ v ɔ n/&lt;/strong&gt;. The voice said &lt;strong&gt;/j ɑ b ə n/&lt;/strong&gt;. Only the third ear sees that difference. The old check gave this take the same verdict as a correct read.&lt;/p&gt;

&lt;h2&gt;
  
  
  Comparing sounds fairly
&lt;/h2&gt;

&lt;p&gt;A plain phone error rate counts every difference the same. That is too blunt: some differences are an accent, others make a different word.&lt;/p&gt;

&lt;p&gt;So the comparison weighs each substitution by phonetic features (close sounds cost less than distant ones) and tolerates accent-level variation. The pass/fail rule is still strict: one clearly wrong sound fails the word. A false alarm costs one re-take; a misread name that ships costs more.&lt;/p&gt;

&lt;h2&gt;
  
  
  The target matters as much as the check
&lt;/h2&gt;

&lt;p&gt;A sound-by-sound check is only as good as the pronunciation it compares against. So each hard word gets a pronunciation with a source, such as an encyclopedia's IPA, a dictionary entry or a native recording. It goes into a small lexicon file, together with the respelling the TTS engine reads.&lt;/p&gt;

&lt;p&gt;The workflow becomes: find the risky words in the script, research each one with a source, write it in the TTS engine's syntax, record, then check every take.&lt;/p&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;On our own takes.&lt;/strong&gt; 8 hard names, each read by 2 voices, once from a researched respelling and once from the raw spelling: 32 reads, 22 right and 10 wrong.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Correct reads that pass: &lt;strong&gt;8 of 22 before, 22 of 22 after&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Wrong reads caught: &lt;strong&gt;10 of 10, before and after&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One honest caveat: these 32 takes are where we set the thresholds, so treat this as a fit, not a test.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;On a held-out benchmark.&lt;/strong&gt; There is no public benchmark for TTS name pronunciation. The closest public test of the core step (did this word sound like its target phones?) is a dataset of 2,500 recordings, 15,967 words, read by non-native English speakers and scored word by word by five experts. We ran the sound check on it with no training on the dataset.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Spotting a word with an incorrect phone: &lt;strong&gt;ROC-AUC 0.843&lt;/strong&gt;, against 0.788 for a plain phone error rate&lt;/li&gt;
&lt;li&gt;On words of two or more syllables, where names live: &lt;strong&gt;0.858&lt;/strong&gt;, against 0.838&lt;/li&gt;
&lt;li&gt;Strict by design: on learners' speech it flags 27% of words and catches 78% of the words the experts marked wrong. TTS voices make far fewer accent-level slips than learners do.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Speed.&lt;/strong&gt; Everything runs on a laptop CPU, with about 3.5 GB of models. With the models kept loaded, a short take is checked in 3 to 4 seconds.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it doesn't do yet
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The name results come from our own takes, because no public benchmark for TTS names exists yet.&lt;/li&gt;
&lt;li&gt;The sound check needs a researched pronunciation. A word that isn't in the lexicon only gets the word and letter checks.&lt;/li&gt;
&lt;li&gt;It doesn't replace a person. We run three checks that fail in different ways, keep a record of every failure, and someone still listens before anything ships.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The code
&lt;/h2&gt;

&lt;p&gt;It's open source under the MIT licence, runs locally and sends no data anywhere. It works as a command-line tool, or as a skill an AI coding agent can follow:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/logitsolutionsai/say-it-right" rel="noopener noreferrer"&gt;https://github.com/logitsolutionsai/say-it-right&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This post was written with AI assistance. The measurements are ours, and the scripts that produce them are in the repository.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>opensource</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Batch normalization, visually: the reset that keeps deep networks steady</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Mon, 09 Nov 2026 17:09:38 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/batch-normalization-visually-the-reset-that-keeps-deep-networks-steady-420p</link>
      <guid>https://dev.to/logitsolutionsai/batch-normalization-visually-the-reset-that-keeps-deep-networks-steady-420p</guid>
      <description>&lt;p&gt;&lt;em&gt;Deep Learning, Visually #20&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watch the Short:&lt;/strong&gt; &lt;a href="https://youtube.com/shorts/yYxpjlvW69s" rel="noopener noreferrer"&gt;youtube.com/shorts/yYxpjlvW69s&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the numbers drift
&lt;/h2&gt;

&lt;p&gt;A deep network is a chain of layers. Each layer takes a weighted sum of the activations from the layer before it, passes that sum through an activation function, and hands the result to the next layer.&lt;/p&gt;

&lt;p&gt;Training changes the weights at every step. So every layer keeps seeing inputs that have shifted since the last step: a new average, a new spread. In a deep network, those shifts pile up. With a big learning rate, the numbers can drift so far that they blow up, and training dies.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc8g4rxw8vlmkutv4ykz1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc8g4rxw8vlmkutv4ykz1.png" alt="A plain network at a big learning rate: the numbers run off the chart and the weights flash red." width="544" height="654"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What batch norm does
&lt;/h2&gt;

&lt;p&gt;Batch normalization adds a reset after each layer. For each neuron, over the mini-batch, it does three things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;It finds the average (μ_B) and the spread (σ_B) of the neuron's weighted sums.&lt;/li&gt;
&lt;li&gt;It slides them so their average sits at zero, and squeezes them so their spread is one.&lt;/li&gt;
&lt;li&gt;It applies a learned scale (γ) and a learned shift (β), so each layer can still pick its own range.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;In one line:&lt;/p&gt;

&lt;p&gt;y = γ · (x − μ_B) / √(σ_B² + ε) + β&lt;/p&gt;

&lt;p&gt;It sits after the weighted sum and before the activation function: Linear → BatchNorm → ReLU. Because β already gives that layer a shift, the layer needs no bias. In conv nets, there is one average and one spread per channel, taken over the mini-batch and every position.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi7nfs9g49ju0kun6v38n.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi7nfs9g49ju0kun6v38n.png" alt="One neuron's 60 weighted sums: centered, squeezed to a set spread, then scaled and shifted by the learned γ and β." width="518" height="593"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Training and testing
&lt;/h2&gt;

&lt;p&gt;During training, batch norm uses each mini-batch's own average and spread, and keeps running averages of them. At test time it uses those saved running averages. In PyTorch, model.train() and model.eval() switch between the two. The defaults are eps = 1e-5 and momentum = 0.1.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it bought
&lt;/h2&gt;

&lt;p&gt;Sergey Ioffe and Christian Szegedy, at Google, tested it on Inception, an image network:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Inception:&lt;/strong&gt; learning rate 0.0015. 31.0M steps to its best accuracy, 72.2%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BN-x5:&lt;/strong&gt; learning rate 0.0075 (5×). 2.1M steps to reach 72.2%, about 14 times fewer. Best 73.0%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BN-x30:&lt;/strong&gt; learning rate 0.045 (30×). 2.7M steps to reach 72.2%. Best 74.8%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;BN-x5 also used changes that batch norm made possible: no dropout, weight decay cut 5×, a learning rate lowered 6× faster, no local response normalization, better shuffling and fewer photo distortions.&lt;/p&gt;

&lt;p&gt;With sigmoid as the activation function, plain Inception never beat 0.1% accuracy, while BN-x5-Sigmoid reached 69.8%.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi6nxxdl9n8yxzmooz8o7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi6nxxdl9n8yxzmooz8o7.png" alt="The paper's result: a learning rate five times bigger, and the same accuracy in fourteen times fewer steps." width="501" height="506"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Our toy test
&lt;/h2&gt;

&lt;p&gt;We tried it on scikit-learn's 8×8 digits, with 1,500 training images and 297 test images. The network had 8 hidden layers of 64 ReLU neurons with a He start, trained by plain SGD on mini-batches of 60.&lt;/p&gt;

&lt;p&gt;First, a sweep: three seeds, 1,000 steps each.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Plain:&lt;/strong&gt; it trained at learning rates 0.03 and 0.1 (96–99% test accuracy). At 0.3, two of three seeds blew up. At 0.5, all three did. At 1.0, two blew up and the third stayed at 7%. At 2.0, all three blew up.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Batch norm:&lt;/strong&gt; every learning rate up to 2.0 trained (96–99%).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then a race at learning rate 0.5, five times the plain net's biggest safe one, same start, same mini-batches.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Plain:&lt;/strong&gt; the training loss fell from 3.2 to 0.67 by step 100, then jumped to 25 at step 130 and hit NaN at step 135. Test accuracy went from 83% at step 100 to 8% at step 132. Other seeds blew up at steps 128 and 348: the exact step is chaotic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Batch norm:&lt;/strong&gt; in layers 2, 4, 6 and 8, the middle stayed at 0.06 or less and the spread stayed between 0.99 and 1.06. Training loss fell to 0.06 by step 125. Test accuracy reached 96% by step 100 and 98% by step 135.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw3bmq1y95w3xtgslyv0u.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw3bmq1y95w3xtgslyv0u.png" alt="The race, same learning rate: the plain network blows up and its loss shoots off the chart; with batch norm the cloud stays centered and the loss stays low." width="527" height="512"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why does it work?
&lt;/h2&gt;

&lt;p&gt;The paper's idea is "internal covariate shift": each layer's inputs keep changing during training, and batch norm reduces that. Santurkar et al. (2018) tested this and argue that batch norm mostly makes the loss landscape smoother. Bjorck et al. (2018) find that it mainly allows larger learning rates. This is still debated.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits and cousins
&lt;/h2&gt;

&lt;p&gt;Small mini-batches hurt it, because their averages are noisy. At a batch size of 2, Group Norm had 10.6% lower error than batch norm (ResNet-50, ImageNet). Transformers use Layer Norm, which normalizes one example at a time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it in PyTorch
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Sequential&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Linear&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bias&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;BatchNorm1d&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ReLU&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Sequential&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Conv2d&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;padding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bias&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;BatchNorm2d&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ReLU&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Ioffe, S. &amp;amp; Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML 2015, PMLR 37:448–456. &lt;a href="https://proceedings.mlr.press/v37/ioffe15.html" rel="noopener noreferrer"&gt;https://proceedings.mlr.press/v37/ioffe15.html&lt;/a&gt; · &lt;a href="https://arxiv.org/abs/1502.03167" rel="noopener noreferrer"&gt;https://arxiv.org/abs/1502.03167&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Santurkar, S., Tsipras, D., Ilyas, A. &amp;amp; Madry, A. (2018). How Does Batch Normalization Help Optimization? NeurIPS 2018. &lt;a href="https://papers.nips.cc/paper/2018/hash/905056c1ac1dad141560467e0a99e1cf-Abstract.html" rel="noopener noreferrer"&gt;https://papers.nips.cc/paper/2018/hash/905056c1ac1dad141560467e0a99e1cf-Abstract.html&lt;/a&gt; · &lt;a href="https://arxiv.org/abs/1805.11604" rel="noopener noreferrer"&gt;https://arxiv.org/abs/1805.11604&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Bjorck, J., Gomes, C., Selman, B. &amp;amp; Weinberger, K. Q. (2018). Understanding Batch Normalization. &lt;a href="https://arxiv.org/abs/1806.02375" rel="noopener noreferrer"&gt;https://arxiv.org/abs/1806.02375&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Wu, Y. &amp;amp; He, K. (2018). Group Normalization. &lt;a href="https://arxiv.org/abs/1803.08494" rel="noopener noreferrer"&gt;https://arxiv.org/abs/1803.08494&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Ba, J. L., Kiros, J. R. &amp;amp; Hinton, G. E. (2016). Layer Normalization. &lt;a href="https://arxiv.org/abs/1607.06450" rel="noopener noreferrer"&gt;https://arxiv.org/abs/1607.06450&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;PyTorch docs: &lt;a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html" rel="noopener noreferrer"&gt;BatchNorm1d&lt;/a&gt; · &lt;a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="noopener noreferrer"&gt;BatchNorm2d&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CS878 Deep Learning (Fall 2023), Lecture 12–13: Regularization &amp;amp; Transfer Learning, Batch Normalization slides.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>deeplearning</category>
      <category>machinelearning</category>
      <category>pytorch</category>
      <category>ai</category>
    </item>
    <item>
      <title>Why did deep networks stop learning?</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Wed, 04 Nov 2026 14:53:33 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/why-did-deep-networks-stop-learning-4ebj</link>
      <guid>https://dev.to/logitsolutionsai/why-did-deep-networks-stop-learning-4ebj</guid>
      <description>&lt;h2&gt;
  
  
  Blame travels backward
&lt;/h2&gt;

&lt;p&gt;When a neural network makes a mistake, training works out how much each dial (each weight) was to blame. That blame starts at the answer and travels backward, layer by layer, so every dial knows which way to turn. This is backpropagation.&lt;/p&gt;

&lt;p&gt;On the way back, the blame passes through each layer's bend, the activation function. At every layer it gets multiplied by that bend's slope. This is the chain rule: one slope per layer, all multiplied together. In a deep network, that long chain of multiplications can shrink to almost nothing, or grow out of control.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quarter at a time
&lt;/h2&gt;

&lt;p&gt;Older networks used an S-shaped bend, the sigmoid: σ(s) = 1 / (1 + e⁻ˢ). Its slope is σ(s)(1 − σ(s)). That slope is steepest in the middle, at s = 0, where it is exactly 0.25. Everywhere else it is smaller, and far from the middle it is almost zero (the lecture calls these neurons saturated: they "kill the gradient flow").&lt;/p&gt;

&lt;p&gt;So each sigmoid layer passes back at most a quarter of the blame. Ten layers in a row, even at their steepest, pass back 0.25¹⁰ = 0.00000095 of it. That is less than a millionth.&lt;/p&gt;

&lt;p&gt;We measured it in a real network: 10 layers of 256 sigmoid neurons with a good (Xavier) start, 512 random inputs and targets, and a squared-error mistake score (numpy's default_rng(0)). Here is the blame that reached each layer's dials, compared with layer 10:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Layer 10: 1&lt;/li&gt;
&lt;li&gt;Layer 9: 0.19&lt;/li&gt;
&lt;li&gt;Layer 8: 0.051&lt;/li&gt;
&lt;li&gt;Layer 7: 0.013&lt;/li&gt;
&lt;li&gt;Layer 6: 0.003&lt;/li&gt;
&lt;li&gt;Layer 5: 0.00069&lt;/li&gt;
&lt;li&gt;Layer 4: 0.00015&lt;/li&gt;
&lt;li&gt;Layer 3: 0.000035&lt;/li&gt;
&lt;li&gt;Layer 2: 0.0000079&lt;/li&gt;
&lt;li&gt;Layer 1: 0.00000031&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqcivtlvffmfkukc9bxnz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqcivtlvffmfkukc9bxnz.png" alt="The blame shrinks plane by plane: layer 1 gets 0.0000003 of what layer 10 gets." width="465" height="623"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The blame shrinks plane by plane: layer 1 gets 0.0000003 of what layer 10 gets.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The first layer gets about three ten-millionths of the blame, so its dials barely move. This is the &lt;strong&gt;vanishing gradient problem&lt;/strong&gt;. Sepp Hochreiter worked it out in his 1991 diploma thesis at the Technical University of Munich, long before deep learning took off. Bengio, Simard and Frasconi analysed the same problem in 1994.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F14vyul0ssr8spr79dyql.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F14vyul0ssr8spr79dyql.png" alt="Sepp Hochreiter's 1991 diploma thesis, &amp;quot;Untersuchungen zu dynamischen neuronalen Netzen&amp;quot; (Studies of dynamic neural networks)." width="502" height="501"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Sepp Hochreiter's 1991 diploma thesis, "Untersuchungen zu dynamischen neuronalen Netzen" (Studies of dynamic neural networks).&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Too big: exploding blame
&lt;/h2&gt;

&lt;p&gt;The chain can also go the other way. If the dials start too big, the blame grows at every layer instead of shrinking. We took a healthy ReLU network and made every dial 2× too big. The blame reaching layer 1 was about 500,000× bigger than in the healthy network (512,003× when measured). With dials just 1.5× too big, it was still about 2,000× bigger.&lt;/p&gt;

&lt;p&gt;Huge blame means huge steps. The lecture warns that one giant step can throw the dials into a terrible spot with a large mistake, and in the worst case the numbers overflow to Inf or NaN, and training has to restart from an earlier checkpoint. The lecture also points out that this is not only a problem for recurrent networks: it hits all very deep networks.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgedcequz4na7fj6j65qf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgedcequz4na7fj6j65qf.png" alt="Gradient clipping: a cap line, and every layer's blame scaled down to fit under it." width="384" height="639"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Gradient clipping: a cap line, and every layer's blame scaled down to fit under it.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The fixes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;ReLU.&lt;/strong&gt; ReLU is max(0, s). When a neuron is on, its slope is exactly 1, so it passes the blame back at full strength. In the same 10-layer network with ReLU and a He start, the blame stayed between 0.4 and 2.1 of layer 10's at every layer (layer 1: 0.42). It helps a lot, but it is not a cure-all: a neuron that is off passes back nothing, and the dials can still start too big.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Good starting dials.&lt;/strong&gt; Starting the dials at the right size keeps the blame level from layer to layer (Deep Learning, Visually #18).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Batch normalization.&lt;/strong&gt; It keeps each layer's values in a healthy range during training (coming in #20).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Skip connections.&lt;/strong&gt; Shortcuts that let the blame jump over layers, used in ResNets (coming in #35).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gradient clipping&lt;/strong&gt;, for exploding blame. If the total size of the blame is bigger than a limit, scale all of it down to the limit. The direction stays the same; only the step gets smaller.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx2s7ndir6za9f3hqgtnx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx2s7ndir6za9f3hqgtnx.png" alt="ReLU: the blame reaches the first layers at full strength." width="510" height="588"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;ReLU: the blame reaches the first layers at full strength.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it in PyTorch
&lt;/h2&gt;

&lt;p&gt;Use ReLU layers with a He start, and clip the blame before every step:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch.nn&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;

&lt;span class="n"&gt;layer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Linear&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;init&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;kaiming_normal_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nonlinearity&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;relu&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Sequential&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ReLU&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="n"&gt;loss&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;backward&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utils&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;clip_grad_norm_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;max_norm&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;optimizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or watch the blame vanish yourself. This sends one number through ten bends and asks how much blame comes back:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sigmoid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relu&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
 &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ones&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;requires_grad_&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
 &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;
 &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;x&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;f&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
 &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;backward&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;grad&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It prints tensor([2.7848e-07]) for the sigmoid and tensor([1.]) for ReLU. Change 10 to 30: how small does the sigmoid's blame get now?&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Hochreiter, J. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Technische Universität München. &lt;a href="https://www.bioinf.jku.at/publications/older/3804.pdf" rel="noopener noreferrer"&gt;https://www.bioinf.jku.at/publications/older/3804.pdf&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Bengio, Y., Simard, P., &amp;amp; Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. &lt;a href="https://doi.org/10.1109/72.279181" rel="noopener noreferrer"&gt;https://doi.org/10.1109/72.279181&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Glorot, X., Bordes, A., &amp;amp; Bengio, Y. (2011). Deep Sparse Rectifier Neural Networks. AISTATS, PMLR 15, 315–323. &lt;a href="https://proceedings.mlr.press/v15/glorot11a.html" rel="noopener noreferrer"&gt;https://proceedings.mlr.press/v15/glorot11a.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Pascanu, R., Mikolov, T., &amp;amp; Bengio, Y. (2013). On the difficulty of training recurrent neural networks. ICML, PMLR 28(3), 1310–1318. &lt;a href="https://proceedings.mlr.press/v28/pascanu13.html" rel="noopener noreferrer"&gt;https://proceedings.mlr.press/v28/pascanu13.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CS878 Deep Learning, SEECS NUST: lecture 10–11 (slides 35–43) and lecture 23–24 (vanishing and exploding gradients, gradient clipping).&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>deeplearning</category>
      <category>beginners</category>
    </item>
    <item>
      <title>Why does every AI start out random?</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Wed, 04 Nov 2026 14:23:34 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/why-does-every-ai-start-out-random-ggh</link>
      <guid>https://dev.to/logitsolutionsai/why-does-every-ai-start-out-random-ggh</guid>
      <description>&lt;h2&gt;
  
  
  Twin neurons
&lt;/h2&gt;

&lt;p&gt;A neural network learns by turning its dials, the weights. Every dial needs a first value. Why not give them all the same one?&lt;/p&gt;

&lt;p&gt;Take a small network: 2 inputs, 4 hidden neurons, 1 output. Set every dial to 0.5 and every bias to 0. Now all four hidden neurons see exactly the same thing, so they compute the same value. They make the same mistake, so training gives each one the same push. After the push they are still identical, so the next step repeats it.&lt;/p&gt;

&lt;p&gt;We trained this network for 2,000 steps. The four hidden neurons stayed exactly identical the whole time: the difference between them was 0.0 at every step. They are twins, forever.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw646wj5ua9ng2lrp2f75.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw646wj5ua9ng2lrp2f75.png" alt="Every dial starts at 0.5: the four hidden neurons get the same push at every step. Step 2,000, still twins." width="441" height="475"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Every dial starts at 0.5: the four hidden neurons get the same push at every step. Step 2,000, still twins.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Starting every dial at zero is even worse. Then nothing moves at all: every push is exactly zero, the network answers 0.5 for every input, and the loss stays at 0.693 forever (2 of 4 answers right, no better than a coin).&lt;/p&gt;

&lt;h2&gt;
  
  
  The corner puzzle (this is XOR)
&lt;/h2&gt;

&lt;p&gt;The puzzle has four corners. The output should be 1 when exactly one input is 1: (0,1) and (1,0) give 1, while (0,0) and (1,1) give 0. This is the XOR problem.&lt;/p&gt;

&lt;p&gt;Four twin neurons are no better than one neuron, and one neuron can only draw one straight line. As we saw in part 3, one straight line can't solve XOR. The same-start network gets stuck at 3 of 4: its outputs are 0.00, 0.67, 0.67 and 0.67, so the corner (1,1) is wrong, and the loss is stuck near 0.48.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzj975s94mipzkiai7hpy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzj975s94mipzkiai7hpy.png" alt="Four twins act like one neuron: one straight split. 3 of 4 right, the top-right corner is wrong." width="441" height="498"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Four twins act like one neuron: one straight split. 3 of 4 right, the top-right corner is wrong.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Now start the dials at random instead (numpy's default_rng(1)). The first-layer dials are [[0.35, 0.82, 0.33, −1.30], [0.91, 0.45, −0.54, 0.58]], and the output dials come from the same generator. Every neuron begins a little different, so each gets its own push and learns its own part. The network gets all 4 corners right from step 18. By step 2,000 its outputs are 0.00, 1.00, 1.00 and 0.00. Its answer is no longer one straight split: it marks the two corners that should be 1 and leaves a clear diagonal band through the other two.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqk2ljf6h3gtr6eh17t5n.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqk2ljf6h3gtr6eh17t5n.png" alt="Random start: four different neurons, four different pushes. 4 of 4 from step 18." width="448" height="579"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Random start: four different neurons, four different pushes. 4 of 4 from step 18.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Over 20 random starts (seeds 0 to 19), 19 solve the puzzle.&lt;/p&gt;

&lt;h2&gt;
  
  
  How random?
&lt;/h2&gt;

&lt;p&gt;Random isn't enough: the size of the random dials matters too. The CS878 lecture tests this on a deep network: 10 layers of 500 tanh neurons each, fed 1,000 random inputs. Every neuron adds up 500 inputs, each multiplied by its dial.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dials too small (0.01):&lt;/strong&gt; the spread of the values shrinks in every layer: 0.21, 0.048, 0.011, 0.0024 and on down to 0.0000003 at layer 10. The signal fades to nothing, and tiny values make tiny pushes, so training crawls.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dials too big (1.0):&lt;/strong&gt; the sums get so large that tanh is pushed to its limit. In every layer, 93% of the values are stuck beyond ±0.95. There, tanh is almost flat, so the pushes vanish again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The right size (1/√500 ≈ 0.045):&lt;/strong&gt; the spread stays healthy all the way down: 0.63, 0.49, 0.41, 0.36, 0.32, 0.29, 0.27, 0.25, 0.24, 0.23. About 7% of the values are stuck in layer 1, 0.4% in layer 2, and almost none after that.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn7llr43e3xtw7fcbu2bq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn7llr43e3xtw7fcbu2bq.png" alt="The right dial size: healthy spreads from layer 1 to layer 10." width="471" height="611"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The right dial size: healthy spreads from layer 1 to layer 10.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The rule
&lt;/h2&gt;

&lt;p&gt;A neuron adds up its inputs: s = w₁x₁ + w₂x₂ + … + wₙxₙ. If the dials and the inputs are random with an average of zero, the spreads (variances) add up:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Var(s) = n · Var(w) · Var(x)&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;To keep the spread the same from one layer to the next, make n · Var(w) = 1, which gives Var(w) = 1/n: dials with a spread of 1/√n. More inputs, smaller starting dials. This is &lt;strong&gt;Xavier initialization&lt;/strong&gt;, named after Xavier Glorot, who worked it out with Yoshua Bengio in 2010. Their paper also balances the backward pass, with Var(w) = 2 / (inputs + outputs). It suits tanh.&lt;/p&gt;

&lt;p&gt;ReLU zeroes out half of the values, so ReLU layers need twice as much: Var(w) = 2/n. This is &lt;strong&gt;He initialization&lt;/strong&gt; (Kaiming He and colleagues, 2015).&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it in PyTorch
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch.nn&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;
&lt;span class="n"&gt;layer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Linear&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;init&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;xavier_normal_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                         &lt;span class="c1"&gt;# tanh layers
&lt;/span&gt;&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;init&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;kaiming_normal_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nonlinearity&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;relu&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# ReLU layers
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a 500-to-500 layer, xavier_normal_ draws dials with a spread of √(2 / (500 + 500)) = 1/√500 ≈ 0.045, the right size from the Short. kaiming_normal_ uses √(2/500) ≈ 0.063. PyTorch already starts every nn.Linear at random, in a range that shrinks as its inputs grow.&lt;/p&gt;

&lt;p&gt;The Short's experiment fits in six lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="mi"&gt;01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;**-&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
  &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;randn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;x&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
   &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="nd"&gt;@t.randn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;tanh&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
  &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;std&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It prints the spread after 10 layers for each dial size: 3.0e-07 (fades to nothing), 9.8e-01 (stuck at ±1) and 2.3e-01 (healthy). Swap .tanh() for .relu(): which dial size keeps the signal alive now?&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Glorot, X., &amp;amp; Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS, PMLR 9, 249–256. &lt;a href="https://proceedings.mlr.press/v9/glorot10a.html" rel="noopener noreferrer"&gt;https://proceedings.mlr.press/v9/glorot10a.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;He, K., Zhang, X., Ren, S., &amp;amp; Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV 2015. &lt;a href="https://arxiv.org/abs/1502.01852" rel="noopener noreferrer"&gt;https://arxiv.org/abs/1502.01852&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CS878 Deep Learning, SEECS NUST, lecture 10–11 (slides 52–72).&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>deeplearning</category>
      <category>beginners</category>
    </item>
    <item>
      <title>Why slow down near the finish line?</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Sun, 01 Nov 2026 15:00:40 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/why-slow-down-near-the-finish-line-1hcj</link>
      <guid>https://dev.to/logitsolutionsai/why-slow-down-near-the-finish-line-1hcj</guid>
      <description>&lt;h2&gt;
  
  
  Rolling down a bowl with noisy pushes
&lt;/h2&gt;

&lt;p&gt;Training an AI is like rolling a ball down a bowl. The ball's position is the AI's dials (its weights), the height is how many mistakes it makes, and the bottom means the fewest mistakes. Each training step moves the ball a little downhill:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;w ← w − step size × push&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;The step size is the learning rate and the push is the gradient. But each push comes from just a few examples (a mini-batch), not all the data, so it always points a little off the true downhill direction.&lt;/p&gt;

&lt;p&gt;The bowl in the Short: mistake = ½ · (w1² + 4 · w2²), a long oval with its bottom at (0, 0). The ball starts at (−4, 1.5), 4.27 from the bottom. Each push is the true slope (w1, 4 · w2) plus 0.6 × a normal random number in each direction, from numpy's default_rng(2252). All three runs below use the same noise and take 60 steps.&lt;/p&gt;

&lt;h2&gt;
  
  
  Big steps bounce, small steps crawl
&lt;/h2&gt;

&lt;p&gt;With a big step size, 0.45, the ball is about 0.2 from the bottom by step 6. Then the noise takes over: near the bottom the true slope is tiny, so the push is mostly noise, and a big step turns it into a big jump past the middle. The ball bounces around and never settles. Over the last 10 steps it is 0.50 from the bottom on average.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqt65ic5vc1qri0q7fzaq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqt65ic5vc1qri0q7fzaq.png" alt="Big steps: near the bottom by step 6, then every off push throws the ball past the middle. Step 60, still bouncing." width="518" height="446"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Big steps: near the bottom by step 6, then every off push throws the ball past the middle. Step 60, still bouncing.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;With a small step size, 0.02, the bounces are tiny, but the trip is slow: 3.33 from the bottom after 10 steps, 2.73 after 20, 1.75 after 40 and 1.21 after 60. Same number of steps, still far away.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3ee17au56esuccv83pux.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3ee17au56esuccv83pux.png" alt="Small steps: tiny bounces, but after the same 60 steps the ball is still 1.2 away. The big-step cloud stays as a grey ghost." width="520" height="443"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Small steps: tiny bounces, but after the same 60 steps the ball is still 1.2 away. The big-step cloud stays as a grey ghost.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Lecture 09 puts it in one line: you need a high learning rate when far away and a low learning rate when close.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start big, then cut the step size in half
&lt;/h2&gt;

&lt;p&gt;A learning-rate schedule uses both. Start with the big step size, 0.45, and halve it every 10 steps: 0.45, 0.225, 0.113, 0.056, 0.028, 0.014.&lt;/p&gt;

&lt;p&gt;The first 10 steps match the big-step run exactly. After that, each halving shrinks the bounces: the ball is 0.25 from the bottom at step 20, 0.14 at step 40 and 0.07 at step 60. Over the last 10 steps it averages 0.09, more than five times closer than big steps alone.&lt;/p&gt;

&lt;p&gt;Over 2,000 different noise sequences, the last-10-step averages are: big steps 0.49, small steps 1.31, the schedule 0.09. That is about 5.6 times closer than big steps.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frj9sj2q9shjip514yxl5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frj9sj2q9shjip514yxl5.png" alt="The staircase: 0.45, halved at steps 10, 20, 30, 40 and 50. Big steps end 0.5 away, the schedule 0.09 away." width="526" height="620"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The staircase: 0.45, halved at steps 10, 20, 30, 40 and 50. Big steps end 0.5 away, the schedule 0.09 away.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Other schedules
&lt;/h2&gt;

&lt;p&gt;Halving every n steps is called step decay. Lecture 09 lists other ways to shrink the learning rate α:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Step decay:&lt;/strong&gt; multiply α by a decay rate (often 0.5) every n steps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exponential decay:&lt;/strong&gt; α = tᵉᵖᵒᶜʰ · α0, where the decay rate t is below 1, so α shrinks a little every epoch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;α = α0 / (1 + decay_rate · epoch):&lt;/strong&gt; with α0 = 0.1 and decay_rate = 1.0, the learning rate goes 0.1, 0.05, 0.033, 0.025 over epochs 0 to 3.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A popular smooth option is the cosine schedule, which lowers the learning rate along half a cosine wave. SGDR (Loshchilov and Hutter) adds warm restarts: the learning rate decays, jumps back up, and decays again.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why it works
&lt;/h2&gt;

&lt;p&gt;In 1951, Herbert Robbins and Sutton Monro studied how to find an answer when every measurement is noisy, just like a ball pushed by a few examples at a time. They showed that noisy steps still settle if the step sizes aₜ shrink so that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Σ aₜ = ∞: added up, the steps are unlimited, so the ball can still travel as far as it needs.&lt;/li&gt;
&lt;li&gt;Σ aₜ² &amp;lt; ∞: the steps shrink fast enough for the noise they carry to average out.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, aₜ = a0 / (1 + k · t) meets both. Halving every few steps is the practical version used in training. Strictly, halving forever adds up to a finite total and misses the first condition, but over a fixed training run it does the same job: big steps far away, small steps up close.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fda6yricfjx3t41enwahl.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fda6yricfjx3t41enwahl.png" alt="Robbins and Monro, &amp;quot;A Stochastic Approximation Method&amp;quot; (1951): noisy steps settle when the step sizes shrink." width="520" height="493"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Robbins and Monro, "A Stochastic Approximation Method" (1951): noisy steps settle when the step sizes shrink.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it in PyTorch
&lt;/h2&gt;

&lt;p&gt;In PyTorch a schedule wraps the optimizer. This one halves the learning rate every 10 epochs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;scheduler&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;optim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lr_scheduler&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;StepLR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;optimizer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;step_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;gamma&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Call scheduler.step() once per epoch, after that epoch's optimizer steps. This article's schedule on its own:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;optim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SGD&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ones&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)],&lt;/span&gt;&lt;span class="n"&gt;lr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;45&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;optim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lr_scheduler&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;StepLR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,.&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_last_lr&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
  &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It prints the staircase: 0 [0.45], 10 [0.225], 20 [0.1125], 30 [0.05625], 40 [0.028125], 50 [0.0140625]. Change 10 to 5 and run it again: how small is the step size by step 50 now?&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Robbins, H., &amp;amp; Monro, S. (1951). A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), 400–407. &lt;a href="https://doi.org/10.1214/aoms/1177729586" rel="noopener noreferrer"&gt;https://doi.org/10.1214/aoms/1177729586&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Loshchilov, I., &amp;amp; Hutter, F. (2016). SGDR: Stochastic Gradient Descent with Warm Restarts. &lt;a href="https://arxiv.org/abs/1608.03983" rel="noopener noreferrer"&gt;https://arxiv.org/abs/1608.03983&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CS231n notes, "Annealing the learning rate": &lt;a href="https://cs231n.github.io/neural-networks-3/" rel="noopener noreferrer"&gt;https://cs231n.github.io/neural-networks-3/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CS878 Deep Learning, SEECS NUST, lecture 09 (slides 5–7).&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>deeplearning</category>
      <category>beginners</category>
    </item>
    <item>
      <title>What if every dial had its own step size?</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Sun, 01 Nov 2026 14:30:42 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/what-if-every-dial-had-its-own-step-size-1bil</link>
      <guid>https://dev.to/logitsolutionsai/what-if-every-dial-had-its-own-step-size-1bil</guid>
      <description>&lt;h2&gt;
  
  
  One step size for every dial
&lt;/h2&gt;

&lt;p&gt;An AI learns by turning dials. A network has millions of them, its weights, and training nudges every one a little at a time. To know which way to turn a dial, training measures how the mistake changes when that dial moves. That measurement is the dial's push, the gradient. It points downhill, toward a smaller mistake.&lt;/p&gt;

&lt;p&gt;Plain gradient descent moves every dial with the same rule:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;w ← w − step × push&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;There is one step size, and every dial shares it. The trouble is that hills differ. Take two dials. Both start at w = −2, and both make the smallest mistake at w = 0.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The top dial sits on a steep, narrow hill: mistake = 25 · w². Its push at the start is 100.&lt;/li&gt;
&lt;li&gt;The bottom dial sits on a wide, nearly flat hill: mistake = ½ · w². Its push is 2.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Same position, but on the steep hill the push is fifty times bigger.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2e4z6n43ewpq5a8l2ibx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2e4z6n43ewpq5a8l2ibx.png" alt="Two dials, two hills: the same start, but the steep hill's push is fifty times bigger, and one slider sets the step size for both." width="538" height="561"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Two dials, two hills: the same start, but the steep hill's push is fifty times bigger, and one slider sets the step size for both.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Try a big step size, 0.045. The steep dial jumps right past the bottom: from −2 to 2.5, then −3.125, then 3.906. Every jump is bigger than the last, and it flies off. Try a small step size, 0.015, which suits the steep hill: it settles in three hops (−0.5, −0.125, −0.031). Now the flat dial crawls. It needs 245 steps to get within 0.05 of the bottom. No single step size fits both.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmizpwm7omaxc4deeqrp0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmizpwm7omaxc4deeqrp0.png" alt="With a big step size the steep dial jumps past the bottom, higher each time, and flies off." width="513" height="635"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;With a big step size the steep dial jumps past the bottom, higher each time, and flies off.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Each dial's own step size
&lt;/h2&gt;

&lt;p&gt;The fix is to stop sharing. Each dial remembers how big its pushes usually are, and divides its push by that. A dial with big pushes takes careful steps. A dial with tiny pushes takes bolder ones.&lt;/p&gt;

&lt;p&gt;On the very first step, each push divided by its usual size is exactly 1 for both dials (100 out of 100, and 2 out of 2). So both dials move exactly the same distance: 0.1. After that their paths stay identical. They roll down together, overshoot a little to +0.26, swing back, and both stay within 0.05 of the bottom from step 71.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwds2gakv0mf7elbxrawr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwds2gakv0mf7elbxrawr.png" alt="Both dials take the same steps: a line joining them stays vertical as they roll down together." width="513" height="538"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Both dials take the same steps: a line joining them stays vertical as they roll down together.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What Adam computes
&lt;/h2&gt;

&lt;p&gt;This method is Adam, from a paper by Diederik Kingma and Jimmy Ba, published at ICLR 2015. The name comes from "adaptive moment estimation". It combines two ideas: momentum, from the last episode, and its own step size for every dial.&lt;/p&gt;

&lt;p&gt;For each dial, g is its push at this step and t counts the steps (1, 2, 3, …). Adam keeps two running averages and turns them into a step:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;m = 0.9 · m + 0.1 · g (the average push: this is momentum)&lt;/li&gt;
&lt;li&gt;v = 0.999 · v + 0.001 · g² (the average squared push: how big the pushes usually are)&lt;/li&gt;
&lt;li&gt;m̂ = m / (1 − 0.9ᵗ) and v̂ = v / (1 − 0.999ᵗ) (m and v start at zero, so early on they come out too small; this corrects them)&lt;/li&gt;
&lt;li&gt;step = lr · m̂ / (√v̂ + 1e-8)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The 1e-8 is only there to avoid dividing by zero. The key part is the division by √v̂, which has the same size as the push. If a dial's pushes are all fifty times bigger, its m̂ and its √v̂ are both fifty times bigger, and they cancel. That is why the steep dial and the flat dial move the same.&lt;/p&gt;

&lt;h2&gt;
  
  
  The steeper-or-flatter test
&lt;/h2&gt;

&lt;p&gt;Now make both hills a hundred times steeper, then a hundred times flatter, and keep every setting the same.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A hundred times steeper:&lt;/strong&gt; the shared step size of 0.015 blows up on the steep hill. After one step the dial is at 148, after two at −10,952.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A hundred times flatter:&lt;/strong&gt; the same shared step size crawls. The flat dial needs 24,591 steps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Adam&lt;/strong&gt; (step size 0.1) takes 71 steps on the original hills, 71 on the steeper ones and 71 on the flatter ones, on exactly the same path every time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is a big part of why Adam is so popular: when a problem's scale changes, you don't have to hunt for a new step size.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq1v6etfcotgygiis70of.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq1v6etfcotgygiis70of.png" alt="The old way blows up or crawls for 24,591 steps; Adam takes the same path every time: 71 · 71 · 71." width="513" height="405"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The old way blows up or crawls for 24,591 steps; Adam takes the same path every time: 71 · 71 · 71.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it in PyTorch
&lt;/h2&gt;

&lt;p&gt;In PyTorch, Adam is one line. A learning rate of 1e-3 is the usual starting point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;opt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;optim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Adam&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;lr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1e-3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here are the two dials from this article, for one step:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Parameter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ones&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;optim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Adam&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="n"&gt;lr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;L&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;
&lt;span class="n"&gt;L&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;backward&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It prints &lt;code&gt;tensor([-1.9000, -1.9000])&lt;/code&gt;. Both dials moved exactly 0.1, even though one push was fifty times bigger. Change 25 to 2500 and run it again: still 0.1.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Kingma, D. P., &amp;amp; Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015. &lt;a href="https://arxiv.org/abs/1412.6980" rel="noopener noreferrer"&gt;https://arxiv.org/abs/1412.6980&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CS878 Deep Learning, SEECS NUST, lectures 07–08 (course slides).&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>deeplearning</category>
      <category>beginners</category>
    </item>
    <item>
      <title>How does a rolling ball skip small bumps?</title>
      <dc:creator>Syed Abdul Hanan</dc:creator>
      <pubDate>Sun, 01 Nov 2026 01:20:00 +0000</pubDate>
      <link>https://dev.to/logitsolutionsai/how-does-a-rolling-ball-skip-small-bumps-390d</link>
      <guid>https://dev.to/logitsolutionsai/how-does-a-rolling-ball-skip-small-bumps-390d</guid>
      <description>&lt;p&gt;Momentum, step by step: give gradient descent a memory, and it rolls past the traps that stop it.&lt;/p&gt;

&lt;p&gt;Plain gradient descent looks only at the slope under its feet. Every step is&lt;/p&gt;

&lt;p&gt;&lt;code&gt;w ← w − η ∇L(w)&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;and nothing else. That makes it simple, but it also makes it forgetful: the moment the ground goes flat, it stops. Momentum fixes this with one extra number per weight, a velocity that remembers the slopes it has already rolled down. The examples below follow the CS878 lectures on gradient descent and scaling the optimization, and every number comes from running the update rule.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap
&lt;/h2&gt;

&lt;p&gt;Take one weight and a loss with two dips:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;L(w) = 0.12 (w − 2.5)² − 0.55 · exp(−(w + 1)² / 0.12)&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Start at A, w = −3.5, where the loss is 4.32. There is a shallow dip B at w = −0.904 (loss 0.881) and the real minimum C at w = 2.5 (loss 0). Between them sits a bump only 0.13 higher than B.&lt;/p&gt;

&lt;p&gt;Plain gradient descent with η = 0.1 takes big steps on the steep slope, smaller and smaller ones as it nears B, and by step 21 it has arrived. At B the gradient is zero, so every step after that is zero too. At step 50, at step 100, it is still in B, at a loss of 0.88.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn7n5gro5ry9qkptvnalw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn7n5gro5ry9qkptvnalw.png" alt="Plain gradient descent stops in the shallow dip B: the gradient is zero there, so learning stops." width="485" height="676"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Plain gradient descent stops in the shallow dip B: the gradient is zero there, so learning stops.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Add a velocity
&lt;/h2&gt;

&lt;p&gt;Momentum keeps a velocity v and moves along it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;v_t = β v_(t−1) + ∇L(w_t)&lt;/li&gt;
&lt;li&gt;w_(t+1) = w_t − η v_t&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;With β = 0.9, each step keeps 90% of the old velocity and adds the new gradient. (This is the form PyTorch uses. Lecture notes often write v = βv + (1 − β)∇L, which only rescales the learning rate.)&lt;/p&gt;

&lt;p&gt;Run it from A with the same η = 0.1. By step 6 the ball is near B and its velocity is about 6. The gradient at B is zero, but the velocity isn't, so the ball keeps going: it clears the bump at step 7, overshoots all the way to w = 4.71, swings back without ever returning to the bump, and settles in C by step 89.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpxjy0irlqj0md8j9ekl2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpxjy0irlqj0md8j9ekl2.png" alt="Momentum carries the ball over the bump; plain GD's ghost is still stuck in B." width="525" height="689"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Momentum carries the ball over the bump; plain GD's ghost is still stuck in B.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What v remembers
&lt;/h2&gt;

&lt;p&gt;Unroll the velocity and it becomes a weighted sum of every gradient so far:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;v_t = ∇L_t + β ∇L_(t−1) + β² ∇L_(t−2) + …&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;The latest gradient counts fully, the one before 90%, then 81%, 73%, 66% and so on. The weights add up to 1/(1 − β) = 10, so momentum remembers about ten steps. With β = 0.1 they would be 1, 0.1, 0.01: it forgets almost at once. In practice β is usually around 0.9.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fufusrqgc09ghwsgcjjiv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fufusrqgc09ghwsgcjjiv.png" alt="Eight bars, heights β^k, sum 1/(1 − β) = 10: gradients that agree stack toward 10; gradients that flip cancel, staying within ±0.53." width="509" height="572"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Eight bars, heights β^k, sum 1/(1 − β) = 10: gradients that agree stack toward 10; gradients that flip cancel, staying within ±0.53.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Agree or cancel
&lt;/h2&gt;

&lt;p&gt;That memory does two different things depending on the gradients:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;When they agree (+1, +1, +1 …), they stack: v grows 1, 1.9, 2.71, 3.44 … toward 1/(1 − β) = 10. The step grows up to ten times.&lt;/li&gt;
&lt;li&gt;When they flip sign (+1, −1, +1 …), they cancel: v goes 1, −0.1, 0.91, −0.18, 0.84, −0.25 … and settles at about ±1/(1 + β) ≈ 0.53. The step stays small.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A narrow valley
&lt;/h2&gt;

&lt;p&gt;Now two weights and a valley that is steep across and almost flat along: L = ½(w₁² + 50 w₂²), starting at (−4, 1), where the loss is 33. Count the steps until the loss drops below 0.001, with η = 0.015:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Plain GD: 298 steps.&lt;/strong&gt; A bigger rate doesn't help: at η = 0.045, each step multiplies the across direction by 1 − 0.045 · 50 = −1.25 and it blows up. So plain GD has to crawl along the floor.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Momentum (β = 0.9): 73 steps.&lt;/strong&gt; The along direction keeps agreeing, so it builds speed there. But the across direction overshoots and swings from side to side.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc9elyygi0ykfoaac8tra.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc9elyygi0ykfoaac8tra.png" alt="The narrow valley: plain GD crawls along the floor; momentum is faster but swings across." width="483" height="552"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The narrow valley: plain GD crawls along the floor; momentum is faster but swings across.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Nesterov: look before you step
&lt;/h2&gt;

&lt;p&gt;Nesterov momentum changes one thing: it takes the gradient where the velocity is about to carry the weight, not where the weight is now.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;v_t = β v_(t−1) + ∇L(w_t − η β v_(t−1))&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;If that look-ahead point is already uphill, the gradient there points back, and the step brakes before it overshoots. On the same valley, Nesterov needs &lt;strong&gt;18 steps&lt;/strong&gt;, with almost no swing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fabpckd4vobsph63v5xqr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fabpckd4vobsph63v5xqr.png" alt="Nesterov looks ahead: the gradient at the look-ahead point brakes the swing. 298 · 73 · 18." width="543" height="634"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Nesterov looks ahead: the gradient at the look-ahead point brakes the swing. 298 · 73 · 18.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it in PyTorch
&lt;/h2&gt;

&lt;p&gt;Every weight in a network gets its own velocity, with the same shape as the weight. In PyTorch it is one line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;opt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;optim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SGD&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;lr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.015&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                      &lt;span class="n"&gt;momentum&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nesterov&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the velocity itself, with gradients that flip sign:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.9&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It prints &lt;code&gt;[1.0, -0.1, 0.91, -0.18, 0.84]&lt;/code&gt;. Change every g to 1 and v climbs toward 10.&lt;/p&gt;

&lt;h2&gt;
  
  
  Key takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Plain gradient descent has no memory: it stops wherever the slope is zero, even in a shallow dip.&lt;/li&gt;
&lt;li&gt;Momentum keeps a velocity, v = βv + ∇L, and steps along it, so it can roll through flat spots and over small bumps.&lt;/li&gt;
&lt;li&gt;v is a sum of past gradients weighted by β^k, about 1/(1 − β) = 10 steps of memory at β = 0.9.&lt;/li&gt;
&lt;li&gt;Agreeing gradients stack (up to 10×); flipping ones cancel. That is why momentum speeds up along valleys.&lt;/li&gt;
&lt;li&gt;Nesterov takes the gradient at the look-ahead point, so it brakes before overshooting: 298 → 73 → 18 steps in the narrow valley.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>deeplearning</category>
      <category>beginners</category>
    </item>
  </channel>
</rss>
