<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ahmad Dogar</title>
    <description>The latest articles on DEV Community by Ahmad Dogar (@ahmad_dogar_7d098f5e48f72).</description>
    <link>https://dev.to/ahmad_dogar_7d098f5e48f72</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1755030%2F424dcf81-ba78-4790-977a-16b5b4d3a13c.png</url>
      <title>DEV Community: Ahmad Dogar</title>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ahmad_dogar_7d098f5e48f72"/>
    <language>en</language>
    <item>
      <title>An offline outbox for photo evidence in React Native, and the bugs it taught me</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Mon, 16 Nov 2026 05:38:39 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/an-offline-outbox-for-photo-evidence-in-react-native-and-the-bugs-it-taught-me-252h</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/an-offline-outbox-for-photo-evidence-in-react-native-and-the-bugs-it-taught-me-252h</guid>
      <description>&lt;p&gt;An insurance claim for storm damage lives or dies on its photos. The adjuster wants the overview, every slope, the ridge and hips, the damage close up, the gutters and a measurement or two. If one required shot is missing, someone has to go back up the ladder, and after a storm the roof may already be tarped or repaired.&lt;/p&gt;

&lt;p&gt;The people taking those photos are on roofs in rural areas, often right after the weather that took out the cell towers. ClaimLayer's field app is built around one rule: the phone never needs signal to do its job. Below: how that works, what the server does with the queue, and the bugs that showed up once I got strict about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why "upload when you press the shutter" fails
&lt;/h2&gt;

&lt;p&gt;The obvious version uploads each photo as it is taken. With no signal the request fails, and depending on how the code is written the photo is lost or the user stares at a spinner. Every fix for that leads to the same place: write locally first, send later, and make sending safe to repeat.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjljh6yzjlcuo6gse01f.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjljh6yzjlcuo6gse01f.webp" alt="On the roof the capture goes into an on-device outbox; back in signal syncOutbox sends it to the server and the office console" width="799" height="540"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Capture first, sync later.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Capture writes two operations
&lt;/h2&gt;

&lt;p&gt;When the inspector takes a photo, the app resizes it, hashes it and queues two operations: one for the file and one for the evidence record it belongs to.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;apps/field/src/InspectionScreen.tsx&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;photo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;camera&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;takePictureAsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;quality&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.85&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;exif&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;resize&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;photo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;photo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;height&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1600&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1600&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;resized&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nc"&gt;ImageManipulator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;manipulateAsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;photo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;resize&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;compress&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.82&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ImageManipulator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SaveFormat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;JPEG&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;contents&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nc"&gt;FileSystem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;readAsStringAsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resized&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;FileSystem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;EncodingType&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Base64&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;digest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nc"&gt;Crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;digestStringAsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;CryptoDigestAlgorithm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SHA256&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;contents&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sb"&gt;`${Date.now()}-${digest.slice(0, 8)}`&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;enqueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;storage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`media-${base}`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;claimId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;operation&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'media.upload'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`media-${base}-offline`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;occurredAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;capturedAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;metadata&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;enqueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;storage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`evidence-${base}`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;claimId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;operation&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'evidence.save'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`evidence-${base}-offline`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;occurredAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;capturedAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;inspectionId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;itemKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="no"&gt;ITEMS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;activeItem&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;caption&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;caption&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fekuoh44lctyzl24uy000.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fekuoh44lctyzl24uy000.webp" alt="A photo is resized, hashed and queued as a media upload and an evidence save" width="800" height="813"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;One photo becomes two queued operations.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A few details are worth copying:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Resize on the device.&lt;/strong&gt; 1600 px on the long edge at JPEG 0.82 is plenty for an adjuster and keeps a day of photos small.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Store the file path, not the bytes.&lt;/strong&gt; The queue holds the URI and metadata, so the queue itself stays small.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Record where and when.&lt;/strong&gt; Capture time, device time, EXIF, GPS and accuracy (when permitted) and a SHA-256 travel with every photo.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every operation gets its own idempotency key,&lt;/strong&gt; built from time and content hash, minted on the device before any network call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One capture at a time.&lt;/strong&gt; A busy flag stops two captures racing on the same queue key.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The screen tells the inspector exactly that: "Capture locally first. The original remains on this device until the server confirms upload."&lt;/p&gt;

&lt;h2&gt;
  
  
  Sync: the status code decides
&lt;/h2&gt;

&lt;p&gt;The outbox envelope is defined once in a shared contracts package, so the app and the server agree on its shape:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;packages/contracts/src/index.ts&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;OutboxOperation&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;
  &lt;span class="n"&gt;claimId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;
  &lt;span class="n"&gt;operation&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'media.upload'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="s1"&gt;'evidence.save'&lt;/span&gt;
  &lt;span class="n"&gt;idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;   &lt;span class="c1"&gt;// 16 to 128 chars&lt;/span&gt;
  &lt;span class="n"&gt;occurredAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;
  &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Record&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;unknown&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'pending'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="s1"&gt;'in_flight'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="s1"&gt;'synced'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="s1"&gt;'refused'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="s1"&gt;'abandoned'&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sync walks the queue one item at a time, in capture order, and the response decides what happens to each item.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;apps/field/src/api.ts&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetcher&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`${baseUrl}/api/trpc/inspections.saveEvidence`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'POST'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;deviceId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;claimId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;claimId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;inspectionId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inspectionId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;itemKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;itemKey&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;photos&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;note&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;caption&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;idempotencyKey&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="s1"&gt;'synced'&lt;/span&gt;
    &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="s1"&gt;'refused'&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'pending'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'pending'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;// network error: keep it, try next time&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv8vjc9yl72c0gh3lx19t.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv8vjc9yl72c0gh3lx19t.webp" alt="A 2xx response marks an item synced, a 4xx refused, and a 5xx or network error leaves it pending" width="800" height="760"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What a response does to a queued item.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A 5xx or a dropped connection is temporary, so the item stays pending. A 4xx means the server understood and said no, and retrying the same request forever would just burn battery. Refused items are never deleted. The Sync Center shows queued, refused and synced counts and says "Nothing is discarded silently. Refused items stay here until they are resolved."&lt;/p&gt;

&lt;p&gt;The function takes &lt;code&gt;fetch&lt;/code&gt; as a parameter, which made it easy to test without a network. A test feeds it a fake fetch that returns 503 and checks the evidence is still pending afterwards.&lt;/p&gt;

&lt;h2&gt;
  
  
  The server side: replay, then merge
&lt;/h2&gt;

&lt;p&gt;On the server, each evidence save is idempotent per tenant. A key that was seen before returns the stored response. A new one merges into the inspection's evidence in a single statement:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;packages/db/src/postgres.ts&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sb"&gt;`${tenantId}:${idempotencyKey}`&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;prior&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="sb"&gt;`select response_json from idempotency_keys where key = ${key}`&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prior&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;prior&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;response_json&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;evidence&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;itemKey&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;evidence&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="sb"&gt;`
  insert into inspections (id, tenant_id, claim_id, evidence_json)
  values (${inspectionId}, ${tenantId}, ${claimId}, ${evidence})
  on conflict (id) do update set evidence_json = inspections.evidence_json || ${evidence}, updated_at = now()
  where inspections.tenant_id = ${tenantId}
  returning *`&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'NOT_FOUND'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;// another tenant's inspection looks like "not found"&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="sb"&gt;`insert into idempotency_keys (key, response_json) values (${key}, ${sql.json(rows[0])})`&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdh6vc0q9fz3p96tgte9z.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdh6vc0q9fz3p96tgte9z.webp" alt="Server flow: replay a stored response for a seen key, otherwise merge with a tenant check and store a receipt" width="799" height="692"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The server replays first, then merges.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The tenant check sits in the &lt;code&gt;WHERE&lt;/code&gt; of the upsert itself, so a forged inspection ID from another tenant cannot be written to, and the error does not reveal that it exists.&lt;/p&gt;

&lt;p&gt;Two honest notes on this code. First, the replay check and the receipt insert are separate statements. Two truly simultaneous retries could both pass the check, and the second would fail on the unique key. The phone treats that as a 5xx, keeps the item pending, and the next sync gets the stored replay, so it heals itself. It is still better written as &lt;code&gt;insert ... on conflict do nothing&lt;/code&gt; in the same transaction, which is what other mutations in the codebase do. Second, &lt;code&gt;||&lt;/code&gt; on JSONB is a shallow merge, so each evidence item is last-write-wins as a whole object.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading offline, too
&lt;/h2&gt;

&lt;p&gt;Writes are only half of it. The assignment list loads from the device cache first and refreshes behind it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;refreshAssignments&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cached&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;load&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;assignments&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;assignments&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cached&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Showing cached assignments. Pull to retry.'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On launch the app reads the session, assignments, outbox and device ID from storage in parallel and paints immediately. Sign-in is the only step that genuinely needs signal. Sessions are bound to the device: the server stores a hash of the token and refuses a field session whose &lt;code&gt;x-device-id&lt;/code&gt; header does not match.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three bugs worth knowing about before you build this
&lt;/h2&gt;

&lt;p&gt;Being strict about "what does the server actually have" surfaced problems that a happy-path demo hides.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F43i01oi4q7hy0m7mhyd5.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F43i01oi4q7hy0m7mhyd5.webp" alt="Three different names for one evidence item, replaced by one shared enum" width="800" height="830"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;One item, three names.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One item, three names.&lt;/strong&gt; The phone sent the UI label as the item key ("Ridge &amp;amp; hip"). The seed data used &lt;code&gt;ridge-hip&lt;/code&gt;. The rule packs that decide whether a claim is complete checked &lt;code&gt;component_photos&lt;/code&gt;. Each layer worked on its own, and the rule check could not see evidence captured on the phone. The fix is boring and permanent: one enum of item keys in the shared contracts package, imported by the app, the seed and the rule packs, with a test that fails on drift.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Counts that do not add up.&lt;/strong&gt; The app sends &lt;code&gt;photos: 1&lt;/code&gt; with each save, and the server's shallow merge replaces the whole item. Three photos of the ridge end up recorded as one, and a rule needing two can never be met from the phone. A later photo without a caption also wipes the earlier caption. The fix is to make the server count: increment per unique media hash instead of trusting the client's number.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1gdid0cxukzoe7rd9ekh.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1gdid0cxukzoe7rd9ekh.webp" alt="Three ridge photos recorded as one because each save sends photos: 1, fixed by counting unique hashes on the server" width="800" height="620"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Three photos, counted once.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Metadata ahead of the file.&lt;/strong&gt; Evidence records can reach the server while the photo itself is still on the phone. Right now photo uploads are refused on purpose, because resumable object storage is not set up yet, and the Sync Center shows those as refused instead of pretending. If you build this, decide early which of the two is the source of truth and show the gap to the user.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is still on the list
&lt;/h2&gt;

&lt;p&gt;This is a pilot, and these are the gaps:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Sync is manual. Nothing happens until the inspector taps Sync. Triggering it when the app comes to the foreground or connectivity returns is the obvious next step.&lt;/li&gt;
&lt;li&gt;There is no backoff, timeout or batching yet. One request at a time is fine for dozens of items, not thousands.&lt;/li&gt;
&lt;li&gt;The queue is one JSON array in AsyncStorage, rewritten on every change. SQLite or per-item keys would scale better.&lt;/li&gt;
&lt;li&gt;Resized photos live in a cache directory the OS may clear. They should be copied into the app's document directory.&lt;/li&gt;
&lt;li&gt;The server exposes a time endpoint for clock offset, but the app does not use it yet, so the phone's clock is trusted.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Capture locally, always&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Write the file and a queue entry before touching the network. Tell the user the original stays on the device until confirmed.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Mint idempotency keys on the device&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Every operation gets a key before its first attempt, and the server replays the stored answer for a key it has seen.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Let the status code decide&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;2xx is done, 4xx is refused and visible, 5xx and network errors stay pending.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Never drop silently&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A screen that shows queued, refused and synced counts is worth more than any retry algorithm.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Share the contract&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Item keys, operation shapes and states belong in one package used by every layer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  How do you build an offline-first React Native app that uploads photos later?
&lt;/h3&gt;

&lt;p&gt;Write locally first, send later, and make sending safe to repeat. In ClaimLayer's field app, taking a photo resizes it, hashes it and queues two operations: one for the file and one for its evidence record. Each operation gets an idempotency key on the device before any network call, so a retry never creates a duplicate.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why use idempotency keys in an offline sync queue?
&lt;/h3&gt;

&lt;p&gt;Idempotency keys make retries safe, because the server replays the stored answer for a key it has already seen. In ClaimLayer, the phone mints a key per operation before its first attempt, and the server stores a receipt per tenant. If a sync is interrupted and repeated, the same evidence is not saved twice.&lt;/p&gt;

&lt;h3&gt;
  
  
  How should an offline queue handle 4xx and 5xx responses?
&lt;/h3&gt;

&lt;p&gt;Let the status code decide. A 2xx marks the item synced, a 5xx or dropped connection leaves it pending because the problem is temporary, and a 4xx marks it refused because the server understood and said no. Retrying a refused request forever only burns battery, so refused items stay visible instead of being deleted.&lt;/p&gt;

&lt;h3&gt;
  
  
  What are the limits of storing an outbox in AsyncStorage?
&lt;/h3&gt;

&lt;p&gt;AsyncStorage works for a pilot but does not scale. In ClaimLayer the queue is one JSON array rewritten on every change, which is fine for dozens of items and not thousands. SQLite or per-item keys would scale better. Resized photos also sit in a cache directory the OS may clear, so they should be copied into the document directory.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why did three photos of the same roof ridge end up counted as one?
&lt;/h3&gt;

&lt;p&gt;The app sent &lt;code&gt;photos: 1&lt;/code&gt; with every save, and the server's shallow merge replaced the whole evidence item each time. Three ridge photos were recorded as one, so a rule needing two could never be met from the phone. The fix is to have the server count unique media hashes instead of trusting the client's number.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/offline-first-react-native-outbox" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>reactnative</category>
      <category>expo</category>
      <category>offlinefirst</category>
      <category>postgres</category>
    </item>
    <item>
      <title>Nemotron 3.5 Parse vs LightOnOCR 3 vs Docling on hard invoices: what the numbers mean</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Sun, 15 Nov 2026 21:38:22 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/nemotron-35-parse-vs-lightonocr-3-vs-docling-on-hard-invoices-what-the-numbers-mean-538k</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/nemotron-35-parse-vs-lightonocr-3-vs-docling-on-hard-invoices-what-the-numbers-mean-538k</guid>
      <description>&lt;p&gt;NVIDIA released a small document parser in September, &lt;strong&gt;Nemotron 3.5 Parse&lt;/strong&gt;: 631 million parameters, free to download, and top of its own ParseBench table. A few weeks later Laxmi Kant of KGP Talkie Hindi put it through &lt;a href="https://www.instagram.com/reel/De3tSWKtrY8/" rel="noopener noreferrer"&gt;a test I liked&lt;/a&gt;: two deliberately hard scanned invoices, one GST invoice and one in Hindi, run through Nemotron, &lt;strong&gt;LightOnOCR 3&lt;/strong&gt; and &lt;strong&gt;Docling&lt;/strong&gt; on a MacBook Pro M5 Max.&lt;/p&gt;

&lt;p&gt;The result was not what NVIDIA's table would predict. LightOnOCR 3 read almost everything. Nemotron dropped a column and the grand total. Docling turned Hindi into nonsense.&lt;/p&gt;

&lt;p&gt;The reel is under three minutes, so it shows the scores and not much of the why. I spent an evening on the why: the model cards, Docling's source, and a small reproduction of the Hindi failure on my own Mac. This post covers what the numbers mean and what I would do if I were building invoice extraction today.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzd5113aj360clvv1rpto.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzd5113aj360clvv1rpto.webp" alt="Bar chart of accuracy: Docling 93.7 percent on the GST invoice and 81.7 percent on the Hindi one, Nemotron 3.5 Parse 97 and 87.8 percent, LightOnOCR 3 100 and 99.4 percent." width="800" height="356"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Numbers from the KGP Talkie Hindi reel. I did not rerun the three models.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The test in the reel
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Documents:&lt;/strong&gt; a two-page GST invoice and a Hindi invoice, printed, scanned, tilted, noisy and folded.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What was checked:&lt;/strong&gt; 86 key fields (invoice numbers, tax IDs, totals, bank details) plus every table cell, 689 checks in all.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The strict rule:&lt;/strong&gt; a cell only counts if the value lands in the right row. Reading 12,510.36 correctly but putting it on the carton row instead of the HDPE jar row is a miss.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hardware:&lt;/strong&gt; MacBook Pro M5 Max, every page timed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That row rule matters more than it sounds. Most OCR scores count characters. An accounts team cares whether the amount sits next to the right line item, because that is what gets posted to the ledger.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three results, and what went wrong in each
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Docling: fast, and blind to Devanagari
&lt;/h3&gt;

&lt;p&gt;Docling took about 12 seconds a page and scored 93.7% on the GST invoice and 81.7% on the Hindi one. The reel shows one line that explains most of the gap: the printed text &lt;code&gt;निर्माण / Mfg&lt;/code&gt; came out as &lt;code&gt;fARFUT / Mfg&lt;/code&gt;. The English half survived. The Hindi half was read as if it were Latin letters.&lt;/p&gt;

&lt;p&gt;That is not really Docling's layout model failing. Docling hands the text recognition to a separate OCR engine, and the default choice depends on your machine. Here is the selection order from &lt;code&gt;OcrAutoOptions&lt;/code&gt; in Docling's current source:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu2n7dxx2iaje3nvt0c15.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu2n7dxx2iaje3nvt0c15.webp" alt="Flowchart of Docling's automatic OCR choice: OcrMac on macOS first, then Nemotron OCR on Linux with CUDA, then RapidOCR with onnxruntime, then EasyOCR." width="800" height="356"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On a Mac the first match is OcrMac, which is Apple's Vision framework. So I asked Vision on my own machine (macOS 15.7) which languages its accurate recognizer supports:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;langs.swift&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;import Vision
let r = VNRecognizeTextRequest()
r.recognitionLevel = .accurate
print(try r.supportedRecognitionLanguages().joined(separator: " "))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;en-US fr-FR it-IT de-DE es-ES pt-BR zh-Hans zh-Hant yue-Hans yue-Hant ko-KR ja-JP ru-RU uk-UA th-TH vi-VT ar-SA ars-SA
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No Hindi. To see what that does to an invoice, I rendered three lines in a Devanagari font, including the same grand total that appears in the reel, and ran Vision on them two ways: with Docling's OcrMac default languages, and with automatic language detection.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9g6wwsc9qjkgru7n7nue.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9g6wwsc9qjkgru7n7nue.webp" alt="Three rendered lines of Hindi: निर्माण / Mfg, कुल राशि ₹6,28,866.00, and the amount in words." width="800" height="206"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The test image: Kohinoor Devanagari, black on white, no noise at all.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;output&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;langs=en-US,es-ES,fr-FR,de-DE
   Mg
   del T: 76,28,866.00
langs=auto
   Mg
   엽의 시 생6,28,866.00
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a clean image, with no tilt, no fold and no scanner noise, and it still fails badly. The whole amount-in-words line is gone. Worse, the rupee sign was read as a &lt;strong&gt;7&lt;/strong&gt;, so ₹6,28,866.00 became 76,28,866.00. That is a number that parses fine, passes a type check and is wrong by 70 lakh. It is the kind of error that reaches a ledger.&lt;/p&gt;

&lt;p&gt;The fix is to choose an engine that has a Devanagari model instead of taking the default. EasyOCR has one. I wrapped the same image in a PDF and ran this with Docling 2.150.1:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;docling_hindi.py&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;docling&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;datamodel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;base_models&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;InputFormat&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;docling&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;datamodel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipeline_options&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;EasyOcrOptions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;OcrMode&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;PdfPipelineOptions&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;docling&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;document_converter&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;DocumentConverter&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;PdfFormatOption&lt;/span&gt;

&lt;span class="n"&gt;opts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;PdfPipelineOptions&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;do_ocr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="c1"&gt;# Scans have no text layer, so OCR the whole page. "hi" pulls in EasyOCR's Devanagari model.&lt;/span&gt;
    &lt;span class="c1"&gt;# The default threshold (0.5) dropped two of my three lines; 0.1 kept them.&lt;/span&gt;
    &lt;span class="n"&gt;ocr_options&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;EasyOcrOptions&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lang&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"hi"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"en"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;OcrMode&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;FULL_PAGE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;confidence_threshold&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;converter&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;DocumentConverter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;format_options&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nc"&gt;InputFormat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;PDF&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;PdfFormatOption&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pipeline_options&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;converter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;convert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"hindi.pdf"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;export_to_markdown&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;output&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;निर्माण Mfg

- कुल रााशिः २6,28,866.00

छह लाख अट्ठाईस हज़ार आठ सौ छियासठ रुपये
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Much better: the Hindi words are all there, including the amount in words. But look at the total again. The rupee sign came back as &lt;code&gt;२&lt;/code&gt;, the Devanagari digit two. My first run, with Docling's default confidence threshold of 0.5, also dropped the total line and the amount in words entirely, because EasyOCR scored the total line at only 0.22.&lt;/p&gt;

&lt;p&gt;That &lt;code&gt;२&lt;/code&gt; is a trap in Python specifically. &lt;code&gt;Decimal&lt;/code&gt; accepts any Unicode digit, so strip the commas and you get a clean, wrong number:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;decimal&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;
&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"२628866.00"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'2628866.00'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So with the default engine the total is wrong by 70 lakh, and with the better engine it is wrong by 20 lakh. Neither raises an error. That is why the last section of this post is about checks, not models.&lt;/p&gt;

&lt;h3&gt;
  
  
  Nemotron 3.5 Parse: one pass, and a column goes missing
&lt;/h3&gt;

&lt;p&gt;Nemotron took about 41 seconds a page and scored 97% on GST and 87.8% on Hindi. The misses had a pattern. On page 2 of the GST invoice, items 18 to 28 each lost one number, the last column (&lt;code&gt;Total ₹&lt;/code&gt;), and the grand total of ₹6,28,866.00 and the bank details (IFSC, branch, UPI) were missed too.&lt;/p&gt;

&lt;p&gt;A whole column dropping out of a long table is a table-structure problem rather than a reading problem, and NVIDIA built a specific answer to it. The model card describes three parsing modes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Single pass:&lt;/strong&gt; parse the whole page once.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Double pass:&lt;/strong&gt; use the first result to find tables and charts, parse those crops again at higher detail, and merge the refinements back.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Triple pass:&lt;/strong&gt; also run a separate layout detector at the same time, and refine the union of both sets of table and chart regions, including any that one of them missed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The reel is fair about this: it says Nemotron ran in one pass, while NVIDIA's headline score uses three. That difference is bigger than it looks. Here are the published ParseBench numbers from NVIDIA's own card:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqjnsxx81n4pkhsu0dk0n.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqjnsxx81n4pkhsu0dk0n.webp" alt="Bar chart of published ParseBench scores: Nemotron triple pass 75.20, LightOnOCR-3-4B 75.10, Nemotron double pass 74.92, LightOnOCR-3-0.8B 74.60, Nemotron single pass 72.74." width="799" height="368"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On NVIDIA's own benchmark, single-pass Nemotron (72.74) is behind even the smallest LightOnOCR 3 model (74.60). The reel compared it with LightOn's largest model. At triple pass the two are tied, 75.20 against 75.10. So the reel's result and NVIDIA's table do not contradict each other. They measured different modes.&lt;/p&gt;

&lt;p&gt;I have not run the double pass on these invoices, so I can't say it would recover the missing column. But a column lost inside a long table is the exact case the table-crop refinement exists for, and it is the first thing I would try.&lt;/p&gt;

&lt;h3&gt;
  
  
  LightOnOCR 3: almost perfect, and slow on a laptop
&lt;/h3&gt;

&lt;p&gt;LightOnOCR 3 scored 100% on the GST invoice and 99.4% on the Hindi one: 689 checks, one miss. The miss was in the Hindi amount in words, where &lt;code&gt;हज़ार सैंतीस&lt;/code&gt; came back with two words wrong. It took about 3 minutes 40 seconds a page.&lt;/p&gt;

&lt;p&gt;The reel calls it a 4.5B model, which matches &lt;code&gt;lightonai/LightOnOCR-3-4B&lt;/code&gt;: the Hugging Face API reports 4,539,265,536 parameters, about seven times Nemotron's 631 million. It is built on the Qwen3.5 vision-language architecture, it is Apache-2.0, and it has two modes, an empty prompt for plain transcription and a &lt;code&gt;grounding&lt;/code&gt; prompt that adds a label and bounding box to each block.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the speeds look so different
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5dw4162wyg3nwbcs0ew8.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5dw4162wyg3nwbcs0ew8.webp" alt="Scatter plot of time per page against accuracy: Docling 12 seconds at 90.9 percent, Nemotron 41 seconds at 94.8 percent, LightOnOCR 3 220 seconds at 99.9 percent." width="800" height="372"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The laptop timings tell you more about Apple silicon than about the models. Both model cards are written around NVIDIA GPUs. Nemotron's Transformers example calls &lt;code&gt;.to("cuda")&lt;/code&gt;, and its fast path is vLLM with a small 89.3M-parameter DFlash2 drafter for speculative decoding. NVIDIA measured 18.57 pages per second in single pass on one H200, against 2.76 for LightOnOCR-3-4B in the same table.&lt;/p&gt;

&lt;p&gt;So on GPU servers the gap goes the same way, Nemotron is several times faster, but nobody should expect 220 seconds a page from LightOn in production. That is why the reel's closing question, running all three on an RTX 5090, is the one I would want answered too.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would do with this
&lt;/h2&gt;

&lt;p&gt;The lesson I take is not "use LightOn". It is that every one of these tools fails in a way that looks like a real number. A dropped total, a ₹ read as 7 or as २, a value on the wrong row. You catch those by checking the invoice against itself, which an Indian GST invoice makes easy because it is full of arithmetic.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fou6jfteu6c5r5vztvzvz.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fou6jfteu6c5r5vztvzvz.webp" alt="Flow diagram: parser output is checked for row math, rows adding up to the grand total, and the amount in words matching the figure. If everything passes it is accepted; if anything fails it goes to a slower model and then a person." width="800" height="227"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Here is the core of the check I would put after any of these parsers. It uses &lt;code&gt;Decimal&lt;/code&gt; so paise do not drift, rejects any digit that is not ASCII, and treats a missing cell as a failure instead of a zero:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;check_invoice.py&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;decimal&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;InvalidOperation&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""'₹6,28,866.00' or '6,28,866' -&amp;gt; Decimal. None when it is not a clean number."""&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
    &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"₹"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;","&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Rs."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# Decimal("२6") is 26: Python accepts any Unicode digit. OCR output must be ASCII or rejected.&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="nb"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ch&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="s2"&gt;"0123456789.-"&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;ch&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;InvalidOperation&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;inv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tol&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"0.05"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;inv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"rows"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;qty&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"qty"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rate"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;taxable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cgst&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sgst&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"taxable"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cgst"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"sgst"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"total"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;qty&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;taxable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cgst&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sgst&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"row {r['sr']}: missing or unreadable cell"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;qty&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;rate&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;taxable&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;tol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"row {r['sr']}: qty x rate = {qty * rate}, taxable says {taxable}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cgst&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;sgst&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;tol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"row {r['sr']}: CGST {cgst} != SGST {sgst}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;taxable&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;cgst&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;sgst&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;tol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"row {r['sr']}: taxable + taxes = {taxable + cgst + sgst}, total says {total}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;grand&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;inv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"grand_total"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;grand&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"grand total missing or unreadable"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;rows_sum&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"total"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;inv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"rows"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows_sum&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;grand&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;tol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"rows add up to {rows_sum}, grand total says {grand}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I tested it on a clean two-row invoice and on the four failure shapes from this post:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;output&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;clean: ok
dropped cell: ['row 2: missing or unreadable cell']
₹ read as 7: ['rows add up to 26644.40, grand total says 726644.40']
no grand total: ['grand total missing or unreadable']
₹ read as २: ['grand total missing or unreadable']
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With that in place, the speed table starts to matter more than the accuracy table. Run the fast parser first, Nemotron in double pass or Docling with the right OCR engine, and only send pages that fail the checks to LightOnOCR 3. If most of your invoices are clean, you get close to LightOn's accuracy at a fraction of its cost per page. Add a check that the amount in words matches the figure for Hindi invoices, since that is where even LightOn slipped.&lt;/p&gt;

&lt;h2&gt;
  
  
  How far to trust these numbers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Two invoices.&lt;/strong&gt; 689 checks is a lot of cells, but it is two documents made by one person. It shows how each tool fails. It does not rank them in general.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One Nemotron mode.&lt;/strong&gt; Single pass is the fastest and weakest mode. Double and triple pass were not tested.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A Mac, not a GPU server.&lt;/strong&gt; The timings are real for that laptop and say little about throughput in production.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;My reproduction is small.&lt;/strong&gt; I verified the Docling and Apple Vision behaviour myself. I did not rerun Nemotron or LightOnOCR on the invoices.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your documents are Indian invoices, the cheapest thing you can do is what the reel did: make ten ugly scans of your own, write down the right answers, and score every tool on cells in the right row. It takes an afternoon and it will tell you more than any leaderboard.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is Nemotron 3.5 Parse better than LightOnOCR 3?
&lt;/h3&gt;

&lt;p&gt;On NVIDIA's published ParseBench, triple-pass Nemotron scores 75.20 and LightOnOCR-3-4B scores 75.10, so they are effectively tied, and Nemotron is several times faster on a GPU. In single pass Nemotron drops to 72.74. In the KGP Talkie Hindi invoice test, single-pass Nemotron scored 94.8% and LightOnOCR 3 scored 99.9%.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why can't Docling read Hindi?
&lt;/h3&gt;

&lt;p&gt;Docling sends text recognition to a separate OCR engine, and on macOS its automatic choice is Apple Vision. On macOS 15.7, Vision's accurate recognizer lists 18 languages and Hindi is not one of them. Configure Docling with an engine that has a Devanagari model, such as EasyOCR with &lt;code&gt;lang=["hi", "en"]&lt;/code&gt;, and still validate the numbers: in my test it read the rupee sign as the Devanagari digit २.&lt;/p&gt;

&lt;h3&gt;
  
  
  How big is Nemotron 3.5 Parse?
&lt;/h3&gt;

&lt;p&gt;It has 631 million parameters, with an optional 89.3M-parameter DFlash2 drafter for faster decoding in vLLM. It is released under the OpenMDW 1.1 license. LightOnOCR-3-4B has about 4.54 billion parameters and is Apache-2.0.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which OCR model should I use for GST invoices?
&lt;/h3&gt;

&lt;p&gt;Use a fast parser first and check the result with the invoice's own arithmetic: quantity times rate, CGST equal to SGST, rows adding up to the grand total. Send only the pages that fail to a slower, more accurate model like LightOnOCR 3, and then to a person.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/nemotron-3-5-parse-vs-lightonocr-3-vs-docling-invoices" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ocr</category>
      <category>python</category>
      <category>ai</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Will this LLM run on my laptop? The memory math to do before you download</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Sun, 15 Nov 2026 21:23:48 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/will-this-llm-run-on-my-laptop-the-memory-math-to-do-before-you-download-33pc</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/will-this-llm-run-on-my-laptop-the-memory-math-to-do-before-you-download-33pc</guid>
      <description>&lt;p&gt;The usual way people pick a local model goes like this. A model is trending, you pull 18 GB of weights over a slow connection, you start it, and it types four words a second while your fans spin up. You decide local LLMs are for people with a 4090 and go back to the API.&lt;/p&gt;

&lt;p&gt;I have done exactly that on my own laptop, an M1 Pro with 16 GB of RAM. What I do now is a few minutes of arithmetic first. Whether a model runs on a machine, and roughly how fast, is not a matter of opinion. It falls out of two numbers: how many bytes the model needs to keep in memory, and how many bytes it has to read for every token it writes.&lt;/p&gt;

&lt;p&gt;This post is that arithmetic, worked through with real model configs, plus the llama.cpp flags that act on each part of it. Every number below comes from a small Python script you can copy and point at your own hardware.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two numbers that decide everything
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Memory&lt;/strong&gt; decides whether the model loads at all. It is the weights, plus the KV cache for your context, plus a little working space for the runtime. If that total fits in the memory your GPU can use, everything runs on the GPU. If it doesn't, some of it moves somewhere slower, and the speed drops off a cliff.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bandwidth&lt;/strong&gt; decides how fast it can generate. To produce one token, the model reads every weight it uses out of memory. Every single token. The math inside is cheap next to that read, so generation speed has a hard ceiling:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;the rule&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tokens per second (ceiling) = memory bandwidth / bytes read per token
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Real speed lands under that ceiling, never over it. But the ceiling is enough to rule things in or out before you download anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  Part one: the weights
&lt;/h2&gt;

&lt;p&gt;A model file is its parameter count times the bytes per parameter. Quantization is what changes the second number. These are the effective bits per weight for the GGUF formats llama.cpp, Ollama and LM Studio load, including the small per-block scale factors each format stores:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;bits per weight&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;F16      16.0  bits/weight   ~2.0  GB per billion params
Q8_0      8.5  bits/weight   ~1.06 GB per billion
Q6_K      6.56 bits/weight   ~0.82 GB per billion
Q4_K_M    4.85 bits/weight   ~0.61 GB per billion
MXFP4     4.25 bits/weight   ~0.53 GB per billion  (gpt-oss experts)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So Qwen3-8B at Q4_K_M is about 8.2 x 0.61 = 5.0 GB, and Qwen3-14B is about 9.0 GB. Q4_K_M is the default for a reason: it keeps most of the quality at under a third of the F16 size. Go to Q5 or Q6 when you have room and the work is exact (code, JSON, maths). Below Q4 is for squeezing a model in, and I wouldn't trust it for tool calls, where one broken quote in the JSON is a failed request.&lt;/p&gt;

&lt;h2&gt;
  
  
  Part two: the KV cache, which nobody budgets for
&lt;/h2&gt;

&lt;p&gt;While a model reads your prompt, every layer stores a key and a value vector for every token, so it doesn't have to recompute them for the next token. That store is the KV cache, and it grows linearly with context length. On a long context it can be as big as the weights.&lt;/p&gt;

&lt;p&gt;The size comes straight from the model's &lt;code&gt;config.json&lt;/code&gt;:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;kv cache&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;KV bytes = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_number x context_tokens
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can read those fields without downloading the model:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ curl -sL https://huggingface.co/Qwen/Qwen3-14B/resolve/main/config.json \
    &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s2"&gt;"import json,sys; c=json.load(sys.stdin); print({k: c[k] for k in ['num_hidden_layers','num_key_value_heads','head_dim']})"&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="s1"&gt;'num_hidden_layers'&lt;/span&gt;: 40, &lt;span class="s1"&gt;'num_key_value_heads'&lt;/span&gt;: 8, &lt;span class="s1"&gt;'head_dim'&lt;/span&gt;: 128&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For Qwen3-14B with an F16 cache, that is 2 x 40 x 8 x 128 x 2 bytes = 163,840 bytes, or 160 KB, for every token of context. At 32k tokens the cache alone is 5.4 GB, more than half the size of the weights.&lt;/p&gt;

&lt;p&gt;Two details in the config change this a lot. The first is &lt;code&gt;num_key_value_heads&lt;/code&gt;. Qwen3-14B has 40 attention heads but only 8 KV heads (grouped-query attention), so its cache is five times smaller than it would be with one KV head per attention head. The second is sliding-window attention. gpt-oss-20b alternates layers: 12 use full attention, and 12 only look back 128 tokens, so half its layers have a cache that never grows past 128 tokens. That is why its 32k cache is about 0.8 GB while Qwen3-14B's is 5.4 GB.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Mac trap: 16 GB of RAM is not 16 GB of VRAM
&lt;/h2&gt;

&lt;p&gt;Apple Silicon has unified memory, so the GPU and CPU share the same RAM. That is great for local models, but macOS does not let the GPU take all of it. llama.cpp will tell you the real budget:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ llama-server --list-devices
  MTL0: Apple M1 Pro &lt;span class="o"&gt;(&lt;/span&gt;10922 MiB, 10922 MiB free&lt;span class="o"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On my 16 GB machine Metal allows 10,922 MiB, about 11.5 GB. That is the number to budget against, not 16. Anything past it either falls back to the CPU or fails to load.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl8p9wmvl2mvadb18kwow.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl8p9wmvl2mvadb18kwow.webp" alt="Bar chart of a 16 GB M1 Pro. Metal allows about 11.5 GB for the GPU. Qwen3-8B at Q4_K_M with a 32k context needs about 10.8 GB and fits. Qwen3-14B at Q4_K_M with a 32k context needs about 15.4 GB and spills past the limit." width="800" height="350"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Weights, KV cache and about 1 GB of compute buffers, against the Metal limit on a 16 GB M1 Pro.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You can raise the limit. On recent macOS versions the GPU wired-memory cap is a sysctl, and &lt;code&gt;0&lt;/code&gt; means the default:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ sysctl iogpu.wired_limit_mb
iogpu.wired_limit_mb: 0

&lt;span class="c"&gt;# allow the GPU 13 GB until the next reboot&lt;/span&gt;
$ sudo sysctl iogpu.wired_limit_mb=13312
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Warning:&lt;/strong&gt; This resets on reboot, which is the safe behaviour. Leave at least 3 GB for macOS and your other apps. If you push it to 15 GB on a 16 GB machine, the system starts swapping, and a swapping Mac running a model is slower than one that just used a smaller model.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  A script that answers it for any model
&lt;/h2&gt;

&lt;p&gt;Put the three parts together and you have the whole check. This is the script I use. Add a model by copying five numbers out of its config.json:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;fit.py&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# fit.py: will this model run on my machine, and how fast can it possibly go?&lt;/span&gt;
&lt;span class="c1"&gt;# Numbers come from each model's config.json on Hugging Face.&lt;/span&gt;

&lt;span class="no"&gt;BPW&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"F16"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;16.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Q8_0"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;8.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Q6_K"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;6.56&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Q4_K_M"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;4.85&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"MXFP4"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;4.25&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="no"&gt;KV_BYTES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"f16"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"q8_0"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0625&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;  &lt;span class="c1"&gt;# bytes per cached number&lt;/span&gt;

&lt;span class="no"&gt;MODELS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;#                total B, active B, layers, kv_heads, head_dim, full-attention layers&lt;/span&gt;
    &lt;span class="s2"&gt;"Qwen3-8B"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;      &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;8.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="n"&gt;active&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;8.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;36&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;full&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;36&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"Qwen3-14B"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;     &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;14.8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;active&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;14.8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;full&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"Qwen3-32B"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;     &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;32.8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;active&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;32.8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;full&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"Qwen3-30B-A3B"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;30.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;active&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;3.3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;48&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;full&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;48&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"gpt-oss-20b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;20.9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;active&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;3.6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="n"&gt;full&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;weights_gb&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;quant&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"total"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="no"&gt;BPW&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;quant&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;kv_gb&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"f16"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;per_token_layer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"kv"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"hd"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="no"&gt;KV_BYTES&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;kv_type&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;  &lt;span class="c1"&gt;# K and V&lt;/span&gt;
    &lt;span class="n"&gt;full&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"full"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;
    &lt;span class="n"&gt;sliding&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"layers"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"full"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"window"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;per_token_layer&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;full&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;sliding&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e9&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ceiling_tps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;quant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bandwidth_gbs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;bytes_per_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"active"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="no"&gt;BPW&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;quant&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;  &lt;span class="c1"&gt;# GB read for every token&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;bandwidth_gbs&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;bytes_per_token&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"__main__"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="no"&gt;BUDGET_GB&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10922&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1.048576&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;  &lt;span class="c1"&gt;# what Metal reports as usable on a 16 GB M1 Pro&lt;/span&gt;
    &lt;span class="no"&gt;BW&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;  &lt;span class="c1"&gt;# GB/s, M1 Pro&lt;/span&gt;
    &lt;span class="no"&gt;OVERHEAD&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;  &lt;span class="c1"&gt;# compute buffers and runtime, GB&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"GPU budget: {BUDGET_GB:.1f} GB, bandwidth {BW} GB/s\n"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{'model':15} {'quant':7} {'weights':&amp;gt;8} {'kv@8k':&amp;gt;6} {'kv@32k':&amp;gt;7} {'total@32k':&amp;gt;9} {'fits':&amp;gt;5} {'ceiling':&amp;gt;9}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;MODELS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;quant&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"MXFP4"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"gpt-oss"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"Q4_K_M"&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k32&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;weights_gb&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;quant&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;kv_gb&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8192&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;kv_gb&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32768&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;k32&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="no"&gt;OVERHEAD&lt;/span&gt;
        &lt;span class="n"&gt;fits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"yes"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="no"&gt;BUDGET_GB&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"no"&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{name:15} {quant:7} {w:7.1f}G {k8:5.2f}G {k32:6.2f}G {total:8.1f}G {fits:&amp;gt;5} {ceiling_tps(m, quant, BW):6.0f} t/s"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;MODELS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Qwen3-14B"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"\nQwen3-14B KV at 32k: f16 {kv_gb(m, 32768):.2f} GB, q8_0 {kv_gb(m, 32768, 'q8_0'):.2f} GB"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Its output for my M1 Pro, at 200 GB/s of memory bandwidth:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ python3 fit.py
GPU budget: 11.5 GB, bandwidth 200 GB/s

model           quant    weights  kv@8k  kv@32k total@32k  fits   ceiling
Qwen3-8B        Q4_K_M      5.0G  1.21G   4.83G     10.8G   yes     40 t/s
Qwen3-14B       Q4_K_M      9.0G  1.34G   5.37G     15.3G    no     22 t/s
Qwen3-32B       Q4_K_M     19.9G  2.15G   8.59G     29.5G    no     10 t/s
Qwen3-30B-A3B   Q4_K_M     18.5G  0.81G   3.22G     22.7G    no    100 t/s
gpt-oss-20b     MXFP4      11.1G  0.20G   0.81G     12.9G    no    105 t/s

Qwen3-14B KV at 32k: f16 5.37 GB, q8_0 2.85 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few things jump out before I've downloaded a byte:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qwen3-8B fits with a full 32k context. It is the safe pick for this machine.&lt;/li&gt;
&lt;li&gt;Qwen3-14B does not fit at 32k, but at 8k it needs 9.0 + 1.3 + 1.0 = 11.3 GB, right at the edge. With a q8_0 cache it has a little room. The context length you ask for is the difference between fitting and not.&lt;/li&gt;
&lt;li&gt;Qwen3-32B is out. Even if it loaded, the ceiling is 10 tokens a second.&lt;/li&gt;
&lt;li&gt;The two MoE models are the strange rows. They are bigger than Qwen3-14B, yet their speed ceiling is four to five times higher.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The script's gpt-oss-20b weight figure is low: only its expert weights are MXFP4, and the real file is about 12 GB. It still misses the default budget by a GB or two, which is exactly the kind of near miss the next sections fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  Part three: bandwidth, and why everyone says buy a GPU
&lt;/h2&gt;

&lt;p&gt;Here are rough peak bandwidth numbers for common hardware. These are the spec sheet figures, and real reads get maybe 70 to 90 percent of them:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;memory bandwidth&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Dual-channel DDR5-5600 desktop     ~90 GB/s
Apple M1 Pro / M2 Pro              200 GB/s
Apple M4 Pro                       273 GB/s
RTX 3060 12 GB                     360 GB/s
Apple M4 Max (top bin)             546 GB/s
RTX 4090                         1,008 GB/s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run the dense Qwen3-14B through the formula. It reads all 9 GB for every token. On my Mac that caps it at 200 / 9 = 22 tokens per second. On a desktop with no GPU, 90 / 9 = 10. On a 4090, about 110. That spread is the whole reason GPUs dominate local inference: they are not mostly faster at maths here, they have far wider memory buses.&lt;/p&gt;

&lt;h2&gt;
  
  
  The exception: mixture-of-experts models
&lt;/h2&gt;

&lt;p&gt;Most new open models are mixture-of-experts (MoE). Each layer has many small feed-forward networks, the experts, and a router picks a few of them for each token. Qwen3-30B-A3B has 128 experts per layer and uses 8. The name tells you both numbers: 30B parameters in total, about 3B active per token.&lt;/p&gt;

&lt;p&gt;That splits the earlier math in two:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Memory is set by total parameters.&lt;/strong&gt; Every expert has to live somewhere, because the next token might pick any of them. Qwen3-30B-A3B is still an 18.5 GB file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Speed is set by active parameters.&lt;/strong&gt; Each token reads only attention, the shared weights and its chosen experts: about 2 GB for Qwen3-30B-A3B at Q4_K_M. 200 / 2 gives a ceiling near 100 tokens per second.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Funm6allkxp6eivr3lyfk.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Funm6allkxp6eivr3lyfk.webp" alt="Dense Qwen3-14B reads all 9 GB per token, a ceiling of about 22 tokens per second at 200 GB/s. MoE Qwen3-30B-A3B reads about 2 of its 18.5 GB per token, a ceiling of about 100 tokens per second." width="800" height="300"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Same bandwidth, very different reads per token.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;This changes where the weights can live. A dense model punishes you for keeping any of it in slow memory, because every token touches all of it. An MoE model only touches a small slice per token, so the bulk of it, the experts, can sit in slower memory and only the slice it needs gets read. The parts every token uses (attention, the shared layers, the KV cache) stay on the fast side.&lt;/p&gt;

&lt;h2&gt;
  
  
  Putting experts in system RAM with llama.cpp
&lt;/h2&gt;

&lt;p&gt;llama.cpp has flags for exactly this. From the help text of the build on my machine (version 0.6.0, build 11429):&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ llama-server --help &lt;span class="o"&gt;|&lt;/span&gt; grep -A1 moe
-cmoe, --cpu-moe          keep all Mixture of Experts &lt;span class="o"&gt;(&lt;/span&gt;MoE&lt;span class="o"&gt;)&lt;/span&gt; weights &lt;span class="k"&gt;in&lt;/span&gt; the CPU
-ncmoe, --n-cpu-moe N     keep the Mixture of Experts &lt;span class="o"&gt;(&lt;/span&gt;MoE&lt;span class="o"&gt;)&lt;/span&gt; weights of the first N layers &lt;span class="k"&gt;in&lt;/span&gt; the CPU
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On a PC with a discrete GPU, the usual setup is to ask for everything on the GPU and then send some layers' experts back to system RAM:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# a 30B-A3B MoE on a 12 GB card with 32 GB of system RAM&lt;/span&gt;
$ llama-server -hf unsloth/Qwen3-30B-A3B-GGUF:Q4_K_M \
    -ngl all --n-cpu-moe 30 -c 32768 -fa on --port 8080
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;-ngl all&lt;/code&gt; puts every layer on the GPU, and &lt;code&gt;--n-cpu-moe 30&lt;/code&gt; is the exception: the experts of the first 30 of its 48 layers stay in system RAM, where the CPU computes them. Lower N until the GPU is nearly full; each step moves one more layer of experts onto the card. The best value depends on your card, your CPU and your RAM speed, so try three values and keep the fastest.&lt;/p&gt;

&lt;p&gt;On a Mac the trade is different, and it took me a minute to see why. With unified memory, the experts are in the same RAM either way. What &lt;code&gt;--n-cpu-moe&lt;/code&gt; changes is who computes them: CPU-side weights don't count against the Metal limit, so it frees GPU budget, at the cost of the CPU cores doing that work with less bandwidth than the GPU gets. For gpt-oss-20b on a 16 GB Mac, that is the fix for the near miss above:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# gpt-oss-20b: about 12 GB of weights, 24 layers&lt;/span&gt;
$ llama-server -hf ggml-org/gpt-oss-20b-GGUF \
    -c 16384 --n-cpu-moe 4 -fa on --port 8080
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each of its layers holds roughly 0.45 GB of experts, so four layers on the CPU frees close to 2 GB of Metal budget. The other route is the sysctl from earlier: raise the limit to about 13 GB and keep it all on the GPU. Which one is faster is a measurement, not a guess, and that is what &lt;code&gt;llama-bench&lt;/code&gt; is for.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip:&lt;/strong&gt; This build also has &lt;code&gt;--fit&lt;/code&gt;, on by default, which adjusts any argument you didn't set until the model fits, with a 1 GB margin. It is a good safety net, but it can quietly shrink your context to as little as 4,096 tokens. Read the startup log for &lt;code&gt;n_ctx&lt;/code&gt; before blaming the model for forgetting the start of a long chat.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Shrinking the KV cache
&lt;/h2&gt;

&lt;p&gt;When the weights fit but the total doesn't, cut the cache before you cut the model:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 16k context instead of the model's default, cache stored as 8-bit&lt;/span&gt;
$ llama-server -hf Qwen/Qwen3-14B-GGUF:Q4_K_M \
    -c 16384 -ctk q8_0 -ctv q8_0 -fa on --port 8080
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Set &lt;code&gt;-c&lt;/code&gt; yourself.&lt;/strong&gt; The default is &lt;code&gt;0&lt;/code&gt;, which means the model's full trained context, 40,960 tokens for Qwen3 and 131,072 for gpt-oss. Most chats never need that.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;-ctk q8_0 -ctv q8_0&lt;/code&gt;&lt;/strong&gt; stores keys and values in 8 bits. For Qwen3-14B at 32k, that is 2.85 GB instead of 5.37 GB. Quality loss at q8_0 is small; q4_0 saves more and costs more.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;-fa on&lt;/code&gt;&lt;/strong&gt; turns on flash attention, which a quantized V cache needs and which also cuts the compute buffer.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flx8mue2u0aqytjyamkjh.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flx8mue2u0aqytjyamkjh.webp" alt="Decision flow: if weights plus KV cache plus 1 GB fit the GPU budget, run everything on the GPU. If not, shrink the KV cache. If it still does not fit and the model is MoE, move experts to the CPU with --n-cpu-moe. Otherwise use a smaller quant or a smaller model. Then measure with llama-bench." width="800" height="442"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The order I try things in, cheapest first.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Measure, don't trust the ceiling
&lt;/h2&gt;

&lt;p&gt;The formula tells you what is impossible. &lt;code&gt;llama-bench&lt;/code&gt; tells you what you actually get, and it reports the two phases separately:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ llama-bench -m ~/models/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -fa 1
&lt;span class="c"&gt;# pp512 = prompt processing, tokens/s while reading a 512-token prompt&lt;/span&gt;
&lt;span class="c"&gt;# tg128 = text generation, tokens/s while writing 128 tokens&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those two numbers behave differently, and the difference matters for what you use a local model for. Generation (&lt;code&gt;tg&lt;/code&gt;) is bandwidth-bound, the part this whole post has been about. Prompt processing (&lt;code&gt;pp&lt;/code&gt;) handles many tokens at once, so it is compute-bound, and it is where CPUs and offloaded experts hurt most. A setup that writes at a comfortable speed can still take a long time to read a 20,000-token document before writing anything. If your use is long inputs and short answers, like summarising files or RAG over a repo, measure &lt;code&gt;pp&lt;/code&gt; with your real prompt size, not just &lt;code&gt;tg&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Once a model is running, llama-server speaks the OpenAI chat API, so anything that talks to OpenAI can talk to it:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ curl -s http://localhost:8080/v1/chat/completions \
    -H &lt;span class="s1"&gt;'Content-Type: application/json'&lt;/span&gt; \
    -d &lt;span class="s1"&gt;'{"messages":[{"role":"user","content":"Classify: \"refund not received\". One word."}]}'&lt;/span&gt; \
  &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s2"&gt;"import json,sys; print(json.load(sys.stdin)['choices'][0]['message']['content'])"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Where local still loses
&lt;/h2&gt;

&lt;p&gt;I like running models locally, but I don't want to oversell it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It is a tier below the frontier.&lt;/strong&gt; What fits on a laptop is good at classification, extraction, short summaries and routing. Long agentic coding and hard reasoning still belong to the big hosted models.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is tuned for one user.&lt;/strong&gt; Everything clever above, from expert offload to a quantized cache, assumes a batch of one. The moment several people share the box, you want the whole model on the GPU and a batching server like vLLM.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long prompts are slow.&lt;/strong&gt; See &lt;code&gt;pp&lt;/code&gt; above. Chat feels fine long before document work does.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;RAM is not cheap any more.&lt;/strong&gt; Expert offload assumes spare system memory, and memory prices have climbed hard this year as fabs moved capacity to datacenter HBM. Check what an upgrade costs before planning a build around it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Where it wins is the boring, high-volume work, and work where the data can't leave the machine. I wrote about &lt;a href="https://mahmadmanzoor.com/blog/cut-llm-inference-cost-latency" rel="noopener noreferrer"&gt;cutting LLM API cost&lt;/a&gt; by routing simple calls to small models; a local 8B model is the far end of that idea, with a per-token price of zero. For extraction, a &lt;a href="https://mahmadmanzoor.com/blog/gliner2-vs-llm-extraction-test" rel="noopener noreferrer"&gt;small specialised model&lt;/a&gt; sometimes beats a general LLM outright. And for &lt;a href="https://mahmadmanzoor.com/blog/offline-electron-sqlite-air-gapped" rel="noopener noreferrer"&gt;air-gapped software&lt;/a&gt;, local is the only option there is.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd do this weekend
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Run &lt;code&gt;llama-server --list-devices&lt;/code&gt; (or check your VRAM) and write down the real budget.&lt;/li&gt;
&lt;li&gt;Look up your memory bandwidth. That number caps every model you'll ever run on the machine.&lt;/li&gt;
&lt;li&gt;Copy &lt;code&gt;fit.py&lt;/code&gt;, add the two or three models you are curious about from their config.json, and run it.&lt;/li&gt;
&lt;li&gt;Download only the ones that fit. Start one with an explicit &lt;code&gt;-c&lt;/code&gt; and &lt;code&gt;-fa on&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Run &lt;code&gt;llama-bench&lt;/code&gt; with a prompt size close to your real work. If it is a MoE model, try three values of &lt;code&gt;--n-cpu-moe&lt;/code&gt; and keep the best.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The question used to be which GPU you own. With MoE models and expert offload it is closer to how much memory you have of any kind, and how long you are willing to wait per token. That is a much better question, and you can answer most of it with a calculator.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  How do I know if an LLM will fit in my GPU or Mac memory?
&lt;/h3&gt;

&lt;p&gt;Add three things: the weights (parameters times bytes per weight, about 0.6 GB per billion parameters at Q4_K_M), the KV cache (2 x layers x KV heads x head dim x 2 bytes x context length, from the model's config.json), and about 1 GB for compute buffers. If the total is under your usable GPU memory, it fits. On a Mac, usable means the Metal limit that &lt;code&gt;llama-server --list-devices&lt;/code&gt; prints, which is about two thirds of RAM on a 16 GB machine.&lt;/p&gt;

&lt;h3&gt;
  
  
  How much memory does the KV cache use?
&lt;/h3&gt;

&lt;p&gt;It depends on the model's layer count, KV heads and head size, and it grows linearly with context. Qwen3-14B uses about 160 KB per token with an F16 cache, so 5.4 GB at 32k tokens. An 8-bit cache (&lt;code&gt;-ctk q8_0 -ctv q8_0&lt;/code&gt; in llama.cpp) roughly halves it, and a shorter &lt;code&gt;-c&lt;/code&gt; shrinks it in proportion.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why are MoE models faster than dense models of the same size?
&lt;/h3&gt;

&lt;p&gt;Generation speed is limited by how many bytes are read from memory per token. A dense model reads all its weights for every token. A mixture-of-experts model reads only the experts the router picks, so Qwen3-30B-A3B reads about 2 GB per token at Q4_K_M even though the file is 18.5 GB. Memory still has to hold all of it, but speed follows the active parameters.&lt;/p&gt;

&lt;h3&gt;
  
  
  What does --n-cpu-moe do in llama.cpp?
&lt;/h3&gt;

&lt;p&gt;It keeps the expert weights of the first N layers in system RAM, computed by the CPU, while everything else stays on the GPU. Because each token only reads a few experts, this costs much less speed than offloading whole layers of a dense model. Combine it with &lt;code&gt;-ngl all&lt;/code&gt; and lower N until the GPU is nearly full.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I give the GPU more memory on an Apple Silicon Mac?
&lt;/h3&gt;

&lt;p&gt;Yes. &lt;code&gt;sudo sysctl iogpu.wired_limit_mb=&amp;lt;MB&amp;gt;&lt;/code&gt; raises the GPU wired-memory limit until the next reboot. Leave at least 3 GB for macOS and your apps, or the system starts swapping and everything gets slower.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/will-this-llm-run-on-my-laptop" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>hardware</category>
      <category>llm</category>
      <category>performance</category>
    </item>
    <item>
      <title>Fixing the Vercel AI SDK Bug That Dropped Gemini File Search Citations in streamText</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Fri, 13 Nov 2026 12:35:10 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/fixing-the-vercel-ai-sdk-bug-that-dropped-gemini-file-search-citations-in-streamtext-k29</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/fixing-the-vercel-ai-sdk-bug-that-dropped-gemini-file-search-citations-in-streamtext-k29</guid>
      <description>&lt;p&gt;If you use the Vercel AI SDK with Gemini File Search or a RAG corpus, &lt;code&gt;generateText&lt;/code&gt; gave you citations in &lt;code&gt;result.sources&lt;/code&gt; and &lt;code&gt;streamText&lt;/code&gt; gave you an empty array. Same model, same prompt, same grounding data. The citations were still in the raw provider metadata, so nothing threw and nothing logged. They just never reached the UI.&lt;/p&gt;

&lt;p&gt;I hit this, traced it to one &lt;code&gt;if&lt;/code&gt; in &lt;code&gt;@ai-sdk/google&lt;/code&gt;, and sent the fix upstream. It was merged on 10 November 2026 and shipped in &lt;strong&gt;&lt;code&gt;@ai-sdk/google&lt;/code&gt; 4.0.118&lt;/strong&gt; on 11 November. This post walks through the bug, the fix, the part the maintainer changed in review and why their version is better, and what to do if you are on an older release. The issue is &lt;a href="https://github.com/vercel/ai/issues/24723" rel="noopener noreferrer"&gt;vercel/ai#24723&lt;/a&gt; and the PR is &lt;a href="https://github.com/vercel/ai/pull/24724" rel="noopener noreferrer"&gt;vercel/ai#24724&lt;/a&gt;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip:&lt;/strong&gt; Short version: run &lt;code&gt;npm i @ai-sdk/google@^4.0.118&lt;/code&gt;. After that, &lt;code&gt;streamText&lt;/code&gt; emits File Search and RAG document sources the same way &lt;code&gt;generateText&lt;/code&gt; does.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The symptom
&lt;/h2&gt;

&lt;p&gt;Here is the setup. A route streams a Gemini answer grounded on a File Search store and forwards sources to the client, the way the SDK's own examples do it:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;app/api/chat/route.ts&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;google&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@ai-sdk/google'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;convertToModelMessages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;createUIMessageStreamResponse&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;streamText&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;toUIMessageStream&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;UIMessage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'ai'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nc"&gt;POST&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;UIMessage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;streamText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;google&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'gemini-4-argon'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;file_search&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;google&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fileSearch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fileSearchStoreNames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'fileSearchStores/roof-reports'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;convertToModelMessages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;createUIMessageStreamResponse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;toUIMessageStream&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sendSources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The answer text streamed fine and clearly came from the documents. But no &lt;code&gt;source-document&lt;/code&gt; parts arrived on the client, and &lt;code&gt;await result.sources&lt;/code&gt; on the server was &lt;code&gt;[]&lt;/code&gt;. Swapping &lt;code&gt;streamText&lt;/code&gt; for &lt;code&gt;generateText&lt;/code&gt; with the same inputs returned the citations. The provider docs say File Search responses include citations "via the normal &lt;code&gt;sources&lt;/code&gt; field", so this was a bug, not a missing feature.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the sources come from
&lt;/h2&gt;

&lt;p&gt;Gemini returns grounding as &lt;code&gt;groundingMetadata.groundingChunks&lt;/code&gt;. Each chunk has one of a few shapes: &lt;code&gt;web&lt;/code&gt; for Google Search, &lt;code&gt;image&lt;/code&gt;, &lt;code&gt;maps&lt;/code&gt;, or &lt;code&gt;retrievedContext&lt;/code&gt; for RAG and File Search. A File Search chunk has a &lt;code&gt;fileSearchStore&lt;/code&gt; and no &lt;code&gt;uri&lt;/code&gt;. A Vertex RAG chunk usually has a &lt;code&gt;gs://&lt;/code&gt; URI:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;groundingMetadata (one stream chunk)&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"groundingChunks"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"retrievedContext"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"fileSearchStore"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"fileSearchStores/roof-reports"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"roof-report.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Total roof area: 2,450 square feet."&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"retrievedContext"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"uri"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"gs://bucket/reports/annual.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Annual report"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Revenue grew."&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Inside the provider, &lt;code&gt;extractSources()&lt;/code&gt; turns those chunks into the SDK's source type. &lt;code&gt;web&lt;/code&gt;, &lt;code&gt;image&lt;/code&gt;, &lt;code&gt;maps&lt;/code&gt; and &lt;code&gt;retrievedContext&lt;/code&gt; with an &lt;code&gt;http(s)&lt;/code&gt; URI become &lt;code&gt;sourceType: 'url'&lt;/code&gt;. A &lt;code&gt;retrievedContext&lt;/code&gt; with a &lt;code&gt;gs://&lt;/code&gt; path or a &lt;code&gt;fileSearchStore&lt;/code&gt; becomes &lt;code&gt;sourceType: 'document'&lt;/code&gt;, with a &lt;code&gt;title&lt;/code&gt;, a &lt;code&gt;filename&lt;/code&gt; and a &lt;code&gt;mediaType&lt;/code&gt;. That function was correct. The problem was what the two call paths did with its output.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug: one filter meant for something else
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fubehlxfphbtxu1h3tugn.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fubehlxfphbtxu1h3tugn.webp" alt="Diagram: grounding chunks go through extractSources. doGenerate keeps url and document sources. doStream keeps only url sources, so the gs:// and File Search documents are dropped." width="800" height="391"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Same chunks, two code paths. Only the streaming path filtered by source type.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;doGenerate&lt;/code&gt; pushed every extracted source into the result. &lt;code&gt;doStream&lt;/code&gt; is different because Gemini can repeat the same grounding metadata on several chunks of one response, so it needs to avoid emitting duplicates. The duplicate check looked like this:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;packages/google/src/google-language-model.ts (before)&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;emittedSourceUrls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Set&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// ...inside the stream transform, for each chunk:&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sources&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;extractSources&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;groundingMetadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;candidate&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;groundingMetadata&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;generateId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sources&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="n"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sourceType&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s1"&gt;'url'&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
      &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;emittedSourceUrls&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;emittedSourceUrls&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="n"&gt;controller&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enqueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The intent is "skip a URL we already sent". Document sources have no &lt;code&gt;url&lt;/code&gt; field, so the dedupe was written for URLs only, and the &lt;code&gt;sourceType === 'url'&lt;/code&gt; guard quietly turned it into "send URLs only". Every &lt;code&gt;document&lt;/code&gt; source fell through the &lt;code&gt;if&lt;/code&gt; and was never enqueued. That is why &lt;code&gt;streamText&lt;/code&gt; lost exactly the File Search and &lt;code&gt;gs://&lt;/code&gt; citations while Google Search citations kept working, which made it easy to miss in testing: if you only try web grounding, streaming looks fine.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproducing it without the live API
&lt;/h2&gt;

&lt;p&gt;I wanted a reproduction that used the public &lt;code&gt;streamText&lt;/code&gt; and &lt;code&gt;generateText&lt;/code&gt; APIs, not just a unit test on the model class. My Gemini project had run out of prepaid credits at the time (the API returned HTTP 402 &lt;code&gt;RESOURCE_EXHAUSTED&lt;/code&gt;), so I served a Gemini-shaped response from a local HTTP server and pointed the provider at it with &lt;code&gt;baseURL&lt;/code&gt;. The script imports the packages built from the checkout by absolute path, so no registry version can sneak in:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;repro.mjs&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Run with: AI_REPO=/path/to/ai node repro.mjs&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'node:http'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;pathToFileURL&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'node:url'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;repo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;AI_REPO&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;streamText&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;generateText&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nf"&gt;pathToFileURL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`${repo}/packages/ai/dist/index.js`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;href&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;createGoogle&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nf"&gt;pathToFileURL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`${repo}/packages/google/dist/index.js`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;href&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;candidate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'The roof area is 2,450 sq ft.'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'model'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;finishReason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'STOP'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;groundingMetadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;groundingChunks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;retrievedContext&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;fileSearchStore&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'fileSearchStores/roof-reports'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'roof-report.pdf'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Total roof area: 2,450 square feet.'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;retrievedContext&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'gs://bucket/reports/annual.pdf'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Annual report'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Revenue grew.'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;usageMetadata&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;promptTokenCount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;candidatesTokenCount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;totalTokenCount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;18&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;server&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createServer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;':streamGenerateContent'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeHead&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'content-type'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'text/event-stream'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`data: ${JSON.stringify({ candidates: [candidate], usageMetadata })}\n\n`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeHead&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'content-type'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'application/json'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;candidates&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;candidate&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;usageMetadata&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;server&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;listen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;google&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;createGoogle&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;apiKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'test'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;baseURL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`http://localhost:${server.address().port}/v1beta`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;google&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'gemini-2.5-flash'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'What is the roof area?'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;streamText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;consumeStream&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;srcs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;srcs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="sb"&gt;`${x.sourceType}:${x.title}`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;', '&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="s1"&gt;'(none)'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'generateText sources:'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'streamText sources:  '&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;server&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On &lt;code&gt;main&lt;/code&gt; at the time (commit &lt;code&gt;6d21e8e&lt;/code&gt;, &lt;code&gt;@ai-sdk/google&lt;/code&gt; 4.0.113):&lt;/p&gt;

&lt;p&gt;&lt;code&gt;output, before&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;generateText sources: document:roof-report.pdf, document:Annual report
streamText sources:   (none)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I said plainly in the PR that this proves the SDK's handling end to end but not Google's live streaming payload, and offered to add a live run once credits were back. Reviewers can judge a limit you state. They can't judge one you hide.&lt;/p&gt;

&lt;h2&gt;
  
  
  My fix, and the key I got wrong
&lt;/h2&gt;

&lt;p&gt;The fix is to dedupe every source type instead of filtering to one. My first version kept a set of keys and built the key per type: the &lt;code&gt;url&lt;/code&gt; for URL sources, and &lt;code&gt;title&lt;/code&gt; plus &lt;code&gt;filename&lt;/code&gt; for document sources. I added two &lt;code&gt;doStream&lt;/code&gt; tests, one asserting that File Search and &lt;code&gt;gs://&lt;/code&gt; chunks come out as document sources, and one asserting that the same document repeated on a later chunk is emitted once. Both failed on &lt;code&gt;main&lt;/code&gt; and passed with the change. A patch changeset for &lt;code&gt;@ai-sdk/google&lt;/code&gt; went in with it.&lt;/p&gt;

&lt;p&gt;That key has a hole. &lt;code&gt;filename&lt;/code&gt; for a &lt;code&gt;gs://&lt;/code&gt; source is the last path segment, so &lt;code&gt;gs://bucket/2025/annual.pdf&lt;/code&gt; and &lt;code&gt;gs://bucket/2026/annual.pdf&lt;/code&gt; both become &lt;code&gt;annual.pdf&lt;/code&gt;. If both are titled "Annual report", my key treats two different documents as one and drops the second. For File Search the filename is the store's last segment, so two stores ending in &lt;code&gt;/invoices&lt;/code&gt; collide the same way.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the maintainer changed in review
&lt;/h2&gt;

&lt;p&gt;The maintainer who reviewed it pushed a commit titled &lt;code&gt;deduplicate&lt;/code&gt; that moved the key from "what the source looks like" to "what the source is". The check moved into &lt;code&gt;extractSources()&lt;/code&gt; itself through an optional &lt;code&gt;emittedSourceKeys&lt;/code&gt; set, and each branch now passes its own identity key:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm9h46eb2x07imhdj97oy.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm9h46eb2x07imhdj97oy.webp" alt="Diagram: URL-like chunks are keyed by url and uri, RAG chunks with a path by document and uri, File Search chunks by file-search, store and title. All keys go into one emittedSourceKeys set." width="800" height="356"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Each source is keyed by its identity, and one set is shared across all stream chunks.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;packages/google/src/google-language-model.ts (merged, trimmed)&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight diff"&gt;&lt;code&gt;&lt;span class="gd"&gt;-    const emittedSourceUrls = new Set&amp;lt;string&amp;gt;();&lt;/span&gt;
&lt;span class="gi"&gt;+    const emittedSourceKeys = new Set&amp;lt;string&amp;gt;();&lt;/span&gt;
 ...
             const sources = extractSources({
               groundingMetadata: candidate.groundingMetadata,
               generateId,
&lt;span class="gi"&gt;+              emittedSourceKeys,&lt;/span&gt;
             });
             if (sources != null) {
               for (const source of sources) {
&lt;span class="gd"&gt;-                if (&lt;/span&gt;
&lt;span class="gd"&gt;-                  source.sourceType === 'url' &amp;amp;&amp;amp;&lt;/span&gt;
&lt;span class="gd"&gt;-                  !emittedSourceUrls.has(source.url)&lt;/span&gt;
&lt;span class="gd"&gt;-                ) {&lt;/span&gt;
&lt;span class="gd"&gt;-                  emittedSourceUrls.add(source.url);&lt;/span&gt;
&lt;span class="gd"&gt;-                  controller.enqueue(source);&lt;/span&gt;
&lt;span class="gd"&gt;-                }&lt;/span&gt;
&lt;span class="gi"&gt;+                controller.enqueue(source);&lt;/span&gt;
               }
             }
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;extractSources() after the merge (excerpt)&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;addSource&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;LanguageModelV4Source&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;emittedSourceKeys&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="n"&gt;emittedSourceKeys&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// web, image, maps, http(s) retrievedContext&lt;/span&gt;
&lt;span class="nf"&gt;addSource&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'source'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sourceType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'url'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;generateId&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'url'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// retrievedContext with gs:// or another path&lt;/span&gt;
&lt;span class="nf"&gt;addSource&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'source'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sourceType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'document'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;generateId&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mediaType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;filename&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'document'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// File Search: fileSearchStore, no uri&lt;/span&gt;
&lt;span class="nf"&gt;addSource&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'source'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sourceType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'document'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;generateId&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;mediaType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'application/octet-stream'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fileSearchStore&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'/'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'file-search'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fileSearchStore&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few details in that design are worth copying into your own code:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The key is the identity, not the display fields.&lt;/strong&gt; A RAG source is keyed by its full URI, so two &lt;code&gt;annual.pdf&lt;/code&gt; files in different folders stay separate, and the same URI with a different title is still one source.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The type is part of the key.&lt;/strong&gt; &lt;code&gt;['url', x]&lt;/code&gt; and &lt;code&gt;['document', x]&lt;/code&gt; can never collide, even if the same string shows up in both.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;JSON.stringify&lt;/code&gt; of an array instead of string concatenation.&lt;/strong&gt; Joining with a separator breaks when a value contains that separator. Serialising a tuple can't be ambiguous that way.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The set is optional.&lt;/strong&gt; &lt;code&gt;doGenerate&lt;/code&gt; calls &lt;code&gt;extractSources()&lt;/code&gt; without it and behaves exactly as before, so the non-streaming path didn't change at all.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The maintainer also added &lt;code&gt;it.each&lt;/code&gt; cases for those collisions: different RAG paths with the same title and filename (expect 2), the same RAG URI with different titles (expect 1), and different File Search stores with the same basename. Then approved and merged. The changeset text in the release reads "Deduplicate by document URI or File Search store and title so distinct documents with matching titles and filenames are preserved."&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; The lesson I took from the review: when you dedupe, ask what makes two records the same thing, not what makes them look the same. Title and filename are what a user sees. The URI is what the record is.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  From issue to release
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcov1qxmdq4gob3qy1sdh.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcov1qxmdq4gob3qy1sdh.webp" alt="Timeline: issue 24723 and PR 24724 opened on 7 November 2026, review and merge on 10 November, released in @ai-sdk/google 4.0.118 on 11 November." width="800" height="222"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Four days from bug report to a published package.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The issue went in first, with the exact cause, the offending lines and a minimal grounding payload, so the maintainers could confirm the bug without running anything. The PR linked to it with &lt;code&gt;Fixes #24723&lt;/code&gt;, listed what changed, what the tests cover, and the verification limit. The whole change was 289 lines added and 54 removed across the source, the test file and a changeset, and most of the added lines are tests.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do in your app
&lt;/h2&gt;

&lt;p&gt;Check which version you have:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm ls @ai-sdk/google
&lt;span class="c"&gt;# upgrade to the release with the fix (or anything newer)&lt;/span&gt;
npm i @ai-sdk/google@^4.0.118
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After upgrading, streamed File Search and RAG citations show up wherever you already read sources. On the server they are in &lt;code&gt;await result.sources&lt;/code&gt;. On the client, with &lt;code&gt;sendSources: true&lt;/code&gt;, they arrive as &lt;code&gt;source-document&lt;/code&gt; parts on the message, next to the &lt;code&gt;source-url&lt;/code&gt; parts from web grounding:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;components/Sources.tsx&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;UIMessage&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'ai'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nc"&gt;Sources&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;UIMessage&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;docs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s1"&gt;'source-document'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;ul&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;li&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sourceId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
          &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
          &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="sb"&gt;` (${d.filename})`&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="n"&gt;li&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="n"&gt;ul&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you can't upgrade yet, the raw grounding is still there. &lt;code&gt;streamText&lt;/code&gt; exposes the final step's provider metadata once the stream finishes, so you can read the chunks yourself:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;workaround for versions before 4.0.118&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;streamText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// after the stream completes&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;providerMetadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;finalStep&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;google&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;groundingMetadata&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;groundingChunks&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;docs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;any&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;retrievedContext&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Boolean&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;any&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fileSearchStore&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That only gives you the metadata at the end, not as source parts during the stream, which is the main reason to upgrade instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you are thinking about your first fix
&lt;/h2&gt;

&lt;p&gt;This was my first merged contribution to a project of this size, so a few notes for anyone about to try:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Lead with the cause, not the symptom.&lt;/strong&gt; "Sources are empty" sends a maintainer debugging. "&lt;code&gt;doStream&lt;/code&gt; only enqueues &lt;code&gt;sourceType === 'url'&lt;/code&gt;, here are the lines" lets them agree in a minute.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write the failing test first and say it fails on &lt;code&gt;main&lt;/code&gt;.&lt;/strong&gt; It turns your claim into something the CI checks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;State what you could not verify.&lt;/strong&gt; I couldn't hit the live API. Saying so cost nothing and nobody objected.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expect your fix to change.&lt;/strong&gt; The maintainer's dedupe key was better than mine. Merged with their edit is still a merged fix, and the edit taught me more than an instant approval would have.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you build RAG features on Gemini with the AI SDK, upgrade and check your citations panel with a streamed answer. It probably had nothing in it before.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/vercel-ai-sdk-gemini-file-search-sources-streaming" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>opensource</category>
      <category>ai</category>
      <category>typescript</category>
      <category>gemini</category>
    </item>
    <item>
      <title>Grounding AI-written marketing posts in real weather data, without letting the model make things up</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Wed, 11 Nov 2026 05:03:14 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/grounding-ai-written-marketing-posts-in-real-weather-data-without-letting-the-model-make-things-up-3060</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/grounding-ai-written-marketing-posts-in-real-weather-data-without-letting-the-model-make-things-up-3060</guid>
      <description>&lt;p&gt;Ask a language model for a social post for a roofing company and you get something fluent and generic: "Is your roof ready for winter? Our expert team offers a 50-year warranty and can finish most jobs in 4 days!" The client never offered a 50-year warranty. Nobody said 4 days. The post sounds great and is a liability.&lt;/p&gt;

&lt;p&gt;For a client building marketing tools for home service contractors, I built a pipeline that writes posts from things that are actually happening: a freeze warning tomorrow night in three of the contractor's towns, a high wind advisory, poor air quality. The model writes the words. It does not get to invent the facts. Here is how it is put together.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwav5eusfmlcow0q0aqj4.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwav5eusfmlcow0q0aqj4.webp" alt="The client app home screen with a weekly pipeline of signals and post ideas" width="800" height="431"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The app home, where signals turn into post ideas waiting for review.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The rule that does most of the work
&lt;/h2&gt;

&lt;p&gt;The planning prompt gives the model a list of dated, located facts and the client's real services, and then states one rule:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Only state facts given above: no invented numbers, prices, offers, warranties,
durations, years or phone numbers.
Today is {date}. Write advice before a coming event, never as if it already happened.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxnhdu6skglopbw70nibg.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxnhdu6skglopbw70nibg.webp" alt="Venn diagram: only the overlap of facts in the prompt and what the model wants to say is publishable" width="800" height="588"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Only the overlap is publishable.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Both lines came from real outputs. Without the first, the model added warranties and timelines that sounded plausible. Without the second, a post written on Tuesday about Thursday's storm said "after this week's storm", which is wrong on the day it would be published.&lt;/p&gt;

&lt;p&gt;The rule alone is not a guarantee, which is why nothing is ever posted automatically. A person approves every post. The point of the pipeline is to make that review fast because the facts are already correct.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pipeline
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftjgeyk2urgs7xvzxdhw8.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftjgeyk2urgs7xvzxdhw8.webp" alt="Pipeline from the hourly cron to claim, fetch, drop expired, save, plan, render and human approval" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;From a forecast to an approved post.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Runs that cannot double up
&lt;/h3&gt;

&lt;p&gt;A cron job calls the signals endpoint every hour, and the app also runs a startup bootstrap with retries at 1, 2, 4, 8 and 16 seconds after each deploy. That means the same work is requested more than once, on purpose, so each run has to claim its slot:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="no"&gt;IGNORE&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;signal_runs&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;schedule_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;started_at&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'nws-alerts@area-42:2026-10-25'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;CURRENT_TIMESTAMP&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffvyydw57s0as6j85bnuv.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffvyydw57s0as6j85bnuv.webp" alt="Overlapping runs claim their slot with INSERT OR IGNORE; only the run whose insert lands proceeds" width="800" height="783"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Only one run gets the slot.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The key is &lt;code&gt;provider@area:day&lt;/code&gt;. If the insert did nothing, someone else owns the run. A failed run, or one stuck in &lt;code&gt;running&lt;/code&gt; for over 20 minutes, can be claimed again. A successful one cannot. Weather alerts are fetched hourly, while the daily set (forecast, seasonal items, local news) runs once after 6 AM in the client's own timezone.&lt;/p&gt;

&lt;h3&gt;
  
  
  Weather data that expires
&lt;/h3&gt;

&lt;p&gt;The National Weather Service API is free and good, with two things to know. It wants a &lt;code&gt;User-Agent&lt;/code&gt; that includes contact details. And you resolve a location in two steps: &lt;code&gt;/points/{lat},{lng}&lt;/code&gt; gives you the forecast and grid URLs for that point, which you then fetch. Those point lookups rarely change, so they are cached for 31 days.&lt;/p&gt;

&lt;p&gt;Every forecast period has a validity window, written as an ISO 8601 interval like &lt;code&gt;2026-10-25T18:00:00-07:00/PT3H&lt;/code&gt;. A signal is only useful inside that window. The code parses it and drops anything already over before it is saved:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;windowOf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;validTime&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;duration&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;validTime&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'/'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getTime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;isoDurationMs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;duration&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;live&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;signals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;windowOf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;validTime&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjyefi6tt6aml1itylqd.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjyefi6tt6aml1itylqd.webp" alt="Timeline with a now marker: an expired warning is dropped, a current one is kept" width="800" height="743"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A forecast period has a window, and the window can close.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Without that check, an expired warning can reach the planner next to a current one, and the model will happily write about both. An expired signal is now never written to the database at all, and old signals for the same source are retired in the same transaction that saves the new ones.&lt;/p&gt;

&lt;p&gt;Requests to NWS get two attempts with a 15 second timeout, and retry on 429 or 5xx while respecting &lt;code&gt;Retry-After&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Small details that changed the output
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Keyword order matters.&lt;/strong&gt; A regex that checked "wind" before "wind chill" filed cold-weather alerts as wind events. Match the longer, more specific phrase first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dedupe the forecast cards.&lt;/strong&gt; Several periods can carry the same headline. Each signal gets a fingerprint of the provider plus a SHA-256 of its external ID, and duplicates are dropped.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Plan per industry.&lt;/strong&gt; A regex once labelled a safety consultancy as a remodeler. Each client now has an explicit industry plan rather than a guess from their website.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Local news, carefully
&lt;/h3&gt;

&lt;p&gt;Local news is a second source. The scraper reads &lt;code&gt;robots.txt&lt;/code&gt;, only fetches from an allowlist, caps pages at 2 MB with a 15 second navigation timeout, and asks a model whether each article is relevant to the client's trade, caching that answer so the same article is never judged twice. In one test with 24 local sites, 6 blocked plain fetches.&lt;/p&gt;

&lt;h3&gt;
  
  
  From plan to image
&lt;/h3&gt;

&lt;p&gt;The planner sees signals in a compact line format, roughly &lt;code&gt;[Category: kind, severity] (when) [towns] title: body (why)&lt;/code&gt;, writes the copy, and a layout step turns it into HTML. Playwright renders that to an image, either on a client photo or an AI background. Costs are tracked per post, stored as integer micro-dollars so sums never drift: about $0.09 to $0.12 on a client photo, and about $0.16 with a generated background.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug that froze every run
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmmq94ink9bedcwgb5zzf.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmmq94ink9bedcwgb5zzf.webp" alt="A cloned response is a tee; waiting on cancel hung the run, cancelling in the background fixed it" width="800" height="763"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Why every run froze.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;At one point every signal run froze. Nothing had crashed. Runs started and never finished.&lt;/p&gt;

&lt;p&gt;The cause was a helper that read a response body with a size limit. When the body got too large it cancelled the reader and threw. But the response had been cloned earlier (one copy for logging, one for parsing), and a cloned body is a tee. Cancelling one half of a tee does not settle until the other half is cancelled or read too. So &lt;code&gt;await reader.cancel()&lt;/code&gt; waited forever, and so did the run.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// before: hangs when the response was cloned&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Response too large'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;// after: cancel in the background and fail now&lt;/span&gt;
&lt;span class="k"&gt;void&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;catch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Response too large'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The regression test races an endless cloned stream against a 5 second timer and fails if the helper is still waiting. It is a small test that protects a very quiet failure: a hung run looks exactly like a slow day with no news.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would tell someone building this
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Give the model facts, not topics&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;"Freeze warning, Thursday 9 PM to Friday 9 AM, in these three towns" produces a better post than "write about winter prep", and it is checkable.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Forbid what it tends to invent&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Name the categories outright: numbers, prices, offers, warranties, durations, years, phone numbers.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Tell it what day it is&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And whether the event is coming or past. Models write in a vague present tense otherwise.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Make runs claimable&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A unique key per provider, area and day turns retries and overlapping crons into no-ops.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Expire data at the source&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Drop anything outside its validity window before it is saved, not when it is shown.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Keep a human in the loop&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Grounding makes review fast. It does not make review optional.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  How do you stop an LLM from inventing facts in marketing posts?
&lt;/h3&gt;

&lt;p&gt;Give the model a list of dated, located facts and tell it to state only those facts. My planning prompt names the categories the model tends to invent: numbers, prices, offers, warranties, durations, years and phone numbers. A person still approves every post, because the rule reduces made-up claims but does not guarantee none slip through.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does an LLM need to be told today's date when writing posts?
&lt;/h3&gt;

&lt;p&gt;Without the date, a model writes in a vague present tense and gets timing wrong. In my pipeline, a post written on Tuesday about Thursday's storm said "after this week's storm", which is false on the day it would be published. The prompt now states today's date and says to write advice before a coming event, never as if it already happened.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you handle weather alerts that have already expired?
&lt;/h3&gt;

&lt;p&gt;Parse each forecast period's validity window and drop anything already over before it is saved. National Weather Service periods carry an ISO 8601 interval such as &lt;code&gt;2026-10-25T18:00:00-07:00/PT3H&lt;/code&gt;. Without the check, an expired warning can reach the planner next to a current one, and the model will write about both.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you stop an hourly cron job and a startup retry from running the same work twice?
&lt;/h3&gt;

&lt;p&gt;Make each run claim its slot with a unique key before it starts. I use &lt;code&gt;INSERT OR IGNORE&lt;/code&gt; on a key shaped &lt;code&gt;provider@area:day&lt;/code&gt;, and only the run whose insert lands proceeds. A failed run, or one stuck in &lt;code&gt;running&lt;/code&gt; for over 20 minutes, can be claimed again, but a successful one cannot.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why can reading a cloned HTTP response body hang forever?
&lt;/h3&gt;

&lt;p&gt;A cloned response body is a tee, and cancelling one half does not settle until the other half is cancelled or read too. In my pipeline, &lt;code&gt;await reader.cancel()&lt;/code&gt; on an oversized body waited forever and froze every signal run. The fix was to cancel in the background with &lt;code&gt;void reader.cancel().catch(() =&amp;gt; {})&lt;/code&gt; and throw immediately.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/grounding-llm-content-real-data" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>typescript</category>
      <category>sqlite</category>
    </item>
    <item>
      <title>How to evaluate text-to-speech voices before you ship them</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Tue, 10 Nov 2026 06:11:39 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/how-to-evaluate-text-to-speech-voices-before-you-ship-them-1c5m</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/how-to-evaluate-text-to-speech-voices-before-you-ship-them-1c5m</guid>
      <description>&lt;p&gt;Picking a voice for a product usually goes like this: open the provider's demo page, type "Hello, how are you today?", listen to six voices, pick the one that sounds nicest. Then the voice goes into a support agent that has to read order numbers, or a narrator that has to get through Urdu place names, and it falls apart. Or it sounds right but takes two seconds before anything plays, which in a conversation feels like forever.&lt;/p&gt;

&lt;p&gt;I built Signal Tank to make that choice with evidence. You write or pick a real script, choose a voice and settings, generate a take, and every take is saved next to its numbers so you can compare them side by side. The code is small and public, and the patterns in it apply to any TTS provider.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/mahmadmanzoor/signal-tank" rel="noopener noreferrer"&gt;Signal Tank on GitHub&lt;/a&gt;: A latency-aware text-to-speech lab: server proxy, preset voices, safe cloning and Web Audio&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fit6ibnx359boylliky6h.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fit6ibnx359boylliky6h.webp" alt="Signal Tank lab page: script, voice, delivery and review steps" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The lab: script, voice, delivery, review. Every take stays in the session for comparison.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Test with scripts that look like production
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fywstq3l7usl8e2jrwp3q.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fywstq3l7usl8e2jrwp3q.webp" alt="A saved take holds the script, voice, settings and two timings" width="800" height="777"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Every take is a row.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The lab ships five sample scripts: narration, a support agent reading structured details, Urdu, Spanish and Arabic. They exist because the failures I cared about only show up with real content: numbers and codes read wrong, a switch between languages, a long sentence where the voice loses energy halfway. A greeting tests none of that.&lt;/p&gt;

&lt;p&gt;Every take stores the script, the voice, the settings (format, latency mode, temperature, top-p, speed, volume) and two timings. Comparing voices then becomes comparing rows, not memories.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two numbers, not one
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnsx95ordw956n25svby4.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnsx95ordw956n25svby4.webp" alt="Browser, proxy and provider, with a timeline from click to first byte to complete" width="800" height="471"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Two numbers, not one.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;"Latency" for TTS hides two different numbers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Time to first byte&lt;/strong&gt; is how long until audio starts arriving. With streaming playback, this is how fast the product feels.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time to complete&lt;/strong&gt; is how long until the whole file has arrived. If your player needs the complete file, this is what the user actually waits for.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A voice can win one and lose the other, and the latency mode you choose shifts the balance. The browser can measure both by reading the response as a stream instead of calling &lt;code&gt;response.blob()&lt;/code&gt;:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;public/app.js&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;started&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'/api/tts'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'POST'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getReader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;firstByte&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;done&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;break&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;firstByte&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;firstByte&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;started&lt;/span&gt;
  &lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;started&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;blob&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Blob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'content-type'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnzibjijil9u7bjn0cc6k.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnzibjijil9u7bjn0cc6k.webp" alt="Today playback waits for the whole file; next it will play while chunks arrive" width="800" height="461"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Measured now, playable later.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;performance.now()&lt;/code&gt; is monotonic and sub-millisecond, unlike &lt;code&gt;Date.now()&lt;/code&gt;. Both numbers include the whole round trip through your own server, which is what a user experiences, so measure from the browser rather than trusting the provider's dashboard.&lt;/p&gt;

&lt;p&gt;One limit: the lab currently waits for the complete file before playing it. That is fine for comparing voices, but it means the "first byte" number is a measurement, not something you can hear yet. Incremental playback through Media Source Extensions or an &lt;code&gt;AudioWorklet&lt;/code&gt; is the next step.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep the key on the server
&lt;/h2&gt;

&lt;p&gt;The browser never talks to the TTS provider. It posts a form to &lt;code&gt;/api/tts&lt;/code&gt;, a Vercel function that holds the key, validates everything, and streams the audio back:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;api/tts.js&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;counted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pipeThrough&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;TransformStream&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nf"&gt;transform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;controller&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;bytes&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;byteLength&lt;/span&gt;
    &lt;span class="n"&gt;controller&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enqueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nf"&gt;flush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'signal.tts.complete'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;requestId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;durationMs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;started&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;counted&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'Cache-Control'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'no-store'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'Content-Type'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;contentType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'X-Request-ID'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;requestId&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;TransformStream&lt;/code&gt; passes chunks straight through, so streaming is not delayed, while it counts bytes and logs the total when the stream ends. The server logs when headers arrived and when the body finished, which gives you the provider's share of the latency separately from the network to the user. The request ID goes back in a header so a slow take in the UI can be matched to its log line. Logs carry character counts and settings only, never the script, and a test checks that.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do not let the client pick arbitrary voices
&lt;/h3&gt;

&lt;p&gt;An early version sent the provider's voice reference ID from the browser. That meant anyone could call my endpoint, on my credits, with any voice in the provider's library, including voices cloned from other people. Now the server owns a frozen allowlist and the client sends only a preset name:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;lib/fish.js&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="no"&gt;VOICE_PRESETS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;freeze&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;balanced&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Balanced'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s1"&gt;'female-american'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Female · American'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;referenceId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'…'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s1"&gt;'male-british'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Male · British'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;referenceId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'…'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="c1"&gt;// …&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;form&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'voicePreset'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="no"&gt;VOICE_PRESETS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;voicePreset&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nf"&gt;inputError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Choose one available voice profile.'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;referenceId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nf"&gt;inputError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Manual voice references are not supported.'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flx8sjl3deocx1j8il486.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flx8sjl3deocx1j8il486.webp" alt="The browser used to send a provider voice ID; now it sends a preset name checked against an allowlist" width="800" height="640"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The client picks a name, never an ID.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;getAll(...).length !== 1&lt;/code&gt; is a small check worth stealing. Multipart forms allow the same field more than once, and different layers may read the first or the last value. Requiring exactly one closes that gap. Every numeric setting is range-checked too: temperature and top-p 0 to 1, speed 0.5 to 2, volume -20 to 20.&lt;/p&gt;

&lt;h3&gt;
  
  
  Map provider errors to messages you control
&lt;/h3&gt;

&lt;p&gt;Upstream errors are translated into a short, fixed set: a rejected key, no credits (402), rejected settings (422), busy (429), and everything else as a generic 502. The user gets a useful message, and nothing from the provider's internals leaks into the UI. A missing key returns a clear 503 telling you which environment variable to set.&lt;/p&gt;

&lt;h2&gt;
  
  
  Recording reference audio for a cloned voice
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa4kzew4na45vat1hifqk.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa4kzew4na45vat1hifqk.webp" alt="Recording length bar with a 10 to 30 second range and a mic level bar with a silence threshold" width="800" height="617"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What the recorder watches for.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Cloning needs a clean sample of the speaker, and most bad clones come from bad samples. The recorder does three things to help:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Pick a format the browser can actually record&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Safari and Chrome disagree, so the recorder tries &lt;code&gt;audio/webm;codecs=opus&lt;/code&gt;, then &lt;code&gt;audio/mp4&lt;/code&gt;, then &lt;code&gt;audio/ogg;codecs=opus&lt;/code&gt;, and uses the first that &lt;code&gt;MediaRecorder.isTypeSupported&lt;/code&gt; accepts.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Show a live level and warn about silence&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A Web Audio &lt;code&gt;AnalyserNode&lt;/code&gt; (fftSize 256) computes an RMS level every frame. If the smoothed level stays under 0.035 for two seconds, the UI says "No sound detected, check your microphone or speak closer."&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Keep the sample in a useful length&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;At least 10 seconds, stopped automatically at 30, with 15 to 20 recommended. Files are capped at 4 MB to stay under the Vercel request body limit.&lt;/p&gt;

&lt;p&gt;The level meter is a few lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="n"&gt;analyser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getByteTimeDomainData&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;energy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="n"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;centered&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;
  &lt;span class="n"&gt;energy&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;centered&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;centered&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;energy&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mf"&gt;0.012&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;smoothed&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;smoothed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.24&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cloning also requires an explicit consent checkbox and the exact transcript of what was read. The transcript improves the clone, and the consent step is the minimum for working with someone's voice.&lt;/p&gt;

&lt;h2&gt;
  
  
  A checklist before you pick a voice
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Use your real scripts, including the awkward ones: numbers, codes, names, a second language.&lt;/li&gt;
&lt;li&gt;Measure time to first byte and time to complete from the browser, and record which latency mode produced them.&lt;/li&gt;
&lt;li&gt;Run each voice more than once. A single take tells you very little about variance.&lt;/li&gt;
&lt;li&gt;Listen on the device your users have. Laptop speakers hide a lot.&lt;/li&gt;
&lt;li&gt;Keep keys and voice IDs on the server, and allowlist what the client may choose.&lt;/li&gt;
&lt;li&gt;Log sizes and timings, never the text people typed.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/mahmadmanzoor/signal-tank
&lt;span class="nb"&gt;cd&lt;/span&gt; signal-tank &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; npm install
cp .env.example .env   &lt;span class="c"&gt;# add your key&lt;/span&gt;
npm start              &lt;span class="c"&gt;# http://127.0.0.1:3000/lab&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Without a key the lab still runs and the generate button returns a clear message, so you can try the UI first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  How do you measure text-to-speech latency?
&lt;/h3&gt;

&lt;p&gt;Measure two numbers from the browser: time to first byte and time to complete. Read the response as a stream instead of calling &lt;code&gt;response.blob()&lt;/code&gt;, and time it with &lt;code&gt;performance.now()&lt;/code&gt;, which is monotonic and sub-millisecond. Measuring from the browser includes the round trip through your own server, which is what a user actually experiences.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the difference between time to first byte and time to complete in TTS?
&lt;/h3&gt;

&lt;p&gt;Time to first byte is how long until audio starts arriving, which sets how fast a streaming product feels. Time to complete is how long until the whole file has arrived, which is what the user waits for if your player needs the full file. A voice can win one and lose the other.&lt;/p&gt;

&lt;h3&gt;
  
  
  How should I choose a text-to-speech voice for a product?
&lt;/h3&gt;

&lt;p&gt;Test with scripts that look like production, not a greeting. In Signal Tank I use narration, a support agent reading structured details, Urdu, Spanish and Arabic, because numbers, codes and language switches are where voices fail. Save every take with its settings and timings, and run each voice more than once to see variance.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should the browser call the TTS provider directly?
&lt;/h3&gt;

&lt;p&gt;No. Keep the API key on the server and have the browser post to your own endpoint, which validates input and streams the audio back. Let the client send only a preset name, and map it to a voice ID from a frozen server-side allowlist, so nobody can use your credits with any voice in the provider's library.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I record good reference audio for a cloned voice in the browser?
&lt;/h3&gt;

&lt;p&gt;Pick a format the browser can record by trying &lt;code&gt;audio/webm;codecs=opus&lt;/code&gt;, then &lt;code&gt;audio/mp4&lt;/code&gt;, then &lt;code&gt;audio/ogg;codecs=opus&lt;/code&gt; with &lt;code&gt;MediaRecorder.isTypeSupported&lt;/code&gt;. Show a live level meter and warn if the smoothed level stays under 0.035 for two seconds. Require a consent checkbox and the exact transcript of what was read.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/measure-text-to-speech-latency" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>tts</category>
      <category>webaudio</category>
      <category>javascript</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Atmee V2 vs Seedance: How Causeur and a Real-Time Avatar Model Render in Seconds</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Sun, 08 Nov 2026 11:30:21 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/atmee-v2-vs-seedance-how-causeur-and-a-real-time-avatar-model-render-in-seconds-273j</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/atmee-v2-vs-seedance-how-causeur-and-a-real-time-avatar-model-render-in-seconds-273j</guid>
      <description>&lt;p&gt;&lt;strong&gt;Atmee V2&lt;/strong&gt; is a real-time video model for AI avatars, launched on 4 November 2026 by Atmanity, the Santa Clara lab behind &lt;a href="https://www.atmee.ai" rel="noopener noreferrer"&gt;atmee.ai&lt;/a&gt;. It turns one photo and about ten seconds of voice into a character you can video-call: it talks, listens, gestures and reacts while you speak. It can also render ordinary short clips. On the launch demo, the same 10-second clip took &lt;strong&gt;Seedance 2.5 233.6 seconds and $0.50&lt;/strong&gt;, and took &lt;strong&gt;Atmee V2 3.5 seconds and under one cent&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;I went through the launch video, Atmanity's research posts, the API docs, the public benchmarks and the developer tooling. This post covers what Atmee V2 is, how its two models (Causeur and the V2 renderer) fit together, how it compares with Seedance, what the API looks like underneath, and which claims are still unverified. Every clip below is hosted on this site, cut from Atmanity's own launch material, so it keeps playing if a social post disappears.&lt;/p&gt;

&lt;h2&gt;
  
  
  Atmee V2 at a glance
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;What it is:&lt;/strong&gt; a real-time, full-frame video foundation model for live AI avatars, plus a behavior model called Causeur.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Input:&lt;/strong&gt; one image and 10 to 15 seconds of voice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Output:&lt;/strong&gt; a live video call, a voice-only call, or an mp4 clip of 1 to 15 seconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Speed claim:&lt;/strong&gt; up to 50x faster and 10 to 100x cheaper than Seedance 2.5 and Wan 3.0, running live on one GPU.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Demo numbers:&lt;/strong&gt; 3.5 s vs 233.6 s for a 10 s clip; 6.4 s and $0.01 vs 331 s and $1.19 for an 8 s clip.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Price:&lt;/strong&gt; 2 credits a minute for a full V2 conversation, about $0.21 to $0.30.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Access:&lt;/strong&gt; invite code or the Growth plan ($49 a month) and up.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What is Atmee V2?
&lt;/h2&gt;

&lt;p&gt;Atmee V2 is the second avatar engine behind atmee.ai. Atmanity calls it "a real-time, full-frame video foundation model for live avatars". Full frame is the important word: V1 animated a talking head from the shoulders up, while V2 generates the whole shot, face, body, hands and surroundings, in square, 16:9, 9:16, 4:3 and 3:4.&lt;/p&gt;

&lt;p&gt;The input is one image plus a short voice sample (the founder says 10 seconds, atmanity.io says 15, and the API accepts 10 to 60 seconds for a voice clone). The output is one of three things: a live video call, a voice-only call, or a generated mp4 clip.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnqujjgzwxf4eyudlxkt2.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnqujjgzwxf4eyudlxkt2.gif" alt="Atmee V2 live video call demo from the launch video" width="400" height="225"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://mahmadmanzoor.com/shots/post-atmee-v2/live-call.mp4" rel="noopener noreferrer"&gt;Watch the clip with sound&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A live Atmee V2 call on a tablet, from Atmanity's 4 November 2026 launch video. The avatar answers with expression and hand movement, not only lip sync.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How does Atmee V2 work? Causeur decides, Atmee V2 renders
&lt;/h2&gt;

&lt;p&gt;Atmee V2 is two models, not one. In the launch video the founder puts it simply: "we built two models, a body and a brain". Atmanity's site sums it up as "Causeur decides, Atmee v2 renders."&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgp1s0v6e3b8afb7452m1.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgp1s0v6e3b8afb7452m1.webp" alt="Atmee V2 architecture diagram: user audio, user video, app context and persona go into Causeur, the behavior model, which decides words and voice, emotion and gesture, and timing; the Atmee V2 renderer then draws the full frame live on one GPU and streams it into the LiveKit room" width="799" height="444"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The two-model stack. Causeur is the brain that decides what happens; Atmee V2 is the body that draws it.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  What is Causeur?
&lt;/h3&gt;

&lt;p&gt;Causeur is Atmanity's behavior model, "the behavior model that brings an agent alive". It handles personality, emotion, timing and active listening, moment to moment. In the launch video it is "a model that brings context and behavior into the voice, the body, and the response". Concretely, Causeur decides:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;When to speak.&lt;/strong&gt; Whether the user has finished, paused to think, or is interrupting, and whether to answer, wait, or give a short reaction instead of a full reply.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;How to feel.&lt;/strong&gt; Which emotion and expression fit the moment, so the renderer smiles at the joke and not three seconds before it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What to do while listening.&lt;/strong&gt; Nods, small smiles and "go on" signals while the user talks, which is what makes a call feel like someone is there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When to act on the app.&lt;/strong&gt; In a session with app tools, Causeur is the part that calls them (a v1 avatar accepts tools but never calls them).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Atmanity benchmarks Causeur against full-duplex speech models on three turn-taking cases, higher is better:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;User ends a turn:&lt;/strong&gt; Causeur 0.92, Moshi (Kyutai) 0.88, LiveKit 0.74, Deepgram Flux 0.68, Pipecat 0.56.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;User pauses mid-thought:&lt;/strong&gt; Causeur 0.92, Moshi 0.87, Pipecat 0.66, LiveKit 0.59, Deepgram Flux 0.57.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;User interrupts:&lt;/strong&gt; Causeur 0.87, Moshi 0.67, LiveKit, Deepgram Flux and Pipecat 0.02 each.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The interruption row is the one I would watch. Most voice agents today either talk over you or stop dead at any noise, and the standard turn detectors in that chart barely register an interruption at all.&lt;/p&gt;

&lt;h3&gt;
  
  
  What does the Atmee V2 renderer do?
&lt;/h3&gt;

&lt;p&gt;The renderer takes Causeur's decisions plus the voice audio and draws every frame: lip sync, micro-expression, head and body motion, and the scene around the character. Atmanity says it runs live on a single GPU, which is what makes the per-minute price possible. Because it is a general video foundation model that happens to be trained hard on people, the same renderer also produces offline clips, which is why it can be compared with Seedance at all.&lt;/p&gt;

&lt;p&gt;Atmanity's avatar benchmark puts it against other real-time avatar models (metrics are Atmanity's own and not defined in detail):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;LipSync:&lt;/strong&gt; Atmee V2 7.61, Runway 7.16, LemonSlice 6.74, LiveTalk 3.14, Kling AI 2.98.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Naturalness:&lt;/strong&gt; Atmee V2 0.741, Runway 0.629, LemonSlice 0.575, Kling AI 0.526, LiveTalk 0.503.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Identity preservation:&lt;/strong&gt; Atmee V2 0.881, LemonSlice 0.821, Runway 0.811, Kling AI 0.642, LiveTalk 0.543.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Liveliness:&lt;/strong&gt; Atmee V2 1.31, Runway 1.21, Kling AI 1.05, LemonSlice 0.96, LiveTalk 0.77.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  What research is behind Causeur?
&lt;/h3&gt;

&lt;p&gt;Atmanity has not published a Causeur paper yet; its site lists "a multimodal conversational behavior model" as forthcoming. Two papers the same team published earlier tackle exactly the problems Causeur claims to solve. To be clear, this link is my reading, not something Atmanity states:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;MM-When2Speak&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2505.14654" rel="noopener noreferrer"&gt;arXiv 2505.14654&lt;/a&gt;, May 2025) watches a conversation's video, audio and transcript, encodes each separately, fuses them with self-attention and predicts one of three actions: stay silent, give a full response, or give a short reaction. F1 scores were 0.89, 0.84 and 0.72, against 0.71, 0.63 and 0.41 for a text-only LLM. That is the timing half of Causeur.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GLARE&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2609.40317" rel="noopener noreferrer"&gt;arXiv 2609.40317&lt;/a&gt;, NeurIPS 2026) turns speaker audio and one listener photo into a listener video that nods, smiles, laughs, frowns or looks surprised at the right moment. It is trained on about 147 hours of two-person conversations with 64,000+ labelled reactions, uses prosody (pitch, loudness, emphasis) to find the moments that deserve a reaction, and penalises mistimed reactions with a per-frame loss and a reward. On the RealTalk benchmark it beats DyStream, ListenFormer, ViCo, DIM and L2L on reaction F1 (0.59) and timing error (57.4 vs 63.2 for DyStream). That is the active-listening half.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There is also a data loop: Atmanity says atmee.ai, the consumer app, feeds back thousands of hours of real conversations every month for training.&lt;/p&gt;

&lt;h2&gt;
  
  
  Who makes Atmee V2?
&lt;/h2&gt;

&lt;p&gt;Atmanity Inc., founded in October 2024 and based in Santa Clara. The CEO, Chen-Ping Yu, previously founded Phiar, an AR navigation startup that Google acquired in 2022. The team lists six PhDs, and the company names Forerunner, Essence, Recursive Ventures, Hyphen Capital, GFR Fund and Hawktail as backers. Atmee is the product name; Atmanity still shows up in the infrastructure, for example the API host &lt;code&gt;api.atmanity.us&lt;/code&gt; and the &lt;code&gt;Atmanity&lt;/code&gt; GitHub organisation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Atmee V2 vs Seedance: which is faster and cheaper?
&lt;/h2&gt;

&lt;p&gt;For short character clips, Atmee V2 is much faster and cheaper than Seedance in Atmanity's own side-by-side. The demo uses one prompt, "Selfie video of a blonde girl, stroking her hair", and renders a 10-second clip on both models:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Seedance 2.5:&lt;/strong&gt; 233.6 seconds, $0.50 per clip.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Atmee V2:&lt;/strong&gt; 3.5 seconds, under $0.01 per clip.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2gyg2no2irsqu8mllxbv.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2gyg2no2irsqu8mllxbv.gif" alt="Seedance 2.5 vs Atmee V2 side-by-side: same prompt, 233.6 seconds vs 3.5 seconds" width="560" height="315"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://mahmadmanzoor.com/shots/post-atmee-v2/seedance-vs-atmee.mp4" rel="noopener noreferrer"&gt;Watch the clip with sound&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Atmanity's side-by-side from the launch video: same prompt, Seedance on the left (233.6 s, $0.50), Atmee V2 on the right (3.5 s, under $0.01). Play it with sound for the narration.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;That clip is a 67x speed gap (233.6 / 3.5). A week before launch, the founder posted a second side-by-side, an 8-second clip:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Seedance 2.5:&lt;/strong&gt; 331.0 seconds, $1.19.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Atmee V2:&lt;/strong&gt; 6.4 seconds, $0.01.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv5gbiogv5mdt9zrltta7.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv5gbiogv5mdt9zrltta7.gif" alt="Atmee V2 vs Seedance 2.5 on an 8-second clip: 6.4 seconds and $0.01 vs 331 seconds and $1.19" width="440" height="433"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://mahmadmanzoor.com/shots/post-atmee-v2/seedance-vs-atmee-8s.mp4" rel="noopener noreferrer"&gt;Watch the clip with sound&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The pre-launch comparison: Atmee V2 (6.4 s, $0.01) next to Seedance 2.5 (331.0 s, $1.19). Atmanity framed it as 51x faster and 119x cheaper.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Atmanity framed that one as "119x cheaper and 51x faster than Seedance 2.5". The headline claim at launch is more careful: "up to 50x faster and 10 to 100x cheaper" than general video models such as Seedance 2.5 and Wan 3.0. In both samples the output quality is close; Atmee V2 holds face and hair detail at least as well.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Warning:&lt;/strong&gt; Commenters on the 8-second post asked a fair question: both panels perform almost the same motion, which two independent models rarely do from one prompt. Atmanity did not publish the exact workflow (same prompt and seed, or one clip used as reference for the other). Until someone reproduces it, treat the comparison as a vendor demo, not a benchmark.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwzcdyl9or7rdcp3fm3h7.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwzcdyl9or7rdcp3fm3h7.webp" alt="Diagram: Seedance submits a prompt to a job queue and returns an mp4 after 233.6 seconds; Atmee V2 returns the same 10-second clip in 3.5 seconds or streams frames into a live call" width="800" height="356"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The structural difference: Seedance is an async job that returns a file. Atmee V2 returns the file in seconds, or skips the file and streams into a call.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Why is Atmee V2 so much faster than Seedance?
&lt;/h3&gt;

&lt;p&gt;Because it is built for a narrower job. Seedance is a general-purpose video generator: any scene, any camera move, many subjects, up to 30 seconds with generated sound in 2.5. You submit a job and poll for the file, which takes from about 30 seconds to a few minutes. Atmee V2 is specialised for one subject, a character facing the viewer, and it has to run at conversation speed on a single GPU. A model that only has to be good at people talking can spend far less compute per frame than one that has to draw anything.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is Seedance better at anything?
&lt;/h3&gt;

&lt;p&gt;Yes, and it matters if you are choosing between them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Scope.&lt;/strong&gt; Seedance renders whole scenes, multi-shot sequences and non-character content. Atmee V2 is tuned for characters.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Length.&lt;/strong&gt; Atmee clips run 1 to 15 seconds (at 25 fps, 480x480 up to 1280x720; 1080p is listed as coming soon). Seedance 2.5 goes to 30 seconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Access.&lt;/strong&gt; Seedance is a public API through several providers. Atmee V2 is invite-only for now: building V2 avatars and clips needs the Growth plan or an invite code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Independent proof.&lt;/strong&gt; The speed and cost comparison is Atmanity's own. No methodology was published, and I found no third-party benchmark yet.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; The "under $0.01" in the demo looks like a compute cost, not a list price. Atmee's published clip price at 480x480 is 0.3 credit for up to 10 seconds, and a credit costs about 10.5 to 15 US cents depending on the plan. That is roughly 3 to 4.5 cents for the demo's clip, still about ten times under the $0.50 Seedance figure.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Can Atmee V2 hold a live conversation?
&lt;/h2&gt;

&lt;p&gt;Yes, and this is the part Seedance cannot do at all. Seedance and Wan return a finished file; neither can listen to you and answer while you watch. Atmee V2 streams video into a LiveKit room while Causeur handles turn-taking. Here is Atmanity's 36-second V2 reel of live calls:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqt566c35rhkap2glazei.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqt566c35rhkap2glazei.gif" alt="Atmee V2 demo reel: live avatars listening, gesturing and answering" width="400" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://mahmadmanzoor.com/shots/post-atmee-v2/v2-demo.mp4" rel="noopener noreferrer"&gt;Watch the clip with sound&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Atmee V2 live sessions from atmee.ai: active listening, gestures and expression across different characters and roles.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Watch the listeners in that reel, not the speakers. The nods and small smiles while the user talks are Causeur at work; the drawing of them is the renderer.&lt;/p&gt;

&lt;h2&gt;
  
  
  How does the Atmee V2 API work?
&lt;/h2&gt;

&lt;p&gt;There are three ways to use it, all on &lt;code&gt;https://api.atmanity.us&lt;/code&gt; with an &lt;code&gt;X-Api-Key&lt;/code&gt; header from your backend (the API refuses browser calls):&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Full-stack session&lt;/strong&gt; (&lt;code&gt;POST /v1/sessions&lt;/code&gt;): Atmee runs speech recognition, Causeur, the LLM, the cloned voice and the video. Your page joins a LiveKit room and talks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Avatar-only render&lt;/strong&gt; (&lt;code&gt;POST /v1/avatar_sessions&lt;/code&gt;, or the &lt;code&gt;livekit-plugins-atmee&lt;/code&gt; Python package): you keep your own LiveKit voice agent and Atmee only draws the face.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Clips&lt;/strong&gt; (&lt;code&gt;POST /v1/generation/clips&lt;/code&gt;): image plus prompt, optional audio, mp4 out.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgjkxbedxmr0br63b1vmn.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgjkxbedxmr0br63b1vmn.webp" alt="Diagram of an Atmee V2 live session: the backend calls POST /v1/sessions with its API key, gets serverUrl and a user token, the browser joins the LiveKit room, and Atmee runs speech recognition, Causeur, the LLM and voice, and the V2 renderer that publishes video into the room" width="800" height="462"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;One full-stack session. Your backend never streams video; it only starts and ends the session.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Start a live session
&lt;/h3&gt;

&lt;p&gt;Your backend starts the session and hands the page a one-room token. The call answers once the avatar is in the room, usually 15 to 20 seconds; the user then has 60 seconds to join.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST https://api.atmanity.us/v1/sessions &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"X-Api-Key: &lt;/span&gt;&lt;span class="nv"&gt;$ATMEE_API_KEY&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{ "avatarId": "&amp;lt;avatar-id&amp;gt;", "maxDurationSeconds": 300 }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;201 Created&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"sessionId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"550e8400-e29b-41d4-a716-446655440000"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"serverUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"wss://atmanity.livekit.cloud"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"roomName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"rk-8f14e45f-ceea-467a-9f21-4c2f5b3a91cd"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"userToken"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"eyJhbGciOi..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"actualDurationSeconds"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"requestedDurationSeconds"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"durationCapped"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"billingMode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"metered"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The page joins with &lt;code&gt;livekit-client&lt;/code&gt;, attaches the avatar's tracks to a &lt;code&gt;&amp;lt;video&amp;gt;&lt;/code&gt; and an &lt;code&gt;&amp;lt;audio&amp;gt;&lt;/code&gt; element, and publishes the microphone:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;avatar-room.js&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;Room&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;RoomEvent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;Track&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;livekit-client&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;room&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Room&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="nx"&gt;room&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;RoomEvent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;TrackSubscribed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;track&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;el&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getElementById&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;track&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;kind&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nx"&gt;Track&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Kind&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Video&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;avatar&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;voice&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nx"&gt;track&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;attach&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;el&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;room&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;serverUrl&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;userToken&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// within 60 seconds&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;room&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startAudio&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;room&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;localParticipant&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setMicrophoneEnabled&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;End it with &lt;code&gt;POST /v1/sessions/{id}/end&lt;/code&gt;. Billing runs by the second, only while both you and the avatar are in the room.&lt;/p&gt;

&lt;h3&gt;
  
  
  Tell the avatar what is on screen
&lt;/h3&gt;

&lt;p&gt;The most useful part for app builders is the &lt;code&gt;atmee.app&lt;/code&gt; topic. The page sends small JSON messages and the avatar reacts in its own words: &lt;code&gt;state&lt;/code&gt; replaces "what is true now", &lt;code&gt;note&lt;/code&gt; adds a fact for the rest of the session, and &lt;code&gt;event&lt;/code&gt; asks the avatar to remark on something once nobody has the floor.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;app-messages.js&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;send&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
  &lt;span class="nx"&gt;room&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;localParticipant&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sendText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;topic&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;atmee.app&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;state&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;The user is on the pricing page. Starter is 9 dollars a month, Growth is 49.&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;note&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;The user's name is Sara and she is on the free plan.&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;event&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;The user just finished checkout. Congratulate her in one sentence.&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;ttl_s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing comes back on that topic, so the only proof a message landed is what the avatar says next. One practical detail from the docs: wait about 2.5 seconds after the avatar arrives before the first message, or it is dropped.&lt;/p&gt;

&lt;h3&gt;
  
  
  Let the avatar call your app
&lt;/h3&gt;

&lt;p&gt;App tools use the OpenAI function format, declared per session in &lt;code&gt;appTools&lt;/code&gt;. Each call reaches the page as a LiveKit RPC named &lt;code&gt;atmee.tool&lt;/code&gt;. &lt;code&gt;notify&lt;/code&gt; tools let the avatar keep talking; &lt;code&gt;await&lt;/code&gt; tools make it wait for your answer (up to 28 seconds).&lt;/p&gt;

&lt;p&gt;&lt;code&gt;appTools&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"instructions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Put each slide up before you teach it. When the trainee asks for their score, get it."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"tools"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"show_slide"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Put lesson slide n on the trainee's screen."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"parameters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"object"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"properties"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"n"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"integer"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"required"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"n"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"get_score"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Get the trainee's quiz score so far."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"parameters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"object"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"properties"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"x-atmee"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"await"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"timeoutSeconds"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;tools.js&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;room&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;registerRpcMethod&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;atmee.tool&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;payload&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;args&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;show_slide&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nf"&gt;showSlide&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;n&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;ok&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;get_score&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/api/score&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;then&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()));&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`no tool &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Register the handler before &lt;code&gt;room.connect&lt;/code&gt;, so no call arrives at a page without one. Rules, scores and hidden answers belong in your app; the tool hands the avatar the result. Otherwise the avatar answers before your app acts, and may invent an outcome it later has to take back.&lt;/p&gt;

&lt;h3&gt;
  
  
  Give your existing LiveKit agent a face
&lt;/h3&gt;

&lt;p&gt;If you already run a voice agent on LiveKit Agents, the plugin adds the avatar without moving your speech, LLM or tools:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;agent.py&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;livekit.plugins&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;atmee&lt;/span&gt;

&lt;span class="n"&gt;avatar&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;atmee&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;AvatarSession&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;avatar_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;...&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# reads ATMEE_API_KEY
&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;avatar&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;room&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;room&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;      &lt;span class="c1"&gt;# before session.start
&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;MyAgent&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;room&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;room&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Your agent's TTS audio drives the face. Atmee never sees your LiveKit secret: you mint the avatar's token yourself. Version 0.2.0 and later render V2 avatars; a Pipecat plugin exists too.&lt;/p&gt;

&lt;h3&gt;
  
  
  Generate a clip
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST https://api.atmanity.us/v1/generation/clips &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"X-Api-Key: &lt;/span&gt;&lt;span class="nv"&gt;$ATMEE_API_KEY&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Idempotency-Key: 6f1c2a9e-3b7d-4c11-9a52-0d8e4f7b2c61"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"image":"https://example.com/host.png","prompt":"she holds up the bottle and smiles, the camera slowly pushing in","audio":"https://example.com/line.mp3","durationSeconds":12}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The response carries a short-lived mp4 link (24 hours). Without &lt;code&gt;audio&lt;/code&gt; the clip is silent; with it, the character speaks the line. For longer videos, stitch 15-second clips with ffmpeg.&lt;/p&gt;

&lt;h2&gt;
  
  
  How much does Atmee V2 cost?
&lt;/h2&gt;

&lt;p&gt;Everything is paid in credits. A credit costs about 10.5 to 15 US cents depending on the plan (prices read on 8 November 2026):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Full-stack V2 session:&lt;/strong&gt; 2 credits a minute, about $0.21 to $0.30 a minute. V1 is 1 credit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Avatar-only V2 render:&lt;/strong&gt; 1 credit a minute. V1 is 0.5.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Voice-only session:&lt;/strong&gt; 1 credit a minute.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Clips:&lt;/strong&gt; 0.1 credit (480x480, up to 5 s) to 2 credits (1280x720, 15 s).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Plans:&lt;/strong&gt; Free (10 credits, 5-minute sessions, watermark), Starter $9, Growth $49 (needed to build V2 and clips), Pro $199, Scale $599, Enterprise for V2 HD.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the "10 to 100x cheaper" claim is against general video models. Against other live-avatar vendors the price is in the same range: HeyGen LiveAvatar, Anam and Tavus all sit around $0.09 to $0.37 a minute. Atmee competes there on behavior and full-body motion, not on price.&lt;/p&gt;

&lt;h2&gt;
  
  
  What are the limits right now?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Invite-only V2.&lt;/strong&gt; Request a code at atmee.ai/v2signup, or be on Growth or higher. My own account is on the Free plan and reports &lt;code&gt;v2Access: "invite_required"&lt;/code&gt;, which is why the clips here are Atmanity's and not mine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cold starts.&lt;/strong&gt; V2 runs on a GPU pool that scales down when quiet. The first session after a quiet period can take minutes; the docs suggest a client timeout of seven minutes or a retry on &lt;code&gt;503&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One conversation per avatar&lt;/strong&gt; at a time, one user in it. More users need more avatars.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No published latency number.&lt;/strong&gt; The docs give 15 to 20 seconds to start a session, but no per-turn response time to compare with vendors who quote 150 to 600 ms.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Claims, not audits.&lt;/strong&gt; The 50x, the 10 to 100x and the one-GPU figures come from Atmanity. Treat them as a strong first data point until someone outside the company measures them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Who should use Atmee V2?
&lt;/h2&gt;

&lt;p&gt;Pick Atmee V2 when the product is a conversation with a face: a tutor, a sales role-play with a tough buyer, a museum guide, a support agent, a game character, a recruiter screening call. The agenda feature (a graph of conversation steps per avatar) and app tools make those shapes straightforward. Pick Seedance or a similar model when you need a cinematic shot, a scene with no person in it, or a clip longer than 15 seconds. And if you only need a quick clip of a person moving or speaking, Atmee V2 is the faster and cheaper option to try first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is Atmee V2?
&lt;/h3&gt;

&lt;p&gt;Atmee V2 is a real-time video model by Atmanity for AI avatars. From one image and about ten seconds of voice, it creates a character that can hold a live video call with expressions and gestures, or render short mp4 clips. It launched on 4 November 2026.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is Atmee V2 better than Seedance?
&lt;/h3&gt;

&lt;p&gt;For character clips and live avatars, yes on speed and cost: Atmanity's side-by-side shows a 10-second clip in 3.5 seconds for under $0.01 on Atmee V2, against 233.6 seconds and $0.50 on Seedance 2.5. Seedance is still the broader tool for full scenes, longer clips and non-character video.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is Causeur in Atmee V2?
&lt;/h3&gt;

&lt;p&gt;Causeur is Atmanity's behavior model, the "brain" of Atmee V2. It decides personality, emotion, timing and active listening moment to moment: when to speak, when to wait, when to nod or smile, and when to call an app tool. The Atmee V2 renderer, the "body", then draws those decisions as live video.&lt;/p&gt;

&lt;h3&gt;
  
  
  How is Atmee V2 different from Atmee V1?
&lt;/h3&gt;

&lt;p&gt;V1 animates a talking head from the shoulders up at 512x512 and is cheaper (1 credit a minute). V2 renders the full frame, face, body, gestures and scene, in five aspect ratios, adds Causeur-driven emotion and timing, and costs 2 credits a minute. Anyone can talk to a V2 avatar; building one needs an invite or the Growth plan.&lt;/p&gt;

&lt;h3&gt;
  
  
  How fast is Atmee V2?
&lt;/h3&gt;

&lt;p&gt;Atmanity claims up to 50x faster than general video models such as Seedance 2.5 and Wan 3.0, and fast enough to render a live call on one GPU. A live session takes about 15 to 20 seconds to start.&lt;/p&gt;

&lt;h3&gt;
  
  
  How much does Atmee V2 cost?
&lt;/h3&gt;

&lt;p&gt;A full-stack V2 conversation costs 2 credits a minute, about $0.21 to $0.30. Avatar-only rendering is 1 credit a minute, and clips cost 0.1 to 2 credits each. Building V2 avatars needs the $49 Growth plan or an invite code.&lt;/p&gt;

&lt;h3&gt;
  
  
  Who makes Atmee?
&lt;/h3&gt;

&lt;p&gt;Atmanity Inc., an AI lab in Santa Clara founded in 2024 by Chen-Ping Yu, who previously founded the AR startup Phiar, acquired by Google in 2022.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I use Atmee V2 with LiveKit?
&lt;/h3&gt;

&lt;p&gt;Yes. Full-stack sessions run in a LiveKit room your page joins with a token, and the &lt;code&gt;livekit-plugins-atmee&lt;/code&gt; Python package gives an existing LiveKit Agents voice agent an Atmee face. A Pipecat plugin is also available.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/atmee-v2-vs-seedance-real-time-ai-avatar" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>webdev</category>
      <category>video</category>
    </item>
    <item>
      <title>No-login magic links for field crews: hashed, scoped, revocable, and usable offline</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Sat, 07 Nov 2026 05:01:46 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/no-login-magic-links-for-field-crews-hashed-scoped-revocable-and-usable-offline-2fba</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/no-login-magic-links-for-field-crews-hashed-scoped-revocable-and-usable-offline-2fba</guid>
      <description>&lt;p&gt;Yardline handles the handoff between a building supplier and a contractor: a truck arrives on site, someone has to confirm what came off it, note what is missing or damaged, and sign. That someone is usually a crew lead standing on a roof or in a driveway, in gloves, with one bar of signal, who has never heard of Yardline and has no interest in creating an account.&lt;/p&gt;

&lt;p&gt;So the crew gets a link instead. Tap it, see the delivery, mark it received. No password, no app install, no sign-up. The rest of this post is about making that link safe to hand out by text message, and sturdy enough to work when the phone is offline.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcjx8azic61wuvr01e74f.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcjx8azic61wuvr01e74f.webp" alt="Yardline on a phone" width="390" height="844"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Text sizes have a 14px floor. The people using this are often outside, in bright light, and not looking closely.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What a link like this has to guarantee
&lt;/h2&gt;

&lt;p&gt;A link is a password that travels in plain sight: in SMS threads, group chats, forwarded screenshots. Before writing any code I listed what it must and must not allow:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It opens &lt;strong&gt;one delivery&lt;/strong&gt; and nothing else. Not the contractor's other jobs, not the supplier's catalog.&lt;/li&gt;
&lt;li&gt;It &lt;strong&gt;expires&lt;/strong&gt; on its own, by default in a week.&lt;/li&gt;
&lt;li&gt;The office can &lt;strong&gt;revoke&lt;/strong&gt; it at any time, for example when the wrong number got it.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;database leak&lt;/strong&gt; must not hand out working links.&lt;/li&gt;
&lt;li&gt;Search engines must never index it.&lt;/li&gt;
&lt;li&gt;Every action through it is &lt;strong&gt;attributable&lt;/strong&gt; to "the crew on this link", not to the person who created it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Creating the link
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3kxugix3uh4wgaic5p1g.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3kxugix3uh4wgaic5p1g.webp" alt="A crew link from creation and hashing to offline queueing and event append, with revocation at any time" width="800" height="434"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A crew link, start to finish.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;packages/api/src/routers/crew.ts&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nx"&gt;createLink&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;permissioned&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;delivery.track&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;input&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;object&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;deliveryId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;string&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="na"&gt;expiresInDays&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;number&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="p"&gt;}))&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mutation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// ...check the caller is the contractor who owns this delivery...&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;guestUserId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;newId&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;users&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
      &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;guestUserId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;tenantId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tenantId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;crew_lead&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Jobsite crew&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;accountId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;delivery&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;accountId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;

    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;randomBytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;toString&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;base64url&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;expiresAt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;expiresInDays&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;86&lt;/span&gt;&lt;span class="nx"&gt;_400_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveryAccessTokens&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
      &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;newId&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="na"&gt;tenantId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tenantId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;deliveryId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;delivery&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;tokenHash&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;hashToken&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;token&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="na"&gt;createdBy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;session&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;guestUserId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;expiresAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;appUrl&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/receive/&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;token&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;expiresAt&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each choice maps to one of the guarantees:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;32 random bytes&lt;/strong&gt; from the OS RNG is 256 bits. Guessing is not a realistic attack, so the token itself can be the capability.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Only the SHA-256 is stored.&lt;/strong&gt; The plain token exists in one HTTP response and in the URL the office sends. Someone with a copy of the database sees hashes, which cannot be turned back into links. Because the token is long and random, a fast unsalted hash is fine here. Slow hashing like bcrypt is for low-entropy human passwords.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The link gets its own guest user&lt;/strong&gt; with the &lt;code&gt;crew_lead&lt;/code&gt; role, tied to the contractor's account. Events made through the link are recorded as "Jobsite crew", so the office user who created it is never blamed for what the crew did.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expiry is bounded&lt;/strong&gt; between 1 and 30 days.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;/receive/&lt;/code&gt; pages are &lt;code&gt;noindex&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Checking the link on every call
&lt;/h2&gt;

&lt;p&gt;There is no session cookie for the crew. Every request carries the token, and every request resolves it again:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;packages/api/src/routers/crew.ts&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;row&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;db&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="cm"&gt;/* token id, guest user, the delivery and its jobsite */&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveryAccessTokens&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;innerJoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveries&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveries&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;deliveryAccessTokens&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;deliveryId&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;and&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveryAccessTokens&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tenantId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tenantId&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveryAccessTokens&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tokenHash&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;hashToken&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;token&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
    &lt;span class="nf"&gt;gt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveryAccessTokens&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;expiresAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="nf"&gt;isNull&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveryAccessTokens&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;revokedAt&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="p"&gt;))&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;TRPCError&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;UNAUTHORIZED&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;message&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;This receiving link is invalid or expired.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveryAccessTokens&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;lastUsedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;}).&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveryAccessTokens&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5ey862ckk7fmqztkikpq.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5ey862ckk7fmqztkikpq.webp" alt="The token is hashed and matched in one query, giving either the delivery or one generic error" width="800" height="773"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;One query, one error message.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Tenant, hash, expiry and revocation are all in one &lt;code&gt;WHERE&lt;/code&gt;, so there is no path where one check is forgotten. Wrong token, expired token and revoked token all produce the same message, which tells an attacker nothing about which one it was. &lt;code&gt;lastUsedAt&lt;/code&gt; gives the office a quiet signal that the crew actually opened the link.&lt;/p&gt;

&lt;p&gt;Revoking sets &lt;code&gt;revokedAt&lt;/code&gt; on the delivery's active tokens. The next request, from any phone holding that link, fails.&lt;/p&gt;

&lt;h3&gt;
  
  
  One set of business rules, not two
&lt;/h3&gt;

&lt;p&gt;The tempting shortcut is a separate, simpler set of "crew endpoints" with their own logic. That is how two code paths end up disagreeing about whether a delivery can be received. Instead, the crew router resolves the link and then calls the normal receipts router with a synthetic session:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;withCrewSession&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;access&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;CrewAccess&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nx"&gt;Context&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;session&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;tenantId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tenantId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;access&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;guestUserId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;crew_lead&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Jobsite crew&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;contractorId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;access&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;delivery&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;accountId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;supplierId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;branchId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;deviceId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`crew-link:&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;access&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;// inside a crew mutation&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;access&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;resolveAccess&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;token&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;receiptsRouter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createCaller&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;withCrewSession&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;access&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;deliveryId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;access&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;delivery&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;receiptId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;receiptId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;occurredAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;occurredAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note that the delivery ID comes from the resolved link, never from the request. A crew cannot point their token at a different delivery by changing a parameter.&lt;/p&gt;

&lt;h2&gt;
  
  
  Working with no signal
&lt;/h2&gt;

&lt;p&gt;On load, the receiving page fetches the delivery and caches it in IndexedDB. If the fetch fails, it shows the cached copy and says "Working offline". When the crew taps something, the action is queued with two values created on the phone at the moment of the tap: an idempotency key (&lt;code&gt;crypto.randomUUID()&lt;/code&gt;) and &lt;code&gt;occurredAt&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;When the phone is back online, the queue is flushed in order, and it stops at the first failure:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;apps/web/app/receive/[token]/Receiving.tsx&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;flush&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;useCallback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;action&lt;/span&gt; &lt;span class="k"&gt;of &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;queuedActions&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;q&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;q&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;token&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nx"&gt;token&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;action&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;removeAction&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;action&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;break&lt;/span&gt;   &lt;span class="c1"&gt;// keep order: never apply "signed" before "started"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;run&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;token&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbndaygvizbpukgen5xpp.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbndaygvizbpukgen5xpp.webp" alt="Queued actions flush in order and stop at the first failure" width="800" height="550"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Stop at the first failure.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Stopping at the first failure keeps the order intact. If "start receipt" fails, sending "line 3 damaged" and "signed" after it would only produce confusing errors.&lt;/p&gt;

&lt;p&gt;One small bug to learn from: the original version set "All offline updates synced." after the loop whether it finished or hit &lt;code&gt;break&lt;/code&gt;. The message should only appear when the queue is actually empty.&lt;/p&gt;

&lt;h2&gt;
  
  
  Retries that cannot double count
&lt;/h2&gt;

&lt;p&gt;A phone that loses signal mid-request does not know whether its write landed. It will send it again. On the server, every delivery change is an event appended to a log, and the idempotency key is unique:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;packages/api/src/delivery-events.ts&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deliveryEvents&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="cm"&gt;/* ... */&lt;/span&gt;
    &lt;span class="na"&gt;occurredAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;occurredAt&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="nx"&gt;now&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;recordedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;recordedAt&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="nx"&gt;now&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;idempotencyKey&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="nf"&gt;newIdempotencyKey&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="na"&gt;deviceId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;deviceId&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;session&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;deviceId&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;})&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;onConflictDoNothing&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;deliveryEvents&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;idempotencyKey&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;refreshProjection&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;db&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;tenantId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tenantId&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="nx"&gt;deliveryId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffuhmfeiku2wy53bh1xyj.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffuhmfeiku2wy53bh1xyj.webp" alt="Inserting the event and rebuilding the projection in one transaction instead of two steps" width="800" height="563"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Both land, or neither does.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A repeated key does nothing and is not an error. The phone gets the current state back, the same answer it would have got the first time, and removes the item from its queue.&lt;/p&gt;

&lt;p&gt;The transaction around both statements came from a real bug. Originally the insert committed first, and then rebuilding the projection could throw on an illegal transition. The caller saw a failure, but the bad event was already in the log, and every later replay of that delivery hit the same error. With insert and projection in one transaction, either both land or neither does.&lt;/p&gt;

&lt;h2&gt;
  
  
  When two phones disagree
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F84zjj90ccqz7x86f6vke.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F84zjj90ccqz7x86f6vke.webp" alt="Two lanes, crew phone and office: the crew received at 09:12, the office cancelled at 09:20, the crew synced at 11:40" width="800" height="520"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The crew tapped first but synced last.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Offline writes arrive late, sometimes hours late. If the server processed events in arrival order, the office's 09:20 cancellation would win over a delivery the crew had actually received at 09:12. So events carry &lt;code&gt;occurredAt&lt;/code&gt; from the moment of the tap, and conflicts are resolved by when things happened, not when they arrived:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The earliest &lt;code&gt;occurredAt&lt;/code&gt; wins.&lt;/li&gt;
&lt;li&gt;An office cancellation is refused if the crew had already received.&lt;/li&gt;
&lt;li&gt;A losing event is never deleted. It is superseded by an &lt;code&gt;EventCorrected&lt;/code&gt; entry, so the full history stays readable.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The core that applies these rules is pure: no database, no network. A dependency rule enforces that the core imports no I/O, which keeps the rules easy to test. Trusting phone clocks needs care, and the native field app estimates its clock offset from the server, NTP style, before stamping events.&lt;/p&gt;

&lt;p&gt;Server-side failures to deliver notifications are handled the same patient way: retries back off at 2 s, 8 s, 30 s, 2 min and 10 min, then hourly, and errors are split into permanent and retryable. A 401 used to be retried forever. Now it is permanent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist for your own magic links
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Make the token the capability&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;At least 32 random bytes, base64url. Store only a hash. Return the plain token once.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Scope it to one thing&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Derive the resource from the token server-side. Never accept the resource ID from the client alongside it.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Check everything in one query&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Tenant, hash, expiry and revocation together, with the same error for every failure.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Give it its own identity&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A guest user per link keeps the audit trail honest.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Reuse your business logic&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Resolve the link, build a narrow session, call the same code your logged-in users hit.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Plan for offline from the start&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Idempotency keys and event times created on the device, an ordered queue, and a server that treats a repeated key as success.&lt;/p&gt;

&lt;p&gt;The project runs 91 tests plus a 30-test Playwright end-to-end suite, and the event core is kept free of I/O so most of the conflict rules can be tested without a database.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/magic-links-field-crews-offline" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>webdev</category>
      <category>postgres</category>
      <category>typescript</category>
    </item>
    <item>
      <title>SEO for a React SPA without Next.js, and without giving up the 3D homepage</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Fri, 06 Nov 2026 05:24:56 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/seo-for-a-react-spa-without-nextjs-and-without-giving-up-the-3d-homepage-2if5</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/seo-for-a-react-spa-without-nextjs-and-without-giving-up-the-3d-homepage-2if5</guid>
      <description>&lt;p&gt;This site opens on a small 3D room with an arcade cabinet in it. It is a Vite and React single-page app, and the 3D part loads behind a boot screen. That is great for a visitor and terrible for a search engine. The HTML a crawler receives is a &lt;code&gt;&amp;lt;div id="root"&amp;gt;&amp;lt;/div&amp;gt;&lt;/code&gt; and a script tag.&lt;/p&gt;

&lt;p&gt;The usual advice is "move to Next.js". I did not want to: the app is small, the 3D scene is the whole point, and a framework migration to get some HTML into a response felt out of proportion. This post shows what I did instead. It applies to any Vite or Create React App project that needs to be found.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a crawler actually needs
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpo2ql1l5sftklep57io2.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpo2ql1l5sftklep57io2.webp" alt="The empty SPA shell versus the prerendered classic page" width="800" height="610"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Same app, two very different first responses.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Search engines can run JavaScript, but rendering is deferred and not guaranteed, and other consumers (link previews in chat apps, social cards, smaller engines) usually do not run it at all. To be safe, a page that should rank needs in its first HTML response:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A unique &lt;code&gt;&amp;lt;title&amp;gt;&lt;/code&gt; and meta description.&lt;/li&gt;
&lt;li&gt;A canonical URL, so &lt;code&gt;/classic&lt;/code&gt; and &lt;code&gt;/classic/&lt;/code&gt; and &lt;code&gt;?ref=x&lt;/code&gt; count as one page.&lt;/li&gt;
&lt;li&gt;The real content as text: headings, paragraphs, links.&lt;/li&gt;
&lt;li&gt;Structured data that says what the page is about and who it belongs to.&lt;/li&gt;
&lt;li&gt;A sitemap that lists it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of that needs a framework. It needs the right HTML at the right URL.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhnm3jwbhus4waxa24n31.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhnm3jwbhus4waxa24n31.webp" alt="What a crawler gets for /, /classic, /blog and /sitemap.xml" width="800" height="499"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What a crawler gets for each URL.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: prerender the page that should rank
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxnarcvoit9j13vy7hx07.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxnarcvoit9j13vy7hx07.webp" alt="vite build, prerender script, classic.html and the Vercel rewrite" width="800" height="263"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Prerendering after the build.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The portfolio has two views of the same content: the 3D room at &lt;code&gt;/&lt;/code&gt; and a plain "Classic" view at &lt;code&gt;/classic&lt;/code&gt; that recruiters can skim. The Classic view is ordinary React with no WebGL, so it can render on the server. After &lt;code&gt;vite build&lt;/code&gt;, a small script loads the app through Vite's SSR loader and writes the result into a copy of the built &lt;code&gt;index.html&lt;/code&gt;:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;scripts/prerender.mjs&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;readFileSync&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;writeFileSync&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;node:fs&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;createServer&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;vite&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;vite&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;createServer&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;configFile&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;appType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;custom&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;logLevel&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;error&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;server&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;middlewareMode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;hmr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;ws&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;render&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;jsonLd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;PAGES&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;SITE&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;vite&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ssrLoadModule&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/src/prerender.tsx&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;template&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;readFileSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;dist/index.html&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;utf8&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;ld&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;`&amp;lt;script type="application/ld+json"&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nf"&gt;jsonLd&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;&lt;span class="s2"&gt;&amp;lt;/script&amp;gt;`&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;SITE&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;path&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;template&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&amp;lt;title&amp;gt;.*&lt;/span&gt;&lt;span class="se"&gt;?&lt;/span&gt;&lt;span class="sr"&gt;&amp;lt;&lt;/span&gt;&lt;span class="se"&gt;\/&lt;/span&gt;&lt;span class="sr"&gt;title&amp;gt;/&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;`&amp;lt;title&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;lt;/title&amp;gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;&amp;lt;link rel="canonical" href="&lt;/span&gt;&lt;span class="se"&gt;)[^&lt;/span&gt;&lt;span class="sr"&gt;"&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;*/&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;`$1&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;&amp;lt;meta property="og:url" content="&lt;/span&gt;&lt;span class="se"&gt;)[^&lt;/span&gt;&lt;span class="sr"&gt;"&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;*/&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;`$1&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;&amp;lt;meta property="og:title" content="&lt;/span&gt;&lt;span class="se"&gt;)[^&lt;/span&gt;&lt;span class="sr"&gt;"&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;*/&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;`$1&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;&amp;lt;/head&amp;gt;&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;`    &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;ld&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;\n  &amp;lt;/head&amp;gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;&amp;lt;div id="root"&amp;gt;&amp;lt;/div&amp;gt;&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;`&amp;lt;div id="root"&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;lt;/div&amp;gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="nf"&gt;writeFileSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;dist/index.html&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;page&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;PAGES&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;3d&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
  &lt;span class="nf"&gt;writeFileSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;dist/classic.html&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;page&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;PAGES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;classic&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;render&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;finally&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;vite&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;src/prerender.tsx&lt;/code&gt; is tiny: it exports &lt;code&gt;render()&lt;/code&gt;, which calls &lt;code&gt;renderToString(&amp;lt;App /&amp;gt;)&lt;/code&gt;, plus the page metadata and the JSON-LD builder. The build command becomes:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;package.json&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"scripts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"build"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"tsc -b &amp;amp;&amp;amp; vite build &amp;amp;&amp;amp; node scripts/prerender.mjs"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And Vercel serves the file for the clean URL:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;vercel.json&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"rewrites"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/classic"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"destination"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/classic.html"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  The one gotcha: the router during SSR
&lt;/h3&gt;

&lt;p&gt;On the server there is no &lt;code&gt;window.location&lt;/code&gt;, so the app does not know which page it is rendering. The router here is a tiny hook over &lt;code&gt;useSyncExternalStore&lt;/code&gt;, and its third argument is the snapshot used during server rendering. Pointing it at &lt;code&gt;/classic&lt;/code&gt; makes the prerender render the right view:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;src/lib/route.ts&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;usePath&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;useSyncExternalStore&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;subscribe&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;getPath&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/classic&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the browser the client renders over the same markup, so visitors get the normal app and crawlers get complete HTML. When the user switches views on the client, a small helper updates the title, canonical, description and Open Graph tags so a shared link always matches what is on screen.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: tell search engines who this is
&lt;/h2&gt;

&lt;p&gt;Structured data is how a page says "this is a person, this is their job, these are their profiles". The site emits one JSON-LD graph with three nodes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"@context"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://schema.org"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"@graph"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"@type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Person"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"@id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://mahmadmanzoor.com/#person"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Muhammad Ahmad"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"jobTitle"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Full-Stack &amp;amp; AI Product Engineer"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"address"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"@type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PostalAddress"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"addressLocality"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Lahore"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"addressCountry"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PK"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"knowsAbout"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"React"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Next.js"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Django"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Node.js"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"LLMs"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"sameAs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://github.com/mahmadmanzoor"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://www.linkedin.com/in/…"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"@type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"WebSite"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"@id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://mahmadmanzoor.com/#website"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"publisher"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"@id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://mahmadmanzoor.com/#person"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"@type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ProfilePage"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"mainEntity"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"@id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://mahmadmanzoor.com/#person"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg66usmyfi6fqn3wcpynv.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg66usmyfi6fqn3wcpynv.webp" alt="WebSite, ProfilePage and BlogPosting nodes all pointing at one Person node" width="800" height="577"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;One Person, referenced three ways.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;@id&lt;/code&gt; values let the nodes refer to each other, and the same Person &lt;code&gt;@id&lt;/code&gt; is reused as the author of every blog post. &lt;code&gt;sameAs&lt;/code&gt; links the site to profiles search engines already trust. The page title and description carry the phrases people actually search for, such as "software engineer in Lahore", in normal sentences. Stuffing them in reads badly and does not help.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: a blog that publishes without a redeploy
&lt;/h2&gt;

&lt;p&gt;Prerendering at build time works for pages that change when the code changes. Blog posts are written in a dashboard and saved to storage, so they need to be rendered when requested. A single Vercel function handles &lt;code&gt;/blog&lt;/code&gt; and &lt;code&gt;/blog/&amp;lt;slug&amp;gt;&lt;/code&gt;:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;vercel.json&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"functions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"api/blog-page.ts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"includeFiles"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"dist/blog.html"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rewrites"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/blog"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"destination"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/api/blog-page"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/blog/:slug"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"destination"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/api/blog-page?slug=:slug"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/sitemap.xml"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"destination"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/api/sitemap"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frhvv5v3eve6s6yy4367p.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frhvv5v3eve6s6yy4367p.webp" alt="How a blog URL is answered: rewrite, function, template lookup with two fallbacks, cached HTML" width="799" height="616"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;How a blog URL is answered.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The function reads the post, renders the article to HTML with every value escaped, and injects it into a template that the build writes next to &lt;code&gt;index.html&lt;/code&gt; (with the blog stylesheet already linked, so the article is styled before any JavaScript runs). &lt;code&gt;includeFiles&lt;/code&gt; bundles that template into the function. If it were ever missing, the function falls back to fetching &lt;code&gt;/blog.html&lt;/code&gt; from the deployment, and as a last resort to a minimal built-in page, so a configuration mistake degrades to plain HTML rather than a 500.&lt;/p&gt;

&lt;p&gt;Each post page gets its own title, description, canonical URL, Open Graph image (the post's cover) and a &lt;code&gt;BlogPosting&lt;/code&gt; JSON-LD block that points back to the same Person. The response is cached at the edge briefly:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;api/blog-page.ts&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;CACHE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;public, max-age=0, s-maxage=15, stale-while-revalidate=300&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Fifteen seconds at the edge means a burst of traffic is served from cache, while a fix to a typo shows up almost immediately.&lt;/p&gt;

&lt;h3&gt;
  
  
  The sitemap follows the content
&lt;/h3&gt;

&lt;p&gt;A static &lt;code&gt;sitemap.xml&lt;/code&gt; would need a redeploy for every post. Instead it is another small function that lists the fixed pages plus every published post, with its last-modified date. &lt;code&gt;robots.txt&lt;/code&gt; points to it. Publishing a post in the dashboard makes it appear in the sitemap on the next request.&lt;/p&gt;

&lt;p&gt;Two more details that matter for indexing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The private routes (&lt;code&gt;/sign-in&lt;/code&gt;, &lt;code&gt;/dashboard&lt;/code&gt;) send &lt;code&gt;X-Robots-Tag: noindex, nofollow&lt;/code&gt; and are left out of the sitemap.&lt;/li&gt;
&lt;li&gt;Preview deployments are &lt;code&gt;noindex&lt;/code&gt; too, so the same content on a &lt;code&gt;*.vercel.app&lt;/code&gt; URL never competes with the real domain.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 4: keep it fast
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs0fcoghdins0g2nrgn0m.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs0fcoghdins0g2nrgn0m.webp" alt="A 250 KB budget against 82 KB used, and four Lighthouse scores" width="800" height="727"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Kept fast.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Rankings and Lighthouse both care about speed, and a 3D site is easy to make slow. The rules that kept it fast:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The Classic view ships no Three.js at all. The 3D chunk loads only behind the boot screen on &lt;code&gt;/&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A CI budget fails the build if the initial JS and CSS go over 250 KB gzipped. It currently sits around 82 KB.&lt;/li&gt;
&lt;li&gt;Screenshots are WebP, at most 1280 px wide, with smaller thumbnails for cards.&lt;/li&gt;
&lt;li&gt;Long-lived &lt;code&gt;immutable&lt;/code&gt; cache headers on hashed assets.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The Classic view scores 99 for performance and 100 for SEO, accessibility and best practices on mobile Lighthouse.&lt;/p&gt;

&lt;h2&gt;
  
  
  After deploying
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Verify the domain in Search Console&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Use a DNS TXT record so it covers every subdomain and protocol.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Submit the sitemap&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Submit &lt;code&gt;https://your-domain/sitemap.xml&lt;/code&gt; once. Because it is generated, you do not need to submit it again for new posts.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Request indexing for the key pages&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Use URL Inspection on the prerendered page and check "View crawled page" shows real content, not an empty root div.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Validate the structured data&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Run the page through the Rich Results Test and the schema.org validator.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Link to it from places that are already indexed&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;GitHub profile and repository homepages, LinkedIn, and cross-posts of articles that set a canonical URL back to the original.&lt;/p&gt;

&lt;h2&gt;
  
  
  When you should just use a framework
&lt;/h2&gt;

&lt;p&gt;This approach fits a small site with a handful of indexable routes and one dynamic section. If most of your pages are data-driven, need per-request personalisation, or you find yourself writing a router, a cache layer and a template system, that is the point where Next.js, Remix or Astro will save you time. For this site, about 60 lines of prerender script and one function were enough.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/seo-react-spa-vite-without-nextjs" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>seo</category>
      <category>react</category>
      <category>vite</category>
      <category>webdev</category>
    </item>
    <item>
      <title>I built a Claude Code skill that makes video ads without a video model</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Thu, 05 Nov 2026 10:38:37 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/i-built-a-claude-code-skill-that-makes-video-ads-without-a-video-model-51op</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/i-built-a-claude-code-skill-that-makes-video-ads-without-a-video-model-51op</guid>
      <description>&lt;p&gt;Short vertical video ads are the cheapest reach a small product can buy, and the most annoying thing to make. A 15 second Instagram Reel needs a script with a hook, a character who looks the same in every shot, a voice, captions that match the voice, music that ducks under the voice, a sound on every tap, and three sizes for three feeds. Then someone asks to change one line.&lt;/p&gt;

&lt;p&gt;I tried the obvious route first: text-to-video models. The clips looked good one at a time and fell apart as an ad. The character's face drifted between shots, on-screen text came out as nonsense letters, and every change meant paying to regenerate a whole clip and hoping. So I built &lt;strong&gt;reel-studio&lt;/strong&gt;, a Claude Code skill that makes the whole reel with &lt;strong&gt;no video model anywhere&lt;/strong&gt;. The images come from an image model, but all the motion is drawn in code, so changing one line and re-rendering takes about a minute and costs nothing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flf0qd4ylxnkaxd1wi62i.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flf0qd4ylxnkaxd1wi62i.webp" alt="Animated clip of a reel hook made with reel-studio: an orange house mascot drops in beside a cartoon roofer holding a nail gun, the frame shakes, the title WRONG TOOL! slams in, and the caption Dale! Wrong tool! follows the voice." width="566" height="354"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The hook of a reel rebuilt for this post from the skill's saved library. Everything moving here is code: the drop, the shake, the mouth flaps, the captions.&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; The reels in this post advertise &lt;strong&gt;CredibleAI&lt;/strong&gt;, which runs a contractor's Facebook and Instagram every week from their real job photos, reviews and local news, with each post approved before it goes out. If you run a trade business or know someone who does, &lt;a href="https://credibley.com/signup" rel="noopener noreferrer"&gt;start the 7-day free trial at credibley.com&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What reel-studio is
&lt;/h2&gt;

&lt;p&gt;reel-studio is a skill: a folder Claude Code loads when you ask for a reel, holding a &lt;code&gt;SKILL.md&lt;/code&gt; with the method, some reference files, a few scripts and a small render engine. You type something like "make a 15 second Reel for CredibleAI with two characters" and Claude runs the whole production with you, stopping at every creative decision.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Output:&lt;/strong&gt; MP4s at 1080x1920 (9:16), plus 4:5 and 1:1 feed cuts from the same timeline, A/B hook variants, a no-captions cut, a cover image and the post caption with hashtags per platform.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scripts and images:&lt;/strong&gt; ChatGPT through the Codex CLI, on the ChatGPT plan I already pay for. No API key.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Voices and sound effects:&lt;/strong&gt; ElevenLabs (Fish Audio as a fallback), always behind a credit estimate and an explicit OK.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Music and stock effects:&lt;/strong&gt; a local library first, then Envato Elements through Claude in Chrome.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Motion:&lt;/strong&gt; HTML and JavaScript, screenshotted frame by frame with Playwright and encoded with ffmpeg.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Caption timing:&lt;/strong&gt; whisper.cpp, locally, because Claude cannot hear audio.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Size:&lt;/strong&gt; about 2,600 lines including the reference docs. The engine is 550 lines and the renderer 289.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The whole skill is open source under MIT at &lt;strong&gt;&lt;a href="https://github.com/mahmadmanzoor/reel-studio" rel="noopener noreferrer"&gt;github.com/mahmadmanzoor/reel-studio&lt;/a&gt;&lt;/strong&gt;. Install it with one &lt;code&gt;git clone&lt;/code&gt; into &lt;code&gt;~/.claude/skills/&lt;/code&gt; and ask Claude Code for a reel. If it saves you time, a star on the repo is the best way to tell me it's worth building more of these.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/mahmadmanzoor/reel-studio" rel="noopener noreferrer"&gt;mahmadmanzoor/reel-studio&lt;/a&gt;: Claude Code skill that makes Reels, TikTok and Shorts ads with characters, voiceover, music and captions. No video model: motion is drawn in code.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;git clone https://github.com/mahmadmanzoor/reel-studio ~/.claude/skills/reel-studio
&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cd&lt;/span&gt; ~/.claude/skills/reel-studio &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; npm &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; npx playwright &lt;span class="nb"&gt;install &lt;/span&gt;chromium
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The pipeline: a pick at every stage
&lt;/h2&gt;

&lt;p&gt;The rule at the top of &lt;code&gt;SKILL.md&lt;/code&gt; is the one that makes the whole thing usable: "Every stage ends with the user's choice; never start the next stage until they have chosen." An agent that writes a script, draws images, buys voice lines and renders a video in one go will always get something wrong early, and you only find out after it has spent money on everything downstream.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frfd0h9gy9xpql94n9drh.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frfd0h9gy9xpql94n9drh.webp" alt="The reel-studio pipeline: a one-time tools check, then brief, style, five scripts, character sheet, images, voices, sound effects, music, final cut and render, post caption, deliver. Each stage ends with the user's pick; paid audio waits for an OK." width="800" height="384"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Every arrow is a stop where you choose. The script round always offers five, the music round five tracks, each sound cue five candidates.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Everything chosen goes into one file, &lt;code&gt;reel.json&lt;/code&gt;, the moment it is chosen. That file has a &lt;code&gt;stage&lt;/code&gt; key, so if the session dies or I come back the next day, a new session reads it and carries on from the right stage. It also means the final reel is fully described by one JSON file plus the assets it points at.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;reel.json&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"stage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"brief | style | scripts | sheet | images | voices | sfx | music | render | done"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"brief"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"product"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"platform"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"length"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"captions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"hookVariants"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"cta"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"style"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"humor skit"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"script"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"...the locked script JSON..."&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"characters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"dale"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"sheet"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ref/dale.jpg"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"voice"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"music"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"envato"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"TRCFCAX"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"author"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"timeline"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"...what the renderer draws..."&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Why no video model
&lt;/h2&gt;

&lt;p&gt;A video model gives you pixels. You can't move a caption 40 pixels up, swap one line of dialogue, or make the 4:5 version, without generating again. What I wanted was closer to a motion graphics project file: a description of the reel that a program turns into frames, so every change is an edit and a re-render.&lt;/p&gt;

&lt;p&gt;So the engine treats the reel as a &lt;strong&gt;pure function of time&lt;/strong&gt;. The comment at the top of &lt;code&gt;engine.js&lt;/code&gt; is the whole design:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;engine/engine.js&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="cm"&gt;/*
  A reel is a pure function of time: window.seek(t) paints frame t from window.REEL (the timeline that
  render.mjs builds from reel.json). No CSS transitions or animations, no timers, no Math.random, no state
  carried between frames, so any frame can be drawn in any order by any number of browsers.
*/&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every animation is a function of &lt;code&gt;t&lt;/code&gt;. Easing is computed, not left to CSS: there is a cubic Bezier solver for whips and glides and a damped spring for pops and slams, so a character dropping in overshoots and settles the same way every time.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;engine/engine.js&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="cm"&gt;/** Damped spring step response, 0 → 1 over x in 0..1. Lower z = more overshoot. */&lt;/span&gt;
&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;springer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;z&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.62&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;wd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;w&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cos&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;wd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;w&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nx"&gt;wd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;wd&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
  &lt;span class="k"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;w&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cos&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;wd&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;w&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nx"&gt;wd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;wd&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nx"&gt;end&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Even the "random" parts are deterministic. The talking mouth flaps at an uneven rhythm so it doesn't look mechanical, but the rhythm comes from a seeded generator (&lt;code&gt;seed = (seed * 16807) % 2147483647&lt;/code&gt;), so frame 212 is identical on every render.&lt;/p&gt;

&lt;h2&gt;
  
  
  The timeline: a scene is a list of layers
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;timeline&lt;/code&gt; key is what gets drawn. Scenes run in order and each one has a duration; start times are computed. Inside a scene there are layers: a background, characters with poses, text, UI cards and real product screenshots. Here is the hook of the reel at the top of this post, trimmed:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;reel.json (timeline.scenes[0])&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"hook"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"dur"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2.4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"bg"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"gradient"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"radial-gradient(circle at 50% 35%, #fbe3c2, #e8a76a)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"move"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"push"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"shake"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.15&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"amount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"chars"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"dale"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"x"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.36&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"base"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.95&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"h"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"sway"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"poses"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"src"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"images/dale-nailgun.png"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"homer"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"x"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.72&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"base"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"h"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"enter"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"drop"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"d"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.45&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"poses"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"src"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"images/homer-dive.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"open"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"images/homer-dive-open.png"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"texts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"WRONG&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;*TOOL!*"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"style"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"slam"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"y"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.09&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"size"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Positions are fractions of the frame, times are seconds from the scene's start, and &lt;code&gt;*TOOL!*&lt;/code&gt; means "draw this word in the brand accent". A pose with an &lt;code&gt;open&lt;/code&gt; image gets mouth flaps whenever that character's voice is loud. Voice lines and sound effects sit in their own arrays and point at a scene and a time:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;reel.json (timeline)&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"voice"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"char"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"homer"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"file"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"audio/vo/01.mp3"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"words"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"audio/vo/01.words.json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"scene"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"hook"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.25&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"emphasis"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"tool"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"char"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"dale"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"file"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"audio/vo/03.mp3"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"words"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"audio/vo/03.words.json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"scene"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"gruff"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"phrases"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"I fix roofs."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"I don't / write captions."&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"sfx"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"file"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"audio/sfx/LBTZJA6.wav"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"scene"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"hook"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"gain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;-6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"music"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"file"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"audio/music/A7UWZ9T.wav"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"offset"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"gain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;-8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"duck"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because sound is attached to scenes rather than to absolute times, making a scene 0.3 seconds longer moves every later line and effect with it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rendering: Chromium as a frame printer
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;render.mjs&lt;/code&gt; opens the engine in headless Chromium through Playwright, steps through time, screenshots each frame and pipes the PNGs into ffmpeg. Because frames don't depend on each other, it opens four browsers and lets them race.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fumd03a2yessaqiv0bfup.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fumd03a2yessaqiv0bfup.webp" alt="How render.mjs works: four headless Chromium pages seek and screenshot frames, a reorder buffer feeds ffmpeg. Voice is compressed and ducks the music, everything is mixed, then mastered to -14 LUFS and re-checked after the AAC encode." width="800" height="457"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Picture and sound are built separately and muxed at the end.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Four browsers finish frames out of order, but ffmpeg needs them in order. The renderer keeps finished frames in a map keyed by frame number and only writes when the next one ffmpeg needs is ready:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;render.mjs&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;pending&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;next&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;write&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;flushing&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;flush&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;flushing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;flushing&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;write&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;write&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="nx"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;delete&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;write&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stdin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;b&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;once&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stdin&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;drain&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="nx"&gt;write&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="nx"&gt;flushing&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ws&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(;;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;next&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nx"&gt;total&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nx"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;grab&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;w&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nx"&gt;FPS&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;flush&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;write&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;48&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;setTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;   &lt;span class="c1"&gt;// don't run too far ahead&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;grab&lt;/code&gt; seeks the page, waits two animation frames so layout has settled, and takes the screenshot directly over the Chrome DevTools Protocol (&lt;code&gt;Page.captureScreenshot&lt;/code&gt; with &lt;code&gt;optimizeForSpeed: true&lt;/code&gt;). The video is encoded with libx264 at CRF 17 and tagged bt709 so colours don't shift on phones.&lt;/p&gt;

&lt;p&gt;On my M1 Pro MacBook the reel in this post, 17.9 seconds at 30 fps, rendered like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;node render.mjs demo
picture: 537 frames &lt;span class="k"&gt;in &lt;/span&gt;26s
&lt;span class="o"&gt;{&lt;/span&gt;
 &lt;span class="s2"&gt;"file"&lt;/span&gt;: &lt;span class="s2"&gt;".../demo/out/reel.mp4"&lt;/span&gt;,
 &lt;span class="s2"&gt;"duration"&lt;/span&gt;: 17.9,
 &lt;span class="s2"&gt;"captions"&lt;/span&gt;: &lt;span class="nb"&gt;true&lt;/span&gt;,
 &lt;span class="s2"&gt;"captionGroups"&lt;/span&gt;: 14,
 &lt;span class="s2"&gt;"speakers"&lt;/span&gt;: &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"homer"&lt;/span&gt;, &lt;span class="s2"&gt;"dale"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;,
 &lt;span class="s2"&gt;"master"&lt;/span&gt;: &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;"I"&lt;/span&gt;: &lt;span class="nt"&gt;-14&lt;/span&gt;.1, &lt;span class="s2"&gt;"TP"&lt;/span&gt;: &lt;span class="nt"&gt;-1&lt;/span&gt;.2 &lt;span class="o"&gt;}&lt;/span&gt;,
 &lt;span class="s2"&gt;"final"&lt;/span&gt;: &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;"I"&lt;/span&gt;: &lt;span class="nt"&gt;-14&lt;/span&gt;.1, &lt;span class="s2"&gt;"TP"&lt;/span&gt;: &lt;span class="nt"&gt;-1&lt;/span&gt;.1 &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  The sound: measure after the encode, not before
&lt;/h3&gt;

&lt;p&gt;Every voice clip is delayed to its line's time, the voices are mixed and gently compressed, and the compressed voice is split in two: one copy goes to the mix, the other becomes the key of a sidechain compressor on the music. The music dips whenever someone speaks and comes back in the gaps, with no volume keyframes to maintain.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;render.mjs&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;vs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;chains&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;vs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;&lt;span class="s2"&gt;amix=inputs=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;vs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:normalize=0:duration=first,atrim=0:&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;D&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;,acompressor=threshold=0.08:ratio=3:attack=5:release=120:makeup=1,asplit=2[voice][key]`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="c1"&gt;// ...&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;vs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;chains&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`[mus][key]sidechaincompress=threshold=0.02:ratio=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;M&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;duck&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:attack=15:release=350:makeup=1[music]`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the master goes to &lt;strong&gt;-14 LUFS&lt;/strong&gt;, the loudness Instagram, TikTok and YouTube normalize to, with a true peak under &lt;strong&gt;-1 dB&lt;/strong&gt;. The first version measured the peak on the WAV, saw -1 dB, and shipped. The final MP4 measured -0.8 dB, because AAC encoding adds inter-sample peaks the limiter never saw. Now the loop measures the file people will actually download:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;render.mjs&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// the true peak is measured on the encoded file; if AAC pushed it over -1 dB, lower the ceiling and mux again&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;ceiling&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;ceiling&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="mf"&gt;0.75&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;rep&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;buildAudio&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;audio&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;ceiling&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nf"&gt;ff&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-i&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;video&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-i&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;audio&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-map&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;0:v:0&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-map&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;1:a:0&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-c:v&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;copy&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-c:a&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;aac&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-b:a&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;192k&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;...]);&lt;/span&gt;
  &lt;span class="nx"&gt;fin&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;loud&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;final&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;   &lt;span class="c1"&gt;// ffmpeg ebur128=peak=true on the MP4&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;rep&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;silent&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;fin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;TP&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;ceiling&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip:&lt;/strong&gt; One more bug worth knowing if you build anything like this: an ffmpeg &lt;code&gt;apad&lt;/code&gt; with no length on an input that feeds &lt;code&gt;amix&lt;/code&gt; or a sidechain can make ffmpeg spin for over an hour. Every input here is padded and trimmed to exactly the reel length (&lt;code&gt;apad=whole_dur=D,atrim=0:D&lt;/code&gt;) before it touches anything else.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Voices, mouths and captions
&lt;/h2&gt;

&lt;p&gt;Claude can read a script but it cannot hear the voice it just paid for. Two small programs listen on its behalf.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcddah5sxgih9hkimfikh.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcddah5sxgih9hkimfikh.webp" alt="One voice line drives two things: per-frame loudness gated at 14% of peak flaps the mouth every 2 to 4 frames, and whisper.cpp word timings aligned to the script with difflib become caption groups that never cross a cut." width="800" height="351"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The voice clip is the source of truth for both.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  The mouth
&lt;/h3&gt;

&lt;p&gt;Each speaking pose is drawn twice, closed and open. &lt;code&gt;render.mjs&lt;/code&gt; decodes the clip to 16 kHz mono, measures RMS loudness for every video frame, and builds a string of &lt;code&gt;0&lt;/code&gt;s and &lt;code&gt;1&lt;/code&gt;s per character that the engine reads as it draws:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;render.mjs&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;peak&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(...&lt;/span&gt;&lt;span class="nx"&gt;rms&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nx"&gt;gate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;peak&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.14&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;mouth&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;v&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;char&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||=&lt;/span&gt; &lt;span class="nc"&gt;Array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;N&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;fill&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="c1"&gt;// the mouth opens and shuts at an uneven 2-4 frame rhythm while the voice is loud, and stays shut in pauses&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;open&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;hold&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nx"&gt;rms&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;forEach&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;f&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;at&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;FPS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;f&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;f&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nx"&gt;N&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;gate&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;f&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;0&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;open&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;hold&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;hold&lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;open&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;open&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;hold&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;rnd&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;f&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;open&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;1&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;0&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is not lip sync, and for cartoon characters it doesn't need to be. The mouth moves only while that character is talking, stops dead in the pauses, and never moves for the character who is listening.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy2b3msypfu4jjb1qhl8s.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy2b3msypfu4jjb1qhl8s.webp" alt="Two crops of the same cartoon roofer: on the left with his mouth closed, on the right with his mouth open. Everything except the mouth is identical." width="800" height="427"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The two poses the engine flips between. Only the mouth may differ, or the face jitters.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Getting the open pose right took a script of its own. The image model can make an open-mouth edit of a pose, but it moves other things a few pixels too, and flipping between the two images makes the eyes twitch. &lt;code&gt;img.py remouth&lt;/code&gt; fixes that: it fills the closed mouth with the face colour and pastes in only the connected dark region that grew between the two poses, seeded from the new pixels, with its holes filled so the tongue comes along.&lt;/p&gt;

&lt;h3&gt;
  
  
  The captions
&lt;/h3&gt;

&lt;p&gt;Captions are timed from the audio, not the script. &lt;code&gt;words.py&lt;/code&gt; runs whisper.cpp locally with one word per segment, then aligns what Whisper heard to what the script says. The script wins on spelling (Whisper hears "credible AI", the script says "CredibleAI"); Whisper wins on timing.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;scripts/words.py&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;said&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;\[[^\]]*\]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;          &lt;span class="c1"&gt;# drop delivery tags like [excited]
&lt;/span&gt;&lt;span class="n"&gt;sm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;difflib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SequenceMatcher&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;heard&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;said&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;autojunk&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;timed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;said&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j2&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;sm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_opcodes&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;equal&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;replace&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;i2&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;i1&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;j2&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;j1&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j2&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;j1&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;timed&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;j1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;heard&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;heard&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;b&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;replace&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="c1"&gt;# e.g. "credible AI" heard for "CredibleAI": spread the heard span over the written words
&lt;/span&gt;        &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;heard&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;heard&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i2&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;b&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j2&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;j1&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;timed&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;j1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j2&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;j1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j2&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;j1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The output for one line looks like this, and it is what drives the caption on screen:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;audio/vo/03.words.json&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="nl"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"I"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.16&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"fix"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.63&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"roofs."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.63&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.62&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"I"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.62&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.68&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"don't"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.68&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.98&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"write"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.98&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2.28&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"captions."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2.28&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;3.2&lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The renderer groups words into captions of up to three words or 20 characters, breaking at punctuation and at pauses over 0.35 seconds. A group never spans a cut, and it is gone two frames before its shot ends, so you never see a word flash across a transition. Words listed under &lt;code&gt;emphasis&lt;/code&gt; grow and turn the accent colour when spoken.&lt;/p&gt;

&lt;h2&gt;
  
  
  Images through Codex, on a plan I already pay for
&lt;/h2&gt;

&lt;p&gt;Scripts and images go through the Codex CLI signed in with ChatGPT, so they bill against the ChatGPT plan rather than an API key. The wrapper is 32 lines of zsh. For images it asks Codex to use its built-in image tool and copy the PNG to an exact path, then checks the file exists:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;scripts/codex.sh&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nv"&gt;$mode&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; text &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;codex &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nt"&gt;-m&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$model&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--skip-git-repo-check&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; read-only &lt;span class="nt"&gt;-C&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$dir&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; - &amp;lt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$prompt&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$log&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; 2&amp;gt;&amp;amp;1
&lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nv"&gt;$mode&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; image &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nv"&gt;refs&lt;/span&gt;&lt;span class="o"&gt;=()&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;r &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do &lt;/span&gt;refs+&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;r&lt;/span&gt;:A&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
  &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$prompt&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; print &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s2"&gt;Use your built-in image generation tool. Then copy the generated PNG to exactly this path: &lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;Reply with only that path."&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt; |
    codex &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nt"&gt;-m&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$model&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--skip-git-repo-check&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; workspace-write &lt;span class="nt"&gt;-C&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$dir&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;refs&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; - &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$log&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; 2&amp;gt;&amp;amp;1
&lt;span class="k"&gt;fi

if&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; &lt;span class="nv"&gt;$out&lt;/span&gt; &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;print &lt;span class="nt"&gt;-u2&lt;/span&gt; &lt;span class="s2"&gt;"codex produced no &lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt; — last lines of &lt;/span&gt;&lt;span class="nv"&gt;$log&lt;/span&gt;&lt;span class="s2"&gt;:"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;tail&lt;/span&gt; &lt;span class="nt"&gt;-5&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$log&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The prompt goes in on stdin because &lt;code&gt;-i&lt;/code&gt; takes several files and would swallow a prompt passed as an argument. Text runs are read-only; only image runs get write access, and only to the project folder.&lt;/p&gt;

&lt;p&gt;Character consistency comes from a &lt;strong&gt;reference sheet&lt;/strong&gt;. Before any scene image exists, the skill makes one sheet per character and waits for approval. Every later pose is generated with that sheet attached as a reference image.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv9pblw065xxp0paw1xy4.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv9pblw065xxp0paw1xy4.webp" alt="Character reference sheet for Dale, a gruff cartoon roofer: front, three-quarter, side and back views holding a phone, plus five head shots with different expressions." width="799" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Dale's approved sheet. Every pose in the reel was drawn from this.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A few rules in &lt;code&gt;SKILL.md&lt;/code&gt; came straight from bad renders:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One pose per image, full size, transparent background.&lt;/strong&gt; Pose grids come out around 500 px per cell and look soft at 1080 wide.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backgrounds separately, with nobody in them.&lt;/strong&gt; Characters and backgrounds are separate layers, so the camera can push on the background while the characters stay put.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never an AI mockup of the product's screens.&lt;/strong&gt; The demo uses the real screenshots you give it, or code-drawn notification and chat cards if you have none.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Money: estimates first, and a library so you pay once
&lt;/h2&gt;

&lt;p&gt;Voice and sound effect generation costs ElevenLabs credits, so the skill has one rule it never bends: before any paid generation it shows exactly what it will generate, runs ElevenLabs with &lt;code&gt;estimate_only: true&lt;/code&gt;, and waits for an explicit OK. Subagents are never allowed to spend.&lt;/p&gt;

&lt;p&gt;The second half of that is the library. Every character (sheet, poses, voice), every generated voice line, every licensed track and effect is kept under &lt;code&gt;library/&lt;/code&gt; and checked before anything is paid for again. Voice lines are keyed by provider, voice, model and the exact text with its delivery tags:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;node scripts/cast.mjs line get elevenlabs kSSJYNGo5Kev6qgwVloV eleven_v4 &lt;span class="s2"&gt;"[gruff] I fix roofs. I don't write captions."&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If that voice has said that line before, the file comes back for free. The reel in this post was rebuilt entirely from the library: two saved characters, six saved voice lines, a licensed track and four licensed effects. It cost &lt;strong&gt;zero credits&lt;/strong&gt;, and the only paid step in a normal run, the voices, was skipped because all six lines already existed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Working in parallel without losing control
&lt;/h2&gt;

&lt;p&gt;A run has a lot of independent work: three characters' images, a music shortlist, effect lookups, Whisper on each clip, several renders. The main session stays in charge of anything that needs a person or money and hands the rest out, each on the cheapest model that can do it well.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzy4buxopye4h8vm7ilxb.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzy4buxopye4h8vm7ilxb.webp" alt="The main session talks to the user, gets every pick and cost OK and owns reel.json. It hands work to background Bash, a haiku agent for lookups, up to three sonnet agents for images, a sonnet music agent and one opus agent for the final cut." width="800" height="343"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Cheap models for lookups, mid models for judgment, the big model only for the final cut.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The limits in that diagram are all things that broke. ElevenLabs returns &lt;code&gt;429 rate_limited&lt;/code&gt; if you send two calls at once (nothing is charged, but the call fails). Two renders running at the same time once wrote the same &lt;code&gt;mix.wav&lt;/code&gt;, and one reel came out with the other reel's audio; the mix file is now named per output. And only one agent may drive Chrome at a time, because two agents clicking in the same Envato tab do not end well.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checking the output before you see it
&lt;/h2&gt;

&lt;p&gt;Rendering 18 seconds takes under half a minute, but looking at a whole video to find one bad frame is slow. The renderer has review modes. &lt;code&gt;--stills scene&lt;/code&gt; makes a contact sheet with one frame from each shot, and &lt;code&gt;--safe&lt;/code&gt; draws the areas Reels and TikTok cover with their own UI: the bottom 20% (caption and username), the right 14% (like, comment and share buttons) and the top 7%.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F81i8zqhsarigt4tngfjq.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F81i8zqhsarigt4tngfjq.webp" alt="Contact sheet with one frame per scene: roofer and mascot under 'WRONG TOOL!', roofer in a hard hat, roofer with crossed arms, a CredibleAI screenshot with a notification, roofer smiling at his phone, and the CredibleAI end card." width="800" height="234"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;node render.mjs demo --stills scene: the whole reel at a glance.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgd2jfm26t3vev1vmhfbl.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgd2jfm26t3vev1vmhfbl.webp" alt="The same frame twice: on the left with red overlays marking the platform's bottom caption area, right-hand button column and top bar; on the right without them. The caption and the character's face sit outside the red areas." width="800" height="718"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;--safe draws what Instagram and TikTok will cover. Nothing important goes in the red.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;My first pass of this demo put the captions across Dale's face in three shots. The contact sheet showed it in one image, I moved the caption &lt;code&gt;y&lt;/code&gt; per scene, and re-rendered the sheet in a few seconds.&lt;/p&gt;

&lt;h3&gt;
  
  
  Hook variants and feed sizes from one file
&lt;/h3&gt;

&lt;p&gt;Ads are A/B tested on the first two seconds, so the timeline can carry alternative opening scenes. Scene 1 is hook 1 and &lt;code&gt;hooks[0]&lt;/code&gt; is hook 2. A hook brings its own voice and effects, because swapping only the picture once left scene 1's line playing under the new opening.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw6u9cu0hf7g71tg2329o.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw6u9cu0hf7g71tg2329o.webp" alt="Two opening frames side by side. Hook 1: the roofer and the mascot under 'WRONG TOOL!'. Hook 2: the roofer alone on a dark background under 'Roofers vs SOCIAL MEDIA'." width="800" height="718"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;node render.mjs demo --hook 2 renders the same reel with a different first scene.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Feed sizes come from the same timeline too. With &lt;code&gt;--size 4:5&lt;/code&gt; or &lt;code&gt;--size 1:1&lt;/code&gt; the engine remaps x into the safe width for that size (feeds have no button column), makes characters slightly bigger and text slightly smaller. Most reels need no overrides, and the ones that do get a small &lt;code&gt;timeline.sizes&lt;/code&gt; patch for just that size.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4too488ks7ca284c3m9o.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4too488ks7ca284c3m9o.webp" alt="The same shot rendered three ways: 9:16 vertical, 4:5 portrait and 1:1 square. The character, caption and background are laid out for each size without manual changes." width="800" height="354"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Same timeline, three sizes, no manual layout.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;terminal&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;node render.mjs &amp;lt;project&amp;gt;                    &lt;span class="c"&gt;# out/reel.mp4&lt;/span&gt;
node render.mjs &amp;lt;project&amp;gt; &lt;span class="nt"&gt;--no-captions&lt;/span&gt;      &lt;span class="c"&gt;# out/reel-nocaptions.mp4&lt;/span&gt;
node render.mjs &amp;lt;project&amp;gt; &lt;span class="nt"&gt;--hook&lt;/span&gt; 2           &lt;span class="c"&gt;# out/reel-hook2.mp4&lt;/span&gt;
node render.mjs &amp;lt;project&amp;gt; &lt;span class="nt"&gt;--size&lt;/span&gt; 4:5         &lt;span class="c"&gt;# out/reel-4x5.mp4 (also 1:1)&lt;/span&gt;
node render.mjs &amp;lt;project&amp;gt; &lt;span class="nt"&gt;--cover&lt;/span&gt;            &lt;span class="c"&gt;# out/cover.png&lt;/span&gt;
node render.mjs &amp;lt;project&amp;gt; &lt;span class="nt"&gt;--stills&lt;/span&gt; scene &lt;span class="nt"&gt;--safe&lt;/span&gt;
node render.mjs &amp;lt;project&amp;gt; &lt;span class="nt"&gt;--frame&lt;/span&gt; 6.0        &lt;span class="c"&gt;# one PNG at t = 6.0 s&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  A ready-made format: retro anime suspense
&lt;/h2&gt;

&lt;p&gt;The newest addition is a reel type with fixed beats: a late-1970s TV anime suspense scene builds fake tension ("you won't believe what this does"), hard-cuts to a warm hand-held phone demo of the real product, and ends on a shocked face. The story changes every time; the structure, look, sound and motion rules don't.&lt;/p&gt;

&lt;p&gt;Fixed formats are where an agent can be held to rules, so this one comes with a scaffold and a checker. The scaffold writes a timeline that already follows the format, plus the list of images to make:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;node scripts/retro-scaffold.mjs reels/demo &lt;span class="nt"&gt;--cut&lt;/span&gt; core &lt;span class="nt"&gt;--app&lt;/span&gt; &lt;span class="s2"&gt;"Demo App"&lt;/span&gt; &lt;span class="nt"&gt;--accent&lt;/span&gt; &lt;span class="s2"&gt;"#0E9F6E"&lt;/span&gt;
scaffolded core &lt;span class="nb"&gt;cut&lt;/span&gt;, 10 scenes, 31.49 s: reels/demo
next: fill slots &lt;span class="k"&gt;in &lt;/span&gt;reel.json, write the lines, &lt;span class="k"&gt;then &lt;/span&gt;make the 19 images &lt;span class="k"&gt;in &lt;/span&gt;SHOTS.md
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The checker reads &lt;code&gt;reel.json&lt;/code&gt; and fails the build on the mistakes that were corrected by hand in earlier reels. Every rule in it was a piece of feedback first:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;scripts/retro-check.mjs&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;byId&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;phone&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)?.&lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;fail&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`"phone" has an "in" transition: the reveal must be a hard cut`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;voice&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;v&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;scene&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;gulp&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="nf"&gt;fail&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`"gulp" has speech: it must be silent`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;scenes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;chars&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="p"&gt;[])&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;still&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;fail&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`"&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;": character "&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;" lacks still: true, idle motion reads as bubbling`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;punch&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;fail&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`"&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;": has punch-in zooms, add a real shot instead`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;captions&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;captions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;style&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;steady&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;fail&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`captions.style is "&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;captions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;style&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;", use "steady"`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run against the fresh scaffold, before any images exist, it does what it should:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;node scripts/retro-check.mjs reels/demo
FAIL  18 missing file&lt;span class="o"&gt;(&lt;/span&gt;s&lt;span class="o"&gt;)&lt;/span&gt;: images/bg-world.png, images/listener-listen.png, ...
warn  &lt;span class="s2"&gt;"device"&lt;/span&gt; has no tap markers yet

10 scenes, 31.49 s, reveal at 8.75 s: 1 failed, 1 warning&lt;span class="o"&gt;(&lt;/span&gt;s&lt;span class="o"&gt;)&lt;/span&gt;
This checks the rules, not the look. Still watch the frames and listen.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What it got wrong in real use
&lt;/h2&gt;

&lt;p&gt;The skill keeps a &lt;code&gt;LOG.md&lt;/code&gt; of every mistake from real runs. An entry moves into &lt;code&gt;SKILL.md&lt;/code&gt; as a rule once the same thing happens twice. Some of the entries that changed how I build these tools:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Why are the characters bubbling up and down?"&lt;/strong&gt; No single frame looked wrong. The idle breathing, a talking bob and a slow camera push together made flat cutouts jitter once the video played. Characters in that format are now &lt;code&gt;still: true&lt;/code&gt; and the checker enforces it. Single frames lie about motion.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fast means more shots, not more camera.&lt;/strong&gt; Asked for a faster cut, I added zooms on the talking shots and was told most of those frames were pointless. Seven real shots at the start fixed it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A second music track never blends.&lt;/strong&gt; A heavier track for the opening fought the main groove at the cut. Extending the original track's tense intro, so its lift still landed on the reveal, worked.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A stock track can pass every check and still be unavailable.&lt;/strong&gt; The picked track showed "Subscribe to download" only at the last step. Now download availability is checked before a track is offered.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Whisper smeared a slow narrator.&lt;/strong&gt; A deep, slow voice with long pauses got word timings that ran through the silences, so captions led the voice. &lt;code&gt;ffmpeg silencedetect&lt;/code&gt; showed the real gaps. Slow reads get that check now.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A silent no-op in Pillow.&lt;/strong&gt; &lt;code&gt;ImageDraw.floodfill&lt;/code&gt; on an image made with &lt;code&gt;Image.fromarray&lt;/code&gt; changed nothing and raised nothing, so every pixel counted as inside the mouth. &lt;code&gt;.copy()&lt;/code&gt; first. If a fix appears to do nothing, count the changed pixels.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Off-by-one in the hook numbering.&lt;/strong&gt; &lt;code&gt;--hook 2&lt;/code&gt; first read &lt;code&gt;hooks[1]&lt;/code&gt;, so the first alternative hook could never be rendered. Scene 1 is hook 1 now, and the error message names the index.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Build the core yourself in 40 lines
&lt;/h2&gt;

&lt;p&gt;You can install the full skill from &lt;a href="https://github.com/mahmadmanzoor/reel-studio" rel="noopener noreferrer"&gt;the repo&lt;/a&gt;, but the idea that makes it work fits in two small files, and you can run it today. A page exposes &lt;code&gt;seek(t)&lt;/code&gt;, a script drives it with Playwright and pipes frames to ffmpeg. I ran exactly this to check it.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Install Playwright and ffmpeg&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Any recent Node works. ffmpeg comes from Homebrew on a Mac.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   &lt;span class="nv"&gt;$ &lt;/span&gt;npm i playwright
   &lt;span class="nv"&gt;$ &lt;/span&gt;npx playwright &lt;span class="nb"&gt;install &lt;/span&gt;chromium
   &lt;span class="nv"&gt;$ &lt;/span&gt;brew &lt;span class="nb"&gt;install &lt;/span&gt;ffmpeg
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Write a scene that is a function of time&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;No CSS animations, no timers, no randomness. &lt;code&gt;seek(t)&lt;/code&gt; sets every style from &lt;code&gt;t&lt;/code&gt; alone.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;   &amp;lt;!doctype html&amp;gt;
   &amp;lt;html&amp;gt;&amp;lt;body style="margin:0;width:1080px;height:1920px;overflow:hidden;background:#f6d7b0;font:900 120px system-ui"&amp;gt;
   &amp;lt;div id="title" style="position:absolute;left:0;right:0;top:300px;text-align:center"&amp;gt;WRONG TOOL!&amp;lt;/div&amp;gt;
   &amp;lt;div id="ball" style="position:absolute;width:220px;height:220px;border-radius:50%;background:#f28c28;left:430px"&amp;gt;&amp;lt;/div&amp;gt;
   &amp;lt;script&amp;gt;
     // A frame is a pure function of t: no timers, no Math.random, no state between frames.
     const spring = (x, z = 0.45, w = 14) =&amp;gt; x &amp;lt;= 0 ? 0 : 1 - Math.exp(-z * w * x) * Math.cos(w * Math.sqrt(1 - z * z) * x);
     window.seek = (t) =&amp;gt; {
       const k = Math.min(1, t / 0.6);
       title.style.transform = `scale(${0.4 + 0.6 * spring(k)})`;
       ball.style.top = `${1500 - 900 * Math.abs(Math.sin(t * 2.4)) }px`;
     };
     window.seek(0);
   &amp;lt;/script&amp;gt;&amp;lt;/body&amp;gt;&amp;lt;/html&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Render it with four browsers&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Each browser takes the next frame number, seeks, screenshots, and hands the PNG to an in-order writer that feeds ffmpeg.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;   &lt;span class="c1"&gt;// node render.mjs scene.html out.mp4 3   -&amp;gt; 3 seconds at 30 fps, 4 browsers in parallel&lt;/span&gt;
   &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;chromium&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;playwright&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
   &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;spawn&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:child_process&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
   &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;once&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:events&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
   &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;pathToFileURL&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:url&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
   &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:path&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

   &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;scene.html&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;out.mp4&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;secs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;3&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
   &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;FPS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="nx"&gt;secs&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;FPS&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nx"&gt;WORKERS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
   &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;enc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;spawn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;ffmpeg&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-v&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;error&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-y&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-f&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;image2pipe&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-c:v&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;png&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-framerate&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;FPS&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-i&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
     &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-c:v&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;libx264&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-crf&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;17&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-pix_fmt&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;yuv420p&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;-movflags&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;+faststart&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;stdio&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pipe&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;inherit&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;inherit&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;

   &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
     &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;chromium&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;launch&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
     &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;newPage&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;viewport&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;width&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1080&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;height&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1920&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
     &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;goto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;pathToFileURL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nx"&gt;href&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
     &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
   &lt;span class="p"&gt;}&lt;/span&gt;
   &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;ws&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;length&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;WORKERS&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="nx"&gt;open&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;

   &lt;span class="c1"&gt;// frames finish out of order; hold them until the next one ffmpeg needs is ready&lt;/span&gt;
   &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;done&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;next&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;write&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
   &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;flush&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
     &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;done&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;write&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;done&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;write&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="nx"&gt;done&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;delete&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;write&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="nx"&gt;write&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stdin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;b&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;once&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stdin&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;drain&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
   &lt;span class="p"&gt;}&lt;/span&gt;
   &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ws&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
     &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;next&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;total&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
       &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;window&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;seek&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nx"&gt;FPS&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
       &lt;span class="nx"&gt;done&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;screenshot&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;png&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;}));&lt;/span&gt;
       &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;flush&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
     &lt;span class="p"&gt;}&lt;/span&gt;
   &lt;span class="p"&gt;}));&lt;/span&gt;
   &lt;span class="nx"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stdin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;once&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;close&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
   &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ws&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;browser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;()));&lt;/span&gt;
   &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;: &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;total&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; frames`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Run it&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Three seconds of 1080x1920 video, 90 frames, in a little over two seconds on my machine.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   &lt;span class="nv"&gt;$ &lt;/span&gt;node render.mjs scene.html out.mp4 3
   out.mp4: 90 frames
   &lt;span class="nv"&gt;$ &lt;/span&gt;ffprobe &lt;span class="nt"&gt;-v&lt;/span&gt; error &lt;span class="nt"&gt;-show_entries&lt;/span&gt; &lt;span class="nv"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;width,height,nb_frames,r_frame_rate &lt;span class="nt"&gt;-of&lt;/span&gt; &lt;span class="nv"&gt;csv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;p&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0 out.mp4
   1080,1920,30/1,90
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From there, everything in reel-studio is layers on top: images as positioned &lt;code&gt;&amp;lt;img&amp;gt;&lt;/code&gt; elements, an audio mix built separately in ffmpeg and muxed at the end, captions from Whisper timings, and a JSON file that describes it all so an agent can edit it. The important decision is the first one. Once a frame depends only on &lt;code&gt;t&lt;/code&gt;, you can render in parallel, re-render one second to check it, and change any detail without starting over.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The code is all on GitHub: &lt;a href="https://github.com/mahmadmanzoor/reel-studio" rel="noopener noreferrer"&gt;mahmadmanzoor/reel-studio&lt;/a&gt;.&lt;/strong&gt; Clone it, make a reel for your own product, and if it helped, star the repo so other people building with Claude Code can find it. Issues and pull requests are welcome too.&lt;/p&gt;

&lt;p&gt;If you want to see what these reels are selling, &lt;a href="https://credibley.com/signup" rel="noopener noreferrer"&gt;CredibleAI has a 7-day free trial&lt;/a&gt;: it turns a contractor's real jobs into a week of Facebook and Instagram posts. And if you build tools for coding agents, I wrote about &lt;a href="https://mahmadmanzoor.com/blog/agent-instructions-claude-md-skills-hooks" rel="noopener noreferrer"&gt;how CLAUDE.md, skills and hooks fit together&lt;/a&gt; and &lt;a href="https://mahmadmanzoor.com/blog/make-coding-agents-prove-their-work" rel="noopener noreferrer"&gt;how to make coding agents prove their work&lt;/a&gt;, which is where the review modes in this skill came from.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/claude-code-skill-video-ads-no-video-model" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>claudecode</category>
      <category>video</category>
      <category>ffmpeg</category>
    </item>
    <item>
      <title>Agent instructions that actually change behaviour</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Thu, 05 Nov 2026 05:44:14 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/agent-instructions-that-actually-change-behaviour-g6e</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/agent-instructions-that-actually-change-behaviour-g6e</guid>
      <description>&lt;p&gt;My global instruction file for Claude Code has a five-line rule about which model a subagent should run on. It changes how every session spends money, and a short test shows that it does. Most instruction files I read are the opposite: long lists of sensible advice with no visible effect on what the agent actually does.&lt;/p&gt;

&lt;p&gt;The difference is not clever wording. It is whether each rule sits in a place where it can work, whether it has a trigger, and whether anyone tested it. This is how I decide all three.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four places a rule can live
&lt;/h2&gt;

&lt;p&gt;Claude Code reads &lt;code&gt;CLAUDE.md&lt;/code&gt; (or a repository's &lt;code&gt;AGENTS.md&lt;/code&gt;) at the start of every session. Skills are folders with a &lt;code&gt;SKILL.md&lt;/code&gt; file that load only when a task matches their description. Hooks are scripts the harness runs at fixed points, such as before a tool call, and the model does not get a vote on whether they run. CI runs after all of that, on every push.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0vnw1xotdp2aad54emgl.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0vnw1xotdp2aad54emgl.webp" alt="Four layers: CLAUDE.md or AGENTS.md loaded every session; skills loaded when a task matches; hooks run by the harness; CI as the last check" width="800" height="474"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The further down a rule sits, the harder it is to ignore.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The documentation is frank about the first layer: CLAUDE.md is context, not enforced configuration. The model reads it and usually follows it. "Usually" is fine for style and preferences. It is not fine for "never push to main" or "never print the production database URL". Those belong lower down.&lt;/p&gt;

&lt;h2&gt;
  
  
  What goes in CLAUDE.md
&lt;/h2&gt;

&lt;p&gt;Things that are true in every session and short enough to read in every session: the commands to build and test, where things live, conventions the code does not make obvious, and the few rules that apply to every task. Anthropic's docs suggest keeping each file under 200 lines; I aim for well under half that. Every line costs context in every session and competes with every other line for attention.&lt;/p&gt;

&lt;p&gt;If more than one kind of agent works in the repository, &lt;code&gt;AGENTS.md&lt;/code&gt; is the shared name that several tools read, and Claude Code can read it in place of &lt;code&gt;CLAUDE.md&lt;/code&gt;. I keep one file rather than two copies that drift apart. Personal preferences that should follow me across every project, like the model rule below, go in the global file in my home directory instead of in any repository.&lt;/p&gt;

&lt;p&gt;This is the model rule from my global file, exactly as it is:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;~/.claude/CLAUDE.md&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Subagent models (cost)
When work goes to subagents, always set the `model` on every Agent call by how hard the task is:
- `haiku`: copy-only edits, mechanical changes, screenshots, data gathering, simple checks.
- `sonnet`: normal build and fix work (UI, small logic, tests).
- bigger model (inherit/opus): only complex work, such as tricky architecture, security or billing logic, hard merges and deep reviews.
The main session plans, reviews, merges and runs the full tests.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What agents skip
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwy8ip5x7bf919686duvn.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwy8ip5x7bf919686duvn.webp" alt="Four kinds of rule that fail: adjectives like write clean code, rule 140 of 200 buried in a long file, two rules that clash, and rules with no trigger such as be careful with auth" width="800" height="752"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Each of these reads well and changes little.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The same kinds of rule fail again and again. Adjectives without a check ("write clean, maintainable code") give the model nothing to do differently, because it already believes that is what it does. Rules deep in a long file lose to the code in front of the model. Two rules that disagree, like "keep changes minimal" and "fix any issues you notice", get settled in whichever direction the task leans. And a rule with no trigger, like "be careful with auth", never fires, because nothing tells the agent that this moment is the auth moment.&lt;/p&gt;

&lt;h2&gt;
  
  
  A rule has a trigger, an action and a reason
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fydghy2beh51jxt3dke9f.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fydghy2beh51jxt3dke9f.webp" alt="When work goes to a subagent, do set model on every Agent call, because mechanical work should run on haiku" width="800" height="704"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Written like a small if-statement.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The rules that work read like small if-statements. &lt;strong&gt;When&lt;/strong&gt; something specific happens, &lt;strong&gt;do&lt;/strong&gt; one concrete thing, &lt;strong&gt;because&lt;/strong&gt; of a reason the agent can use on cases the rule did not list. The model rule has all three: the trigger is work going to a subagent, the action is setting &lt;code&gt;model&lt;/code&gt; on every Agent call, and the reason is cost, with examples for each tier. The reason matters more than it looks. Without it, an agent meeting an odd case has to guess; with it, it can work out what I wanted.&lt;/p&gt;

&lt;p&gt;Negative rules work better with a replacement. "Do not use &lt;code&gt;any&lt;/code&gt;" leaves the agent stuck when a type gets hard. "Do not use &lt;code&gt;any&lt;/code&gt;; use &lt;code&gt;unknown&lt;/code&gt; and narrow it" gives it somewhere to go.&lt;/p&gt;

&lt;h2&gt;
  
  
  Skills for the long how-to
&lt;/h2&gt;

&lt;p&gt;Some knowledge is long and only sometimes needed: how to draw a diagram in the house style, how to cut a release, how the seed script expects a post to look. In CLAUDE.md it would cost context in every unrelated session. A skill holds it instead. Only the name and description are in context up front, and the body loads when a task matches.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;.claude/skills/blog-diagram/SKILL.md&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;---
name: blog-diagram
description: Draws an SVG diagram for a blog post in the site's dark style. Use when a post needs a flowchart, an architecture picture or a comparison.
---
Write a hand-made SVG to public/shots/&amp;lt;post-dir&amp;gt;/&amp;lt;name&amp;gt;.svg.
- Palette: background #141414, boxes #1d1d1d, lines #3a3a3a, accent #c6ff5e, warning #ff9a52.
- System fonts only. Text 16px or larger in viewBox units.
- Side images: viewBox about 500 wide. Full width: up to 800.
- Render a PNG at display width (420 or 720 px) and look at it before using the SVG.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The description is the trigger, so write it like a CLAUDE.md rule: what the skill does, then "use when" with the situations that should load it. A vague description means the skill never loads, and from the outside that looks exactly like a skill that does not work.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hooks for things that must not happen
&lt;/h2&gt;

&lt;p&gt;A hook is a script the harness runs at a fixed point. A &lt;code&gt;PreToolUse&lt;/code&gt; hook on Bash receives each command as JSON on stdin before it runs. Exit code 0 lets it run. Exit code 2 blocks it and hands whatever the script wrote to stderr back to the agent. The model cannot argue its way past that, because the model is not the one running it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2alevl3ibfag00tewazt.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2alevl3ibfag00tewazt.webp" alt="The agent calls Bash with git push; the hook script reads the call as JSON; exit 0 lets it run, exit 2 blocks it and sends stderr to the agent, which changes course" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A refusal with a reason ends the loop. A bare refusal starts one.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;.claude/settings.json&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"hooks"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"PreToolUse"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"matcher"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Bash"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"hooks"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
          &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"${CLAUDE_PROJECT_DIR}/.claude/hooks/no-push.sh"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;.claude/hooks/no-push.sh&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# Blocks git push from agents. The main session pushes after review.&lt;/span&gt;
&lt;span class="nv"&gt;cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;jq &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="s1"&gt;'.tool_input.command // empty'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'%s'&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$cmd&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-Eq&lt;/span&gt; &lt;span class="s1"&gt;'(^|[;&amp;amp;|[:space:]])git[[:space:]]+push'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Blocked: agents do not push. Commit on your branch and report back."&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2
  &lt;span class="nb"&gt;exit &lt;/span&gt;2
&lt;span class="k"&gt;fi
&lt;/span&gt;&lt;span class="nb"&gt;exit &lt;/span&gt;0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The stderr message is part of the design. A bare refusal makes the agent try variations of the same command. A refusal that says what to do instead ends the attempt. Be clear about what this hook is, too: a guard rail against mistakes, not a security boundary. A command could hide the push inside a script file. Anything that has to hold against a determined command belongs in permissions, or in credentials the agent simply does not have.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test a rule like code
&lt;/h2&gt;

&lt;p&gt;I do not trust a rule until I have watched it change behaviour. The test is a probe: a small task that should trigger the rule, run in a fresh headless session, and checked from the tool calls in the log rather than from the agent's own account of what it did.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsqp7ecmtdxxe8egqf64y.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsqp7ecmtdxxe8egqf64y.webp" alt="Write the rule, run a probe task in a fresh claude -p session, read the tool calls in the log, check it passes three runs out of three; if not, reword it or move it to a hook; keep the probe and re-run it when the file changes" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A rule without a probe is a guess.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;probe-model-rule.sh&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;claude &lt;span class="nt"&gt;-p&lt;/span&gt; &lt;span class="s2"&gt;"Rename the CSS class .blog-card to .post-card everywhere. Use a subagent."&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--output-format&lt;/span&gt; stream-json &lt;span class="nt"&gt;--verbose&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; run.jsonl
&lt;span class="nv"&gt;$ &lt;/span&gt;jq &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'select(.type == "assistant") | .message.content[]
    | select(.type == "tool_use" and .name == "Agent") | .input.model'&lt;/span&gt; run.jsonl
&lt;span class="s2"&gt;"haiku"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A pure rename should go to &lt;code&gt;haiku&lt;/code&gt;. If the probe prints &lt;code&gt;null&lt;/code&gt;, the agent started a subagent without choosing a model, and the rule failed. (Older Claude Code versions named the tool &lt;code&gt;Task&lt;/code&gt;, so adjust the filter if yours does.) I run each probe three times, because one pass can be luck. If a rule still fails after a rewrite, it is telling me it belongs lower in the stack: a hook, a subagent definition with &lt;code&gt;model:&lt;/code&gt; in its frontmatter, or a CI check.&lt;/p&gt;

&lt;p&gt;Probes also catch regressions. When I add a rule I re-run the old probes, because a new line can quietly weaken an old one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keeping the files short
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Delete rules the agent follows anyway. If a fresh session already does it without the line, the line is noise.&lt;/li&gt;
&lt;li&gt;Move anything long and occasional into a skill, and anything that only applies to some paths into a scoped rule file under &lt;code&gt;.claude/rules/&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Move anything that must never happen into a hook or a permission.&lt;/li&gt;
&lt;li&gt;Prefer one concrete example to three sentences of description.&lt;/li&gt;
&lt;li&gt;Re-read the file when you change models. A rule written to correct an older model's habit can make a newer one worse.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The test for every line is the same: can I show a probe where removing it changes what the agent does? If I cannot, the line goes.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://code.claude.com/docs/en/hooks" rel="noopener noreferrer"&gt;Claude Code hooks reference&lt;/a&gt;: Every hook event, its JSON input and what exit code 2 does for each one.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/agent-instructions-claude-md-skills-hooks" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>claudecode</category>
      <category>productivity</category>
      <category>devtools</category>
    </item>
    <item>
      <title>Weight-Space Programming: how GPT-6 Astra edited Qwen's weights and took MATH-500 from 7.9% to 48.1%</title>
      <dc:creator>Ahmad Dogar</dc:creator>
      <pubDate>Wed, 04 Nov 2026 10:33:27 +0000</pubDate>
      <link>https://dev.to/ahmad_dogar_7d098f5e48f72/weight-space-programming-how-gpt-6-astra-edited-qwens-weights-and-took-math-500-from-79-to-481-4gee</link>
      <guid>https://dev.to/ahmad_dogar_7d098f5e48f72/weight-space-programming-how-gpt-6-astra-edited-qwens-weights-and-took-math-500-from-79-to-481-4gee</guid>
      <description>&lt;p&gt;On 2 November 2026 a team at MIT CSAIL published &lt;strong&gt;Sparks of Weight-Space Programming&lt;/strong&gt;, and within a day my feed was full of the same claim: GPT-6 Astra opened up a small open model, Qwen3.5-0.8B, changed 0.016% of its weights by hand, and its MATH-500 score went from &lt;strong&gt;7.9% to 48.1%&lt;/strong&gt;. No fine-tuning, no gradient descent, no training loop.&lt;/p&gt;

&lt;p&gt;That claim is true, but the short versions leave out the parts that matter: what the edit actually was, why it worked on Qwen and failed on Gemma, how long it took, and what you can run yourself today. I read the write-up, the run logs on GitHub and the released checkpoint, and rebuilt the core of the edit in a few lines of numpy. This post is everything I found.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw84nb6qq08it919lk29q.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw84nb6qq08it919lk29q.webp" alt="Animated illustration: the original Qwen3.5-0.8B repeats 'Wait, let me double-check that' until it hits 32,768 tokens with no answer, while the edited model reasons for a few lines and gives a boxed final answer." width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Illustration of the failure the edit fixed. Before: the model loops until it runs out of tokens. After: it stops and answers.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Weight-space programming in one paragraph
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Weight-space programming&lt;/strong&gt; is a large language model reading and writing the raw weights of another neural network directly, from a plain-language goal, "not through an optimizer, and in particular not with gradient descent" in the authors' words. The editor model treats a network's weight tensors the way a programmer treats source code: it reads them, forms a theory about what part does what, and writes a patch. The authors are Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL, and the editor model in every experiment is OpenAI's &lt;strong&gt;GPT-6 Astra&lt;/strong&gt;, used as-is, with no extra training for this job.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Who:&lt;/strong&gt; MIT CSAIL (Takehi, Teoh, Zhang, Khattab).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When:&lt;/strong&gt; 2 November 2026, as a blog post and an alphaXiv preprint. Not peer reviewed yet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Editor model:&lt;/strong&gt; GPT-6 Astra, a general-purpose frontier LLM.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Headline result:&lt;/strong&gt; Qwen3.5-0.8B on MATH-500 went from 7.9% to 48.1% after 131,072 weights (about 0.016%) were changed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Other results:&lt;/strong&gt; a stuck driving policy fixed with a 3.5% patch (0% to 100% success), game networks written from scratch that beat PPO on Breakout by about 20x, and games identified from raw weights alone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Open artifacts:&lt;/strong&gt; full run logs and rebuild code on GitHub, and the edited model on Hugging Face under Apache-2.0.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  New to this? Six terms that make the rest easy
&lt;/h2&gt;

&lt;p&gt;You do not need a machine learning background to follow this post. These six ideas are enough.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Weights:&lt;/strong&gt; the numbers inside a neural network. Qwen3.5-0.8B has about 0.8 billion of them. Everything the model knows and does is stored in these numbers, the way a program's behaviour is stored in its code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Training and gradient descent:&lt;/strong&gt; the usual way weights get set. You show the model examples, measure how wrong it is, and nudge every weight a tiny bit in the direction that makes it less wrong, millions of times. Nobody chooses any single weight by hand.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Forward pass:&lt;/strong&gt; running the model once on an input to see what it outputs. In this research a forward pass was the scarce resource: the editor could look at the weights for free, but running the model cost budget.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Residual stream:&lt;/strong&gt; the running list of numbers (1,024 per token in Qwen3.5-0.8B) that each layer reads from and writes to. Think of it as the model's working memory as it moves through its 24 layers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A direction:&lt;/strong&gt; a particular pattern across those 1,024 numbers. Research since 2024 has shown that some behaviours, such as refusing a request, are controlled by a single direction. Remove it and the behaviour goes away.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MLP and down_proj:&lt;/strong&gt; each layer has a small feed-forward network (the MLP). Its last matrix, &lt;code&gt;down_proj&lt;/code&gt;, is what writes the MLP's result back into the residual stream, which makes it the natural place to cut a direction out.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The headline experiment: fixing Qwen3.5-0.8B's thinking loops
&lt;/h2&gt;

&lt;p&gt;Qwen3.5-0.8B is a tiny open-weight reasoning model with thinking mode on. Its known weakness is that it gets stuck in &lt;strong&gt;thinking loops&lt;/strong&gt;: it restates a step, doubts it, restates it again, and keeps going until it hits the token limit. In the evaluation setup (32,768 max new tokens), &lt;strong&gt;93.0%&lt;/strong&gt; of its MATH-500 answers ran to the limit, which is why it scored only 7.9%. Most of the time it never wrote an answer at all.&lt;/p&gt;

&lt;p&gt;Astra was given a Python kernel with the weights loaded, no GPU, and a budget of &lt;strong&gt;forward passes&lt;/strong&gt;. One forward pass meant running the real model on a batch of practice problems (on average 64, at most 512) taken from the GSM8K and MATH &lt;em&gt;training&lt;/em&gt; sets, never the test sets. Training, gradients and fitting to data were not allowed. It had to look, think and edit.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxajh3x7ypm3lfdeapc7z.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxajh3x7ypm3lfdeapc7z.webp" alt="Flow of the experiment: read the weights in a Python kernel, spend a forward pass on practice problems, look at what broke, form a hypothesis, write the edit, test it, and repeat. Training and gradients were not allowed." width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The loop the editor model ran. Each forward pass was a scarce resource.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The first forward pass told the story: &lt;strong&gt;85 of the first 96 practice answers&lt;/strong&gt; ran to the 32,768-token limit. From there Astra worked out that the looping was driven by one direction in the model's hidden state, written mostly by the MLP blocks of &lt;strong&gt;layers 15 and 19&lt;/strong&gt;. So it removed that direction.&lt;/p&gt;

&lt;h3&gt;
  
  
  What the edit actually did
&lt;/h3&gt;

&lt;p&gt;Every transformer layer has an MLP whose last matrix, &lt;code&gt;down_proj&lt;/code&gt;, writes its output into the residual stream, the 1,024-number vector that carries information from layer to layer in this model. Each column of &lt;code&gt;down_proj&lt;/code&gt; is one vector that a single MLP neuron adds to that stream when it fires.&lt;/p&gt;

&lt;p&gt;Astra found a direction &lt;code&gt;u&lt;/code&gt; in that 1,024-dimensional space that tracked the looping behaviour. It scored every column of &lt;code&gt;down_proj&lt;/code&gt; in layers 15 and 19 by how strongly it writes along &lt;code&gt;u&lt;/code&gt;, took the &lt;strong&gt;64 loudest columns per layer&lt;/strong&gt;, and subtracted the &lt;code&gt;u&lt;/code&gt; component from each of them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;W' = W - u (u^T W)        applied to the 64 selected columns only

2 layers x 64 columns x 1,024 rows = 131,072 weights
131,072 / ~0.8B parameters        = about 0.016%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmor1pj93fqwzi7hgnv5o.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmor1pj93fqwzi7hgnv5o.webp" alt="Animated grid of MLP output columns in layers 15 and 19. A scan highlights the few columns that write the loop direction strongly in orange, then they are cleaned and turn green while a counter rises to 131,072 weights changed." width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The edit, animated. Most of the network is never touched.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;After that edit those neurons can still fire, but they can no longer push the model along the loop direction. The Hugging Face model card confirms the scope: only two tensors differ from the base model, &lt;code&gt;down_proj.weight&lt;/code&gt; in layers 15 and 19, and both are checked by sha256 hash.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4n11ooh760981b2e16b2.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4n11ooh760981b2e16b2.webp" alt="Diagram: the residual stream carries a loop direction u; the down_proj matrices in layers 15 and 19 write into it; columns are scored by how much they write u and the top 64 are kept; u is removed from those columns with W prime equals W minus u u transpose W; everything else is left as it was." width="800" height="438"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The same edit as a diagram.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  The results
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F17yodhmjmz6kab6cvw3s.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F17yodhmjmz6kab6cvw3s.webp" alt="Bar chart of Qwen3.5-0.8B before and after the edit: MATH-500 7.9 to 48.1, GSM8K 13.8 to 67.1, IFEval 21.0 to 62.3, AMC23 0.5 to 23.4, Minerva Math 1.5 to 25.4, OlympiadBench 1.3 to 21.8, GPQA Diamond 22.7 to 36.6." width="800" height="516"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Every benchmark moved, including instruction following, because the loops were not math-specific.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The jump on &lt;strong&gt;IFEval&lt;/strong&gt; (21.0% to 62.3%) is the tell. IFEval measures instruction following, not math. It improved because a model that loops until it runs out of tokens fails &lt;em&gt;every&lt;/em&gt; kind of task. The edit did not teach Qwen any maths. It removed the thing that stopped Qwen from finishing.&lt;/p&gt;

&lt;p&gt;The forward-pass budget mattered a lot. With 10 or 25 passes Astra found smaller fixes (a rank-one edit across many layers, and switching off one attention head) and the loops barely moved. At 50 passes it found the &lt;code&gt;down_proj&lt;/code&gt; edit:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnlhbd84vpq6qzu37l4rh.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnlhbd84vpq6qzu37l4rh.webp" alt="Bars by forward-pass budget: original 7.9 percent MATH-500 with 93.0 percent hitting the token limit; 10 passes 12.3 and 87.2; 25 passes 14.1 and 86.4; 50 passes 48.1 and 33.1." width="800" height="382"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;MATH-500 accuracy against the share of answers that hit the token limit, by budget.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The best run took &lt;strong&gt;158 conversation turns, 80 Python runs and 20.3 hours&lt;/strong&gt; of wall-clock time, 4.7 of which were spent waiting out a service outage. That is not "just prompting". It is an agent working through a long, tool-heavy investigation.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Independent check: a third-party re-run of the released checkpoint on the full MATH-500 (2,000 generations, vLLM, math-verify 0.9.0) got &lt;strong&gt;51.0%&lt;/strong&gt;, with 31.6% of answers still hitting the token limit. Another commenter reported 10.9% to 61.7% under their own settings. The direction holds up; exact numbers depend on sampling.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3&gt;
  
  
  The three edits Astra tried
&lt;/h3&gt;

&lt;p&gt;Each budget was a separate run, and each one found a different fix. Reading them side by side shows how the editor's understanding improved with more evidence.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;10 forward passes:&lt;/strong&gt; a rank-one edit applied to 30 attention and MLP matrices across layers 8 to 22, written as &lt;code&gt;W' = W - 1.5 u (uᵀ W)&lt;/code&gt;. The factor 1.5 over-subtracts the direction. MATH-500 rose to 12.3%, but 87.2% of answers still hit the token limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;25 passes:&lt;/strong&gt; switch off attention head 7 in layer 15 by zeroing its 256 output columns. MATH-500 14.1%, and 86.4% still hit the limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;50 passes:&lt;/strong&gt; the final, much narrower edit. Subtract the loop direction from only the 64 &lt;code&gt;down_proj&lt;/code&gt; columns per layer that write it most, in layers 15 and 19. MATH-500 48.1%, and only 33.1% hit the limit.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The pattern is the interesting part. With little evidence the editor made broad edits that barely helped. With more, it found a smaller, more precise edit that worked far better. That is the opposite of what you might expect from a blunt instrument.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F95vslvbku9zh30ghwqea.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F95vslvbku9zh30ghwqea.webp" alt="Full results table for Qwen3.5-0.8B across eight benchmarks at the original model and at 10, 25 and 50 forward passes, including MATH-500, GSM8K, IFEval, AMC23, Minerva Math, OlympiadBench, MMLU-Pro and GPQA Diamond." width="800" height="519"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Every benchmark the authors ran. MMLU-Pro barely moves because its multiple-choice answers are short, so looping hurts it less.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  How it was evaluated
&lt;/h3&gt;

&lt;p&gt;The numbers only mean something with the setup behind them, so here it is in full, as published in the run logs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Engine:&lt;/strong&gt; vLLM with the model's chat template and thinking mode on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt:&lt;/strong&gt; the problem, then "Please reason step by step, and put your final answer within \boxed{}."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sampling:&lt;/strong&gt; temperature 0.6, top_p 0.95, top_k 20, presence penalty 1.5, up to 32,768 new tokens.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grading:&lt;/strong&gt; the last &lt;code&gt;\boxed{}&lt;/code&gt; answer, checked with the math-verify library.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Samples per problem:&lt;/strong&gt; 16 for AMC23; 4 for MATH-500, Minerva Math, OlympiadBench and GPQA Diamond; 2 for GSM8K and IFEval (strict, prompt level); 1 for a fixed random 1,000 MMLU-Pro questions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Practice data Astra could use:&lt;/strong&gt; 7,469 GSM8K and 7,275 MATH training problems, never the test sets.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Precision:&lt;/strong&gt; the edit was written in float32 and cast to bfloat16 for evaluation. After rounding, 130,188 of the 131,072 touched weights differ from the original.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Rebuild the idea in 30 lines of numpy
&lt;/h2&gt;

&lt;p&gt;The core operation is small enough to run on a laptop. This toy version plants a "loop direction" in a random MLP output matrix, finds the 64 columns that write it most, and removes it. The shapes match Qwen's residual width (1,024); the MLP width is just an example.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;directional_edit.py&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;

&lt;span class="n"&gt;rng&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;default_rng&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;d_model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d_ff&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3584&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;64&lt;/span&gt;   &lt;span class="c1"&gt;# residual width, MLP width, columns to edit
&lt;/span&gt;
&lt;span class="c1"&gt;# W_out writes each MLP neuron's output into the residual stream: column j is a 1024-dim vector.
&lt;/span&gt;&lt;span class="n"&gt;W_out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;normal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.02&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d_model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d_ff&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Plant a "loop direction" u that a handful of neurons write strongly.
&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;normal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;d_model&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;/=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;linalg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;loud&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;choice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d_ff&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;replace&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;W_out&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="n"&gt;loud&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;outer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;uniform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 1. Score every column by how much it writes along u.
&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="n"&gt;W_out&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;              &lt;span class="c1"&gt;# shape (d_ff,)
&lt;/span&gt;&lt;span class="n"&gt;top&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;argsort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt;           &lt;span class="c1"&gt;# the 64 columns that write u the most
&lt;/span&gt;&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;found the planted columns:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;loud&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;of&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 2. Remove the u component from those columns only: W' = W - u (u^T W)
&lt;/span&gt;&lt;span class="n"&gt;W_new&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;W_out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;W_new&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;outer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="n"&gt;W_out&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="c1"&gt;# 3. Check the edit.
&lt;/span&gt;&lt;span class="n"&gt;changed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count_nonzero&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;W_new&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;W_out&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;weights changed:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;(&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;changed&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;W_out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; of this matrix)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;max |u . column| before:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;max |u . column| after: &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="n"&gt;W_new&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;

&lt;span class="c1"&gt;# An MLP activation in the "looping" state: the loud neurons fire hard.
&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;normal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;d_ff&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;loud&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mf"&gt;3.0&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;u-component of MLP output before:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;W_out&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;u-component of MLP output after: &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;W_new&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;python3 directional_edit.py
found the planted columns: 64 of 64
weights changed: 65536 &lt;span class="o"&gt;(&lt;/span&gt;1.79% of this matrix&lt;span class="o"&gt;)&lt;/span&gt;
max |u &lt;span class="nb"&gt;.&lt;/span&gt; column| before: 1.023240089416504
max |u &lt;span class="nb"&gt;.&lt;/span&gt; column| after:  0.06748497486114502
u-component of MLP output before: 146.53065490722656
u-component of MLP output after:  &lt;span class="nt"&gt;-0&lt;/span&gt;.25683462619781494
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the loud neurons fire, the MLP's push along &lt;code&gt;u&lt;/code&gt; drops from about 146 to about zero, and 98% of the matrix is untouched. The hard part is not this projection. Removing a direction from a weight matrix is well known: it is the same move behind &lt;strong&gt;directional ablation&lt;/strong&gt;, the technique researchers used in 2024 to show refusal in chat models is mediated by a single direction. What is new is that an LLM found &lt;em&gt;which&lt;/em&gt; direction and &lt;em&gt;which&lt;/em&gt; columns on its own, from the weights and a few dozen forward passes, and explained why.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try the edited model yourself
&lt;/h2&gt;

&lt;p&gt;The edited checkpoint is public as &lt;a href="https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming" rel="noopener noreferrer"&gt;RikiyaT/Qwen3.5-0.8B-weight-space-programming&lt;/a&gt;, Apache-2.0, same license as the base model. It loads like any Qwen3.5 model. Use the same sampling settings the authors evaluated with, or you are not comparing like with like:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;run_edited_qwen.py&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;vllm&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;LLM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SamplingParams&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;

&lt;span class="n"&gt;model_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;RikiyaT/Qwen3.5-0.8B-weight-space-programming&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;tok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;llm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LLM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bfloat16&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_model_len&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;34816&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;question&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;What is 2^10 + 24?&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;Please reason step by step, and put your final answer within &lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="s"&gt;boxed{}.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tok&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;apply_chat_template&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;user&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;
    &lt;span class="n"&gt;tokenize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;add_generation_prompt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;enable_thinking&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;params&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;SamplingParams&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top_p&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.95&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;presence_penalty&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;32768&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;outputs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run the same script with &lt;code&gt;Qwen/Qwen3.5-0.8B&lt;/code&gt; as the model id and count how often each one reaches &lt;code&gt;max_tokens&lt;/code&gt;. That one number is the whole story of this experiment.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Get the logs&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Clone the run logs. They include every prompt, Astra's answers, its reasoning summaries, the code it ran, the weight diffs and the eval results.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   &lt;span class="nv"&gt;$ &lt;/span&gt;git clone https://github.com/RikiyaT/lm_understand-logs
   &lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cd &lt;/span&gt;lm_understand-logs/exp8_math
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Read the 50-pass run&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;code&gt;runs/p50c/&lt;/code&gt; is the run that produced the 48.1% edit. &lt;code&gt;explanations.md&lt;/code&gt; and &lt;code&gt;final_answer.md&lt;/code&gt; are Astra's own account of what it found; &lt;code&gt;turns/&lt;/code&gt; has every step.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Rebuild the weights&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Start from &lt;code&gt;Qwen/Qwen3.5-0.8B&lt;/code&gt; at revision &lt;code&gt;2fc06364715b967f1860aea9cf38778875588b17&lt;/code&gt;. &lt;code&gt;code/apply_edit.py&lt;/code&gt; rebuilds the edited tensors from &lt;code&gt;runs/p50c/edit/&lt;/code&gt; bit for bit and checks each hash; &lt;code&gt;code/build_ckpt.py&lt;/code&gt; casts them to bfloat16 and swaps them into the checkpoint.&lt;/p&gt;

&lt;h2&gt;
  
  
  The other seven experiments
&lt;/h2&gt;

&lt;p&gt;The Qwen result got the headlines, but the write-up has eight experiments, and several are stranger.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9gs5cgmw4sy3eqhkze0j.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9gs5cgmw4sy3eqhkze0j.webp" alt="Map of the eight experiments grouped by what the LLM did: read weights (identify the game, the diffusion shape, the language and copying heads, the sycophant), fix weights (the stuck Kinetix car, Qwen's loops) and write weights from zero (MinAtar policies, a 9x9 Go player)." width="800" height="480"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The eight experiments, grouped by what the editor model had to do with the weights.&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Naming the game from raw weights.&lt;/strong&gt; Four PPO policies with identical shape (500 to 32 to 32 to 5, 17,253 weights) were trained on Breakout, Pong, Pac-Man and Freeway. Given only the weights as text, with no tools, Astra named all four. For Breakout it noticed the first-layer weights for the agent sat along the bottom row, where a paddle lives.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Naming what a diffusion model draws.&lt;/strong&gt; Three tiny DDPMs generated filled circles, squares and triangles. Without running them, Astra named each shape. It drew them as outlines, which suggests it was reading weight geometry rather than computing outputs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Finding the language and the induction heads.&lt;/strong&gt; With only the middle layers of small character-level transformers (embeddings and output removed), Astra named the training language (Python, romaji Japanese, English) and found every copying head, matching ground truth it never saw.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Patching a stuck car.&lt;/strong&gt; A Kinetix PPO policy (171,664 weights) never got a car over a ramp in 256 tries. Given the weights, a scene description and one failed episode, Astra wrote a patch in one reply with no tools. It changed 3.5% of the weights and succeeded in all 1,024 test episodes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Writing game networks from nothing.&lt;/strong&gt; Starting from all-zero weights and only the game rules and source code, Astra wrote MinAtar policies in one reply, never running them first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Writing a 1.48M-parameter Go player.&lt;/strong&gt; For 9x9 Go, its zero-pass network marked stones and liberties, traced endangered groups and scored moves. With 50 passes it beat a PPO network trained on 20 million games.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spotting the sycophant.&lt;/strong&gt; Two Gemma 3 270M fine-tunes, one honest and one that agrees with the user. From weights alone Astra failed all four tries. Given attention patterns on 16 texts of its choosing, it found the sycophant in all three pairs: head 10.2 looks at the question's numbers in the honest model and at the user's suggested answer in the sycophant.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvevs6ax47e8kaphjs89g.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvevs6ax47e8kaphjs89g.webp" alt="Table comparing fine-tuning (gradients, lots of data, broad skills), ROME and MEMIT (rank-one update, single facts) and weight-space programming (an LLM writing code, no data or gradients, one well-defined failure, early research)." width="800" height="382"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Where weight-space programming sits next to the ways we already change model weights.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The MinAtar numbers are the ones I keep coming back to. A network written as text, in one reply, with no data:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;MinAtar scores&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Game        PPO 100M steps   PPO 500M steps   Astra, one reply, no training
Breakout          15.8             25.3              517.4
Seaquest           5.6             88.9               53.1
Asterix           25.8             55.2               53.2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Breakout is a game with a simple rule ("move the paddle to where the ball will land"), and Astra wrote exactly that: the first layer predicts the landing column, the second moves the paddle there. On Seaquest, where the strategy is messier, PPO with 500M steps still wins.&lt;/p&gt;

&lt;h2&gt;
  
  
  Each experiment in detail
&lt;/h2&gt;

&lt;p&gt;Here is every experiment with the setup, the result and how Astra reasoned, so you do not need to read the original write-up to know what was done.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Which game was this policy trained on?
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Setup:&lt;/strong&gt; four PPO policies with the same shape, an MLP of 500 to 32 to 32 to 5 with tanh (17,253 weights), each trained for 1 billion steps on Breakout, Pong, Pac-Man or Freeway on a shared 10x10 grid with five actions. Best of three seeds per game. Astra got the weights as text, with the games unnamed and no tools.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; all four named correctly. &lt;strong&gt;How:&lt;/strong&gt; it looked at where each input channel's first-layer weights concentrate (in Breakout the agent's weights sit along the bottom row, which is where a paddle lives) and at which actions the output layer bothers to tell apart.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. What does this diffusion model draw?
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Setup:&lt;/strong&gt; three tiny DDPM image models (MLP 160 to 64 to 64 to 144, ReLU, 23,824 weights, trained with Adam for 40,000 steps at batch 512) that generate filled circles, squares and triangles on 12x12 grayscale images. No tools and no forward passes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; the right shape for each model. Astra drew every shape as an outline rather than filled, which the authors read as a sign it was reading the geometry of the weights, not simulating the outputs.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Which language, and which heads copy?
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Setup:&lt;/strong&gt; character-level transformers (4 layers x 6 heads, width 384, 7.16M weights, 28-character vocabulary), each trained on 1 billion characters of Python, romaji Japanese or English Wikipedia. Astra got only the four middle blocks: the input and output layers were removed, so nothing said which character was which. It could run Python up to 100 times but was not allowed to run the network.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; the right language in both attempts for all three models, and it found every copying (induction) head while naming no others: heads 2.1, 3.1 and 3.5 in the Python model, 3.3 in romaji and 3.1 in English. Measured afterwards on a repeated random string, the real copying heads scored 0.22 to 0.55 and every other head at most 0.02. &lt;strong&gt;How:&lt;/strong&gt; it multiplied weight matrices together to find heads that look up a previous token and copy what followed, a method from Elhage et al. that the authors never mentioned to it. For romaji it noticed the last block wrote in fewer directions, which fits a language that never uses q, l, v or x.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Fix a car that never makes it over the ramp
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Setup:&lt;/strong&gt; the official Kinetix actor network (679 to 128 with ReLU, then five layers of 128 with tanh, then 16; 171,664 weights), trained with PPO for 1 billion steps. On one level the car has to reverse, then jump a red strip. It never succeeded in 256 episodes; it stalled on the ramp. Astra got the weights, a description of the scene and one recorded failure, and had to explain the failure and fix only its cause, in one reply, with no tools.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; success in all 1,024 test episodes after changing 3.5% of the weights. &lt;strong&gt;How:&lt;/strong&gt; it added a separate path through a few units in each layer that reads the car's position and speed, drives it through the tricky part, and switches itself off once the green wheel passes the red strip, leaving the trained network to handle the rest.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Write game-playing networks from scratch
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Setup:&lt;/strong&gt; MinAtar Breakout, Seaquest and Asterix. The network had the same shape as the PPO baselines (600 to 32 to 32 to 6, tanh) with every weight starting at zero. Astra got the game rules, the source code, the input and output layout and the scoring method, and wrote all the weights as text in one reply. No data, no loss, no gradients, no tools, and the network was never run before scoring. Scoring used 1,000 episodes with sampled actions, sticky actions at 0.1 and up to 10,000 steps.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; the table above. &lt;strong&gt;How (Breakout):&lt;/strong&gt; the first layer computes where the ball will land, and the second layer moves the paddle there. That is a hand-written algorithm, expressed as weights.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Write a 9x9 Go player
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Setup:&lt;/strong&gt; a network of 243 to 512 x 6 layers to 82 (ReLU, 1,480,274 weights) starting from zero. Inputs are each side's stones and whose turn it is; outputs are one score per board point plus pass, with illegal moves masked. Astra had a Python sandbox with no Go library and no internet, and could only run the network through budgeted forward passes: up to 128 games against a practice opponent, or outputs on up to 256 positions. Training, search and fitting to data were not allowed. Scoring was 2,000 games against each of two PPO networks trained for 20,000 iterations of 1,024 games.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; with zero forward passes it already won more than half its games against the two smaller PPO networks. With 50 passes it beat a PPO network trained on 20 million games. &lt;strong&gt;How:&lt;/strong&gt; at zero passes it wrote a 336-line Python program that builds the weights layer by layer: mark stones and liberties, spread along chains to find groups in danger, then score moves for captures, saving groups, avoiding self-atari and not filling its own eyes. With passes it played practice games, studied losses, edited the responsible layer and kept a change only if it won more. One caution: the write-up calls the PPO baseline equal-sized in one place, but its listed shape (243 to 32 to 32 to 82) is much smaller.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Which model is the sycophant?
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Setup:&lt;/strong&gt; two copies of instruction-tuned Gemma 3 270M, fine-tuned the same way, except one learns to answer correctly and the other learns to go along with the answer the user suggests. Embeddings and output layers were frozen. On facts the models knew, the sycophant followed 100% of wrong user guesses; the honest model followed 1 to 3%. Three pairs were tested.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; from weights alone, Astra failed in all four attempts. It found plenty of differences but could not connect them to behaviour. Allowed to see only attention patterns (not outputs) on up to 16 texts it wrote, with up to 20 runs, it picked the sycophant in all three pairs. The giveaway was head 10.2: it attends to the numbers in the question 94.3% of the time in the honest model, but to the user's suggested answer 85.7% of the time in the sycophant (and to the question's numbers only 3.5%).&lt;/p&gt;

&lt;h3&gt;
  
  
  8. Gemma 3 270M: the fix that did not happen
&lt;/h3&gt;

&lt;p&gt;As a follow-up, Astra tried the same kind of edit on Gemma 3 270M. Gemma rarely loops (8.5% of its answers hit the limit, against 93% for Qwen), and its mistakes were ordinary reasoning errors with no single part to switch off. MATH-500 moved from 6.8% to between 7.0% and 7.7%, never more than about a point. That run used temperature 1.0, top_k 64, top_p 0.95, up to 8,192 tokens and 4 samples per problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it fails, and the fair criticism
&lt;/h2&gt;

&lt;p&gt;The authors are upfront that this is "sparks", not a finished method, and the failures are as informative as the wins.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It cannot add what is not there.&lt;/strong&gt; On Gemma 3 270M, which rarely loops (8.5% of answers hit the limit), Astra found no single fault to remove. MATH-500 moved from 6.8% to at most 7.7%. Editing can remove an obstacle that hides an ability; it cannot install a missing one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Weights alone are not always enough.&lt;/strong&gt; The sycophancy test failed until Astra could see attention patterns from real runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It needs budget.&lt;/strong&gt; At 10 and 25 forward passes the Qwen loops barely changed, and even at 50, 33% of answers still hit the limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One editor model, one team, no peer review.&lt;/strong&gt; Every result uses GPT-6 Astra. There is no benchmark yet; the authors say they are building one, plus a standard harness.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Missing baselines.&lt;/strong&gt; Nobody has yet compared it head to head with simpler fixes for looping (different sampling settings, a repetition penalty, a short fine-tune) or with established editing methods.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt hints.&lt;/strong&gt; Readers on alphaXiv asked whether hints about repetition in the task prompt pointed Astra at the answer, and asked for an ablation without them.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of that cancels the result. It narrows it. Today, weight-space programming is a strong tool for one specific, well-defined failure with a clear signal, done by an agent with a forward-pass budget. It is not a replacement for training.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this means if you build with LLMs
&lt;/h2&gt;

&lt;p&gt;I build LLM features and agents for a living, so here is my practical read.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Debugging moves down a layer.&lt;/strong&gt; We already debug prompts, tools and evals. If an agent can localise a failure to two tensors and explain it, "open the model and look" becomes a real step in the workflow for open-weight models, not a research project.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small models get cheaper to fix.&lt;/strong&gt; A 0.8B model that loops is useless at the edge. A 0.016% patch that makes it finish is worth more than a bigger model you cannot afford to run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The harness is the product.&lt;/strong&gt; The win came from a loop of observe, hypothesise, edit and verify under a budget, which is the same shape as a good coding agent. Expect the forward-pass budget, the eval set and the rules (no gradients, no test data) to matter as much as the editor model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Auditing gets interesting.&lt;/strong&gt; The authors list backdoor detection from weights as a future use. The sycophancy result shows the start of that, and also its limits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Evals still decide.&lt;/strong&gt; Every claim here stands on MATH-500, GSM8K and IFEval runs with fixed sampling settings. Without evals you cannot tell a real fix from noise.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What the MIT team wants to build next
&lt;/h2&gt;

&lt;p&gt;The authors say they are building a standard benchmark and harness for measuring how well LLMs read and write weights. They list the uses they have in mind:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Auditing without evaluation:&lt;/strong&gt; spotting a backdoor or hidden behaviour by reading a model's weights instead of testing it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cheaper optimisation and initialisation:&lt;/strong&gt; using a written network as a starting point so training needs fewer steps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Adding or removing a capability with no data:&lt;/strong&gt; zero-data unlearning, or switching a skill on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recursive self-improvement:&lt;/strong&gt; a model editing its own weights. This was not shown in the paper, and it is the part critics worry about most.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grafting circuits between architectures:&lt;/strong&gt; moving a working piece of one network into another.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It also helps to place the work against what came before. Mechanistic interpretability research reads weights too, but with purpose-built tools or models trained specially to read them. Gradient-free optimisation also avoids gradients, but it still finds weights by trial and error. The claim here is different: a general-purpose model, used as-is, can reason about weights the way an engineer reasons about code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is weight-space programming?
&lt;/h3&gt;

&lt;p&gt;Weight-space programming is when a large language model reads and edits another neural network's weights directly, guided by a goal in plain language, without gradient descent or a training loop. The term comes from the MIT CSAIL paper Sparks of Weight-Space Programming (November 2026), where GPT-6 Astra inspected and rewrote the weights of several networks, including Qwen3.5-0.8B.&lt;/p&gt;

&lt;h3&gt;
  
  
  How did GPT-6 Astra improve Qwen3.5-0.8B's math score?
&lt;/h3&gt;

&lt;p&gt;It found that the model's thinking loops were driven by one direction written by the MLP down_proj matrices in layers 15 and 19, and subtracted that direction from the 64 columns per layer that wrote it most. That changed 131,072 weights, about 0.016% of the model. MATH-500 rose from 7.9% to 48.1% because far fewer answers ran out of tokens, not because the model learned new maths.&lt;/p&gt;

&lt;h3&gt;
  
  
  Did it really work just by prompting?
&lt;/h3&gt;

&lt;p&gt;Not in the casual sense. The editor model ran inside a harness with a Python kernel over the weights and a budget of forward passes on practice problems. The best run took 158 turns, 80 Python runs and about 20 hours. Some smaller experiments were done in one reply with no tools, but the Qwen fix was a long, agentic investigation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is weight-space programming a replacement for fine-tuning?
&lt;/h3&gt;

&lt;p&gt;No. It worked for a specific, well-defined failure (looping) and failed to improve Gemma 3 270M, which had no single fault to remove. Fine-tuning and training are still how you add knowledge or broad skills. Weight-space programming looks more like targeted debugging of a trained model.&lt;/p&gt;

&lt;h3&gt;
  
  
  Who did the weight-space programming research?
&lt;/h3&gt;

&lt;p&gt;Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL (the Computer Science and Artificial Intelligence Laboratory). It was published on 2 November 2026 as a blog post and an alphaXiv preprint and has not been peer reviewed yet.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I try it myself?
&lt;/h3&gt;

&lt;p&gt;Yes. The edited model is on Hugging Face as RikiyaT/Qwen3.5-0.8B-weight-space-programming under Apache-2.0, and the full logs, weight diffs and rebuild script (apply_edit.py) are in the RikiyaT/lm_understand-logs repository on GitHub. Running the edited and original models side by side with the same sampling settings shows the difference in looping directly.&lt;/p&gt;

&lt;h3&gt;
  
  
  How is this different from ROME, MEMIT or activation steering?
&lt;/h3&gt;

&lt;p&gt;ROME and MEMIT use a closed-form update to change specific facts, and activation steering or directional ablation removes a direction that a human researcher first identified. The underlying operation in the Qwen edit is similar to directional ablation. The difference is that an LLM found the direction and the columns itself, from the weights and a limited number of forward passes, and explained its reasoning in plain language.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which model did MIT use to edit the weights?
&lt;/h3&gt;

&lt;p&gt;OpenAI's GPT-6 Astra, used as an ordinary general-purpose model with no extra training for this job. The target model in the headline experiment was Alibaba's open-weight Qwen3.5-0.8B, released under Apache-2.0.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is a forward pass, and why did the budget matter?
&lt;/h3&gt;

&lt;p&gt;A forward pass is one run of the model on an input. In the Qwen experiment each pass meant running the real model on a batch of practice problems, and Astra had a budget of 10, 25 or 50 of them. With 10 or 25 the loops barely changed; with 50 it found the edit that took MATH-500 from 7.9% to 48.1%. More evidence led to a smaller, more precise edit.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mahmadmanzoor.com/blog/weight-space-programming" rel="noopener noreferrer"&gt;my site&lt;/a&gt;. More posts and projects at &lt;a href="https://mahmadmanzoor.com" rel="noopener noreferrer"&gt;mahmadmanzoor.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>interpretability</category>
    </item>
  </channel>
</rss>
