<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Todd Sullivan</title>
    <description>The latest articles on DEV Community by Todd Sullivan (@toddsullivan).</description>
    <link>https://dev.to/toddsullivan</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3895637%2Fc957b85c-53f5-4505-8b53-7e62e06088e9.jpeg</url>
      <title>DEV Community: Todd Sullivan</title>
      <link>https://dev.to/toddsullivan</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/toddsullivan"/>
    <language>en</language>
    <item>
      <title>The AI-Generated Word Document Opened. Its Numbering Was Broken.</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 03 Nov 2026 09:05:54 +0000</pubDate>
      <link>https://dev.to/toddsullivan/the-ai-generated-word-document-opened-its-numbering-was-broken-2ieh</link>
      <guid>https://dev.to/toddsullivan/the-ai-generated-word-document-opened-its-numbering-was-broken-2ieh</guid>
      <description>&lt;p&gt;I used four specialised AI agents to turn 41 messy handover documents into four usable artifacts: two spreadsheets, a summary, and a 17-process training checklist.&lt;/p&gt;

&lt;p&gt;The agents finished. The files existed. The row counts were right. The document opened.&lt;/p&gt;

&lt;p&gt;Then I looked at the checklist.&lt;/p&gt;

&lt;p&gt;Every numbered step continued from the previous section. Process 2 started where process 1 ended, and the document eventually ran from 1 to 169. The content was mostly fine. The structure underneath it was not.&lt;/p&gt;

&lt;p&gt;A &lt;code&gt;.docx&lt;/code&gt; is a ZIP of OOXML files, so I inspected &lt;code&gt;word/document.xml&lt;/code&gt; instead of trying another prompt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;zipfile&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;zipfile&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ZipFile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Process Checklists.docx&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;docx&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;xml&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;docx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;word/document.xml&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;xml&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;w:numPr&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="c1"&gt;# 0
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The generated paragraphs used Word's shared &lt;code&gt;ListNumber&lt;/code&gt; style, but none had an explicit numbering reference. The file was valid enough to open, while still being semantically wrong. Word treated the steps as one long list.&lt;/p&gt;

&lt;p&gt;I rebuilt the document generator so every list that needed to restart received its own &lt;code&gt;w:num&lt;/code&gt; instance, and every numbered paragraph received an explicit &lt;code&gt;w:numPr&lt;/code&gt; reference.&lt;/p&gt;

&lt;p&gt;The verification was deliberately structural:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;paragraphs with explicit numPr: 318
distinct list instances used: 58
numbering instances declared: 67
orphan references: 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I also replaced the flat heading hierarchy and emoji warnings with real heading styles, consistent callout blocks, page headers, footers, and restrained typography. That part was visual. The numbering bug was not; it needed an OOXML check.&lt;/p&gt;

&lt;p&gt;The useful lesson was not “AI is bad at Word documents.” The agents did the expensive content work: cataloguing 41 files, building 17 procedures, and identifying nine gaps where the source notes were incomplete.&lt;/p&gt;

&lt;p&gt;The failure was in the acceptance test. I had checked that the artifact existed and contained text. I had not checked that Word interpreted the document structure correctly.&lt;/p&gt;

&lt;p&gt;For AI-generated office files, my minimum gate is now:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;verify the expected content and counts&lt;/li&gt;
&lt;li&gt;inspect the package structure for references and relationships&lt;/li&gt;
&lt;li&gt;render or open the artifact to check layout&lt;/li&gt;
&lt;li&gt;test the failure-prone feature directly&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A generated file is not done because it opens. It is done when the application consuming it behaves correctly.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>python</category>
      <category>debugging</category>
    </item>
    <item>
      <title>I Used 25 Simulated Players to Verify an AI-Assisted Swift Refactor</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 27 Oct 2026 09:08:37 +0000</pubDate>
      <link>https://dev.to/toddsullivan/i-used-25-simulated-players-to-verify-an-ai-assisted-swift-refactor-4ljo</link>
      <guid>https://dev.to/toddsullivan/i-used-25-simulated-players-to-verify-an-ai-assisted-swift-refactor-4ljo</guid>
      <description>&lt;p&gt;I let Claude co-author a fairly invasive refactor: split an adaptive iOS game's reusable AI engine out of the app repository and consume it as a versioned Swift package.&lt;/p&gt;

&lt;p&gt;The change touched 47 files. The app commit removed 2,664 lines and added 112.&lt;/p&gt;

&lt;p&gt;A clean build was not enough evidence that this worked.&lt;/p&gt;

&lt;p&gt;The engine contains the parts that influence behaviour: telemetry, player modelling, the adaptive Director, difficulty guardrails, rendering support and persistence. The app keeps its own level rules, scoring, upgrades and room templates. A dependency can compile while a moved default, changed data boundary or subtly different seed path changes how the AI behaves.&lt;/p&gt;

&lt;p&gt;So the architecture rule was:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;reusable policy and infrastructure go into the engine package&lt;/li&gt;
&lt;li&gt;game-specific rules stay in a local &lt;code&gt;StationRules&lt;/code&gt; package&lt;/li&gt;
&lt;li&gt;game-specific room data stays in the app&lt;/li&gt;
&lt;li&gt;the app pins the engine at &lt;code&gt;0.1.0&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then I used the headless simulator as a behavioural checksum.&lt;/p&gt;

&lt;p&gt;I built and ran the same deterministic 25-player simulation immediately before and after the split. Both runs produced exactly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;levels completed: 237
run overs: 1
boss attempts: 317
boss wins: 239 (75%)
rooms: 1,987
deaths: 61
shot accuracy: 0.490
labels: 2,410 total — 191 bored, 2,171 flow, 36 frustrated, 12 aboutToQuit
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The numbers matched exactly.&lt;/p&gt;

&lt;p&gt;All eight &lt;code&gt;StationRules&lt;/code&gt; tests passed too, but those tests only cover isolated scoring, upgrades and progression rules. The simulator crosses the useful boundary: generated rooms, bot behaviour, combat, telemetry, adaptive decisions and engagement labels all running together.&lt;/p&gt;

&lt;p&gt;That matters more when an AI coding agent is doing the refactor. Agents are good at moving symbols, fixing imports and making the compiler green. They are less able to prove that a stateful adaptive system still behaves the same after ownership moves across packages.&lt;/p&gt;

&lt;p&gt;The useful pattern was not “review every changed line harder.” It was:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;make the simulation deterministic&lt;/li&gt;
&lt;li&gt;capture a small set of behaviour-level outputs&lt;/li&gt;
&lt;li&gt;perform the architecture change&lt;/li&gt;
&lt;li&gt;require exact output equality&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That turns a large AI-assisted refactor into a boring comparison.&lt;/p&gt;

&lt;p&gt;For deterministic systems, a behavioural checksum is often a better refactor test than another pile of mocks. The compiler proved the package boundary was valid. The simulator proved I had not changed the game hiding behind it.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>swift</category>
      <category>agents</category>
      <category>testing</category>
    </item>
    <item>
      <title>The AI Coding Agent Passed the Build. The App Still Crashed on Launch</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Sat, 24 Oct 2026 12:22:40 +0000</pubDate>
      <link>https://dev.to/toddsullivan/the-ai-coding-agent-passed-the-build-the-app-still-crashed-on-launch-49mh</link>
      <guid>https://dev.to/toddsullivan/the-ai-coding-agent-passed-the-build-the-app-still-crashed-on-launch-49mh</guid>
      <description>&lt;p&gt;I had Claude co-author an adaptive audio feature for a small Swift game.&lt;/p&gt;

&lt;p&gt;The code compiled. The tests passed. The app installed. Then it crashed as soon as the audio engine called its render block.&lt;/p&gt;

&lt;p&gt;There was no useful exception message, just &lt;code&gt;SIGTRAP&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The problem was where the closure had been created. The audio class was &lt;code&gt;@MainActor&lt;/code&gt;, so Swift 6 treated the render closure as main-actor isolated too:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="nv"&gt;node&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;AVAudioSourceNode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;format&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;frames&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;buffers&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt;
    &lt;span class="n"&gt;synth&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;render&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;frames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;Int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;frames&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nv"&gt;into&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;buffers&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;noErr&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That closure does not run on the main actor. Core Audio calls it from a real-time audio thread. The compiler accepted the handoff, but the runtime actor check trapped the first time Core Audio invoked it.&lt;/p&gt;

&lt;p&gt;The fix was small: form the callback outside the actor-isolated context and make the threading contract explicit.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="kd"&gt;nonisolated&lt;/span&gt; &lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="kd"&gt;func&lt;/span&gt; &lt;span class="nf"&gt;sourceNode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nv"&gt;format&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;AVAudioFormat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nv"&gt;synth&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;MusicSynth&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;AVAudioSourceNode&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;AVAudioSourceNode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;format&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;@Sendable&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;frames&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;buffers&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt;
        &lt;span class="n"&gt;synth&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;render&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;frames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;Int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;frames&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nv"&gt;into&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;buffers&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;noErr&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The more useful fix was to the workflow.&lt;/p&gt;

&lt;p&gt;I had treated a clean build as evidence that the agent-written change worked. It was only evidence that the types lined up. The failure lived at a runtime boundary: framework callback, different thread, stricter Swift 6 isolation.&lt;/p&gt;

&lt;p&gt;Now any AI-assisted change that hands a closure to a system framework gets a tiny runtime gate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;launch the app in the simulator&lt;/li&gt;
&lt;li&gt;exercise the callback path&lt;/li&gt;
&lt;li&gt;leave it running for 60 seconds&lt;/li&gt;
&lt;li&gt;only then install or ship the build&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is not an argument against coding agents. Claude helped diagnose and fix the crash as well. It is an argument for testing the boundary the compiler cannot prove.&lt;/p&gt;

&lt;p&gt;AI-generated code is especially good at looking locally correct. Framework callbacks fail globally: actor isolation, lifecycle, permissions, background execution, hardware state.&lt;/p&gt;

&lt;p&gt;The lazy guardrail is not another review checklist. It is one runnable smoke test that crosses the real boundary.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>swift</category>
      <category>ios</category>
      <category>debugging</category>
    </item>
    <item>
      <title>My AI Model Was 89% Accurate. It Missed 99% of the State I Cared About</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 20 Oct 2026 08:07:34 +0000</pubDate>
      <link>https://dev.to/toddsullivan/my-ai-model-was-89-accurate-it-missed-99-of-the-state-i-cared-about-3122</link>
      <guid>https://dev.to/toddsullivan/my-ai-model-was-89-accurate-it-missed-99-of-the-state-i-cared-about-3122</guid>
      <description>&lt;p&gt;I trained a small CoreML classifier to estimate whether a player was bored, in flow, frustrated, or about to quit.&lt;/p&gt;

&lt;p&gt;The training pipeline looked respectable:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;headless game simulator&lt;/li&gt;
&lt;li&gt;bots with randomised aim, dodging, recklessness and patience&lt;/li&gt;
&lt;li&gt;deterministic seeds&lt;/li&gt;
&lt;li&gt;player-level train/test split&lt;/li&gt;
&lt;li&gt;class-weighted loss&lt;/li&gt;
&lt;li&gt;132,139 labelled rows across 2,400 simulated players&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The test accuracy was &lt;strong&gt;89.4%&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Then I looked at the classes instead of the headline number:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;bored          recall 95.5%
flow           recall 94.8%
frustrated     recall 98.2%
aboutToQuit    recall  1.1%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model was almost completely missing the state I cared about most.&lt;/p&gt;

&lt;p&gt;There was another problem hiding underneath it. While watching the simulator, I found bots freezing in cleared rooms. The navigation was fine. The bot's stuck recovery was not.&lt;/p&gt;

&lt;p&gt;Every 30 frames it checked whether it had moved. If not, it sidestepped for 40 frames:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;clock&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;distance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pos&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lastPos&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;detour&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;40&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A sidestep into a wall was still active at the next check, so it could trigger another sidestep and loop forever. I had generated plausible-looking training rows from an agent that was sometimes just stuck.&lt;/p&gt;

&lt;p&gt;The fix was tiny: make the detour shorter than the check interval and alternate sides.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;clock&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;distance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pos&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lastPos&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;detour&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;
    &lt;span class="n"&gt;detourSide&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;detourSide&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I also added a simulator invariant: if a bot stays in a cleared room for more than 60 seconds, count it as a stall and fail the data run. The original issue appeared in 75 of 450 sessions. After the fix, the detector found zero stalls across 2,400 simulated players.&lt;/p&gt;

&lt;p&gt;Clean data did not magically fix the rare class. Overall accuracy stayed at 89.4%, while &lt;code&gt;aboutToQuit&lt;/code&gt; recall actually fell from 15.4% to 1.1%.&lt;/p&gt;

&lt;p&gt;That was the useful result.&lt;/p&gt;

&lt;p&gt;The simulator bug and the label problem were separate failures. Fixing corrupt behaviour made the dataset honest; it did not make the label learnable. The rare state still overlaps heavily with frustration and needs better labels or real player data, not another round of metric tuning.&lt;/p&gt;

&lt;p&gt;My rule now is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;simulators need behavioural invariants, not just successful exits&lt;/li&gt;
&lt;li&gt;splits must be by actor, not by row&lt;/li&gt;
&lt;li&gt;every important class gets its own metric&lt;/li&gt;
&lt;li&gt;low-confidence model output falls back to deterministic rules&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An AI pipeline can be reproducible, testable and 89% accurate while still failing its actual job.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>coreml</category>
      <category>gamedev</category>
    </item>
    <item>
      <title>The CoreML Model Was Fine. The Photo Was Too Big.</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 13 Oct 2026 08:03:32 +0000</pubDate>
      <link>https://dev.to/toddsullivan/the-coreml-model-was-fine-the-photo-was-too-big-ca7</link>
      <guid>https://dev.to/toddsullivan/the-coreml-model-was-fine-the-photo-was-too-big-ca7</guid>
      <description>&lt;p&gt;I hit a very ordinary computer-vision failure this week: the model was doing what I asked, but the image pipeline was making the task impossible.&lt;/p&gt;

&lt;p&gt;The app counts livestock on-device with a small YOLO/CoreML model. No network, no account, no server-side fallback. That constraint is useful, but it means the phone pipeline has to be honest about what the model actually sees.&lt;/p&gt;

&lt;p&gt;The first photo path looked sensible:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="nv"&gt;request&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;VNCoreMLRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;imageCropAndScaleOption&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;scaleFit&lt;/span&gt;
&lt;span class="n"&gt;handler&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perform&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One image in, detections out.&lt;/p&gt;

&lt;p&gt;Then I tested it on a real paddock photo with roughly &lt;strong&gt;250 sheep&lt;/strong&gt; in it. The count came back as &lt;strong&gt;14&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That was not a model mystery. A 12MP photo was being squeezed into the model input. By the time the flock reached the network, distant sheep were just a few pixels each. There was nothing clever for the detector to recover.&lt;/p&gt;

&lt;p&gt;The fix was boring and much better than changing the model:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;run the full image once&lt;/li&gt;
&lt;li&gt;run overlapping tiles over the same image&lt;/li&gt;
&lt;li&gt;map tile-relative boxes back into image coordinates&lt;/li&gt;
&lt;li&gt;drop boxes cut by an inner tile edge&lt;/li&gt;
&lt;li&gt;merge the remaining detections with class-agnostic NMS&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The important bit is that tiling changes the object size the model sees without changing the model.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="nv"&gt;regions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;CGRect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;y&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;width&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;height&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
    &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="k"&gt;Self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;tiles&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;aspect&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;aspect&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;across&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;overlap&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.25&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;region&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;regions&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;regionOfInterest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;region&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On a 4:3 photo that becomes &lt;strong&gt;12 tiles plus the full-frame pass&lt;/strong&gt;. More inference, but still on-device and still only a fraction of a second for a still photo.&lt;/p&gt;

&lt;p&gt;The same paddock photo went from &lt;strong&gt;14&lt;/strong&gt; to &lt;strong&gt;173&lt;/strong&gt; detections. Not perfect, but suddenly useful. Close-up hen photos held steady, which mattered just as much: the fix improved recall where scale was the problem without inflating easy counts.&lt;/p&gt;

&lt;p&gt;The non-obvious part was the merge step.&lt;/p&gt;

&lt;p&gt;Default IoU-only NMS is not enough when tiling. A partial head box can sit inside a whole-animal box and have low IoU, so I also merge on intersection-over-smaller. I also make the merge class-agnostic because the same animal can come back under two labels from different passes. Count the animal, not the label argument.&lt;/p&gt;

&lt;p&gt;The lesson was simple: before blaming the model, check what pixels you actually gave it.&lt;/p&gt;

&lt;p&gt;For on-device AI, the model is often the least flexible part of the system. The image pipeline around it is where the cheap wins hide.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>coreml</category>
      <category>computervision</category>
      <category>ios</category>
    </item>
    <item>
      <title>My AI Cron Job Failed Because .env Wasn't a Shell Script</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 06 Oct 2026 08:03:11 +0000</pubDate>
      <link>https://dev.to/toddsullivan/my-ai-cron-job-failed-because-env-wasnt-a-shell-script-17hd</link>
      <guid>https://dev.to/toddsullivan/my-ai-cron-job-failed-because-env-wasnt-a-shell-script-17hd</guid>
      <description>&lt;p&gt;I hit a small failure in an AI publishing cron job this morning that is exactly the kind of boring edge that makes agents unreliable.&lt;/p&gt;

&lt;p&gt;The job needed one secret:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DEVTO_API_KEY=...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The lazy version was to source the whole env file and let the shell export everything.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-a&lt;/span&gt;
&lt;span class="nb"&gt;.&lt;/span&gt; ~/.hermes/.env
&lt;span class="nb"&gt;set&lt;/span&gt; +a
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is fine until the env file stops being a shell script.&lt;/p&gt;

&lt;p&gt;This one had unrelated values in it. One contained an app path with spaces. Another looked enough like shell syntax to be treated as a command. The job still found the API key later, but the logs had the real problem:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Chrome.app/Contents/MacOS/Google: No such file or directory
fyqi: command not found
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing about the Dev.to API was broken. The agent had just widened the trust boundary for no reason.&lt;/p&gt;

&lt;p&gt;The fix was not to make the env parser clever. It was to stop parsing the file as code.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Path&lt;/span&gt;

&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;home&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;joinpath&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.hermes/.env&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ignore&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;match&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;^DEVTO_API_KEY=(.*)$&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;M&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;api_key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;match&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="se"&gt;\'&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is deliberately boring. It reads exactly the value the job needs and ignores everything else.&lt;/p&gt;

&lt;p&gt;The useful rule for agent jobs is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;if a task needs one credential, load one credential.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Do not source a shared env file just because it is convenient. AI agents already have enough ways to do surprising things. Giving a scheduled job a file full of executable shell lines is a cheap way to add another one.&lt;/p&gt;

&lt;p&gt;This is especially true for personal agent infrastructure, where the same &lt;code&gt;.env&lt;/code&gt; tends to collect API keys, browser paths, tunnel tokens, feature flags, old experiments, and comments that were never meant to be interpreted by &lt;code&gt;sh&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The failure mode is annoying rather than dramatic, which is why it is worth fixing. The job does not need a config framework. It just needs to stop executing config.&lt;/p&gt;

&lt;p&gt;Small boundary. Small parser. Fewer weird mornings.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>agents</category>
      <category>security</category>
    </item>
    <item>
      <title>The 8B Local Model Was Worse Than the 4B One</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 29 Sep 2026 08:02:49 +0000</pubDate>
      <link>https://dev.to/toddsullivan/the-8b-local-model-was-worse-than-the-4b-one-47h0</link>
      <guid>https://dev.to/toddsullivan/the-8b-local-model-was-worse-than-the-4b-one-47h0</guid>
      <description>&lt;p&gt;I tried the obvious local-LLM upgrade on a small Mac mini: move from a 4B model to an 8B model.&lt;/p&gt;

&lt;p&gt;It was worse.&lt;/p&gt;

&lt;p&gt;The machine is an 8GB Apple Silicon mini running a personal assistant stack. The useful path was already working with &lt;code&gt;mlx-community/Qwen3-4B-4bit&lt;/code&gt; through Swama's OpenAI-compatible endpoint:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;4B weights: ~2.3GB
warm plain reply: ~1s
warm tool-call round: ~3-4s model time
full tool response: ~7-18s, depending on external APIs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So I pulled the 8B quantized model to see if the extra capability was worth it.&lt;/p&gt;

&lt;p&gt;The first trivial request was enough:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8B weights: ~4.3GB
"say hello": 14s
throughput: 6.8 tok/s
swap churn: ~1.6GB in/out
free memory: basically zero
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That was not an edge case. That was the cheapest possible prompt.&lt;/p&gt;

&lt;p&gt;A real assistant request is heavier: system prompt, recent conversation, tool schemas, maybe a tool round, then a final answer. If the 8B model swaps on hello, it is not going to survive a weather/tool/home-status request while the rest of the app is running.&lt;/p&gt;

&lt;p&gt;The annoying part is that the model choice looked sensible on paper. 8B should be smarter. It probably is, in isolation. But local AI work is not model leaderboard work. It is memory pressure, cold starts, prompt prefill, tool reliability, and whether the box is still usable after the request.&lt;/p&gt;

&lt;p&gt;The fix was not glamorous:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;MAX_HISTORY_MESSAGES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;

&lt;span class="nd"&gt;@lru_cache&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;maxsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get_ai_handler&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;SwamaAIHandler&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cap the prompt history. Reuse the handler. Keep long-term memory as distilled facts, not raw transcript replay. Use the 4B model that stays resident and answers quickly enough.&lt;/p&gt;

&lt;p&gt;I also had to add boring production guardrails:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;run the real Swama binary, not the symlink, because first inference crashed loading MLX's metallib&lt;/li&gt;
&lt;li&gt;keep the model warm because idle eviction made cold starts painful&lt;/li&gt;
&lt;li&gt;strip &lt;code&gt;&amp;lt;think&amp;gt;&lt;/code&gt; blocks from streamed responses because &lt;code&gt;/no_think&lt;/code&gt; was not enough after tool calls&lt;/li&gt;
&lt;li&gt;bias the prompt toward tool use for live data, because small models will confidently answer from stale weights if you let them&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The lesson was simple: on tiny local hardware, the best model is the one that leaves enough machine behind for the product.&lt;/p&gt;

&lt;p&gt;The 4B model shipped. The 8B model got deleted.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>edgeai</category>
      <category>devops</category>
    </item>
    <item>
      <title>The AI Coding Agent Wasn't the Risk. The Missing Deploy Gate Was.</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 22 Sep 2026 08:03:37 +0000</pubDate>
      <link>https://dev.to/toddsullivan/the-ai-coding-agent-wasnt-the-risk-the-missing-deploy-gate-was-3cac</link>
      <guid>https://dev.to/toddsullivan/the-ai-coding-agent-wasnt-the-risk-the-missing-deploy-gate-was-3cac</guid>
      <description>&lt;p&gt;I had a small reminder this week that AI-assisted development does not really change the old rule: if a human or an agent can edit a deployment path, the deployment path needs to prove the important thing still works.&lt;/p&gt;

&lt;p&gt;The bug was not dramatic. A routing service sits behind a tiny Caddy gate because the upstream server has no auth of its own. Requests without &lt;code&gt;?key=&lt;/code&gt; should get &lt;code&gt;403&lt;/code&gt;; requests with a valid key should route.&lt;/p&gt;

&lt;p&gt;That was already checked after deploy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;check 403 &lt;span class="s2"&gt;"without key"&lt;/span&gt; &lt;span class="s2"&gt;"https://&lt;/span&gt;&lt;span class="nv"&gt;$FQDN&lt;/span&gt;&lt;span class="s2"&gt;/route?&lt;/span&gt;&lt;span class="nv"&gt;$ROUTE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
check 200 &lt;span class="s2"&gt;"with key"&lt;/span&gt;    &lt;span class="s2"&gt;"https://&lt;/span&gt;&lt;span class="nv"&gt;$FQDN&lt;/span&gt;&lt;span class="s2"&gt;/route?&lt;/span&gt;&lt;span class="nv"&gt;$ROUTE&lt;/span&gt;&lt;span class="s2"&gt;&amp;amp;key=&lt;/span&gt;&lt;span class="nv"&gt;$KEY&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Good enough for the live app. Not good enough for App Review.&lt;/p&gt;

&lt;p&gt;The app also needs a separate review key. A reviewer opening a fresh install cannot use a key already typed into my own devices. So Caddy accepts two values for the same query param:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;@authorized query key=LIVE_KEY key=REVIEW_KEY
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The failure mode was easy to miss: a deploy could ship a Caddyfile that still accepted the live key but silently dropped the review key. The health check would pass. The app would still work for me. Then days later App Review would open the build, hit the first-run screen, and reject it.&lt;/p&gt;

&lt;p&gt;That is the worst kind of infrastructure bug: green deploy, delayed failure, wrong human paying the cost.&lt;/p&gt;

&lt;p&gt;The fix was nine lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# The key issued to Apple's Beta App Review, when this machine holds one.&lt;/span&gt;
&lt;span class="c"&gt;# Absent file, no review key in play, nothing to check.&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; .secret-review &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;check 200 &lt;span class="s2"&gt;"review key"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="s2"&gt;"https://&lt;/span&gt;&lt;span class="nv"&gt;$FQDN&lt;/span&gt;&lt;span class="s2"&gt;/route?&lt;/span&gt;&lt;span class="nv"&gt;$ROUTE&lt;/span&gt;&lt;span class="s2"&gt;&amp;amp;key=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; .secret-review&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nv"&gt;ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No framework. No secret-management migration. Just make the deploy assert every key path that matters.&lt;/p&gt;

&lt;p&gt;Claude co-authored this change, which is the bit that made it worth writing down. Not because the agent did something dangerous. Because agent-written code has the same problem as human-written code, only faster: it can make a local patch that looks right while missing the operational contract around it.&lt;/p&gt;

&lt;p&gt;The useful pattern is simple:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;let the agent edit the boring deployment code&lt;/li&gt;
&lt;li&gt;make the deploy script test the real external behaviour&lt;/li&gt;
&lt;li&gt;fail before the broken state becomes somebody else's problem&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For anything with AI in the development loop, I care less about whether the diff looks plausible and more about whether the system can prove the side effect survived.&lt;/p&gt;

&lt;p&gt;In this case the proof is three HTTP codes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;without key -&amp;gt; 403
with key    -&amp;gt; 200
review key  -&amp;gt; 200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is not a big security architecture. It is just a tripwire in the place I am guaranteed to step before shipping.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>security</category>
      <category>claude</category>
    </item>
    <item>
      <title>The On-Device People Counter Only Worked After I Made It Boring</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 15 Sep 2026 08:02:58 +0000</pubDate>
      <link>https://dev.to/toddsullivan/the-on-device-people-counter-only-worked-after-i-made-it-boring-1557</link>
      <guid>https://dev.to/toddsullivan/the-on-device-people-counter-only-worked-after-i-made-it-boring-1557</guid>
      <description>&lt;p&gt;I built a tiny line-crossing people counter for a Seeed XIAO ESP32S3 Sense this week. Camera on the board, model on the board, counts on the board. No laptop in the loop unless I want the more accurate host-mode version.&lt;/p&gt;

&lt;p&gt;The first useful constraint was admitting what the hardware could actually do.&lt;/p&gt;

&lt;p&gt;The host detector runs at 416×416 and is about &lt;strong&gt;1.1 GFLOPs&lt;/strong&gt;. That is fine on a laptop. On an ESP32-S3 it is fantasy. The on-device detector is roughly &lt;strong&gt;15× smaller&lt;/strong&gt; and still takes about &lt;strong&gt;118 ms&lt;/strong&gt; per frame.&lt;/p&gt;

&lt;p&gt;At QVGA the budget ended up like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;JPEG decode      ~30 ms
preprocess        9 ms
model           118 ms
postprocess       2 ms
---------------------
total          ~160 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So the real target is not “real time” in the desktop-CV sense. It is about &lt;strong&gt;6 fps&lt;/strong&gt;, which is enough for someone walking through a doorway if the rest of the pipeline does not get clever.&lt;/p&gt;

&lt;p&gt;The counter is deliberately plain:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;capture the newest frame&lt;/li&gt;
&lt;li&gt;detect people&lt;/li&gt;
&lt;li&gt;use the bottom-center of each box as the foot point&lt;/li&gt;
&lt;li&gt;match points with a nearest-centroid tracker&lt;/li&gt;
&lt;li&gt;count when a track crosses a configured line&lt;/li&gt;
&lt;li&gt;log the event to serial, optionally expose &lt;code&gt;/counts&lt;/code&gt; over WiFi&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The foot point mattered more than I expected. Box centroids move around with posture and crop shape. If the counting line is on the floor, the bottom-center of the box is a better approximation of the thing that actually crosses the line.&lt;/p&gt;

&lt;p&gt;The tracker is also intentionally boring:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight cpp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// ponytail: no motion model, no re-id. Fine for a doorway at ~6 fps;&lt;/span&gt;
&lt;span class="c1"&gt;// swap in ByteTrack if people cross paths in frame.&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Tracker&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// id -&amp;gt; centroid, max distance, max age&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No Kalman filter. No re-identification. No cloud fallback. Just keep a track alive for a few missed frames and drop it when it gets too stale.&lt;/p&gt;

&lt;p&gt;That is not because those techniques are bad. It is because this is a doorway counter on a microcontroller. People mostly move in one direction, the camera points down, and the useful failure mode is obvious: if people cross paths in frame, this tracker is too small and should be replaced.&lt;/p&gt;

&lt;p&gt;That shape is easier to operate than a “smart” pipeline pretending it has more signal than it does.&lt;/p&gt;

&lt;p&gt;The board can serve its own tiny monitoring page when WiFi is configured:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GET /counts
{"in":14,"out":11,"occupancy":3}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But it also works serial-only with empty WiFi settings. That was another good constraint. The core counting path should not depend on a dashboard, a backend, or a phone app being alive.&lt;/p&gt;

&lt;p&gt;The bit I like most is that the counting logic has a host self-check separate from the ESP-IDF build:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;make &lt;span class="nt"&gt;-C&lt;/span&gt; firmware/test
&lt;span class="c"&gt;# ok: 6 counter tests&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For edge AI, that split is worth doing. The model and camera are the annoying parts to test. The line-crossing rules do not need to be. Pull the state machine into boring C++, beat it up on the host, then let the board worry about frames and inference.&lt;/p&gt;

&lt;p&gt;The lesson was the same one on-device AI keeps teaching me: the model is only one budget. Latency, memory, camera format, mounting angle, testability, and failure honesty matter just as much.&lt;/p&gt;

&lt;p&gt;A smaller model plus a boring counter beats an impressive pipeline that cannot keep up with someone walking through a door.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>esp32</category>
      <category>computervision</category>
      <category>edgeai</category>
    </item>
    <item>
      <title>The Credit Counter Race That Let 50 AI Requests Through</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 08 Sep 2026 08:03:40 +0000</pubDate>
      <link>https://dev.to/toddsullivan/the-credit-counter-race-that-let-50-ai-requests-through-4al4</link>
      <guid>https://dev.to/toddsullivan/the-credit-counter-race-that-let-50-ai-requests-through-4al4</guid>
      <description>&lt;p&gt;I fixed a boring little backend bug this week that would have been expensive in exactly the way AI features like to be expensive: quietly, under concurrency, while every individual request looked valid.&lt;/p&gt;

&lt;p&gt;The app has Claude-backed generation behind a monthly credit allowance. Simple shape:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;authenticate the user&lt;/li&gt;
&lt;li&gt;check their subscription and monthly allowance&lt;/li&gt;
&lt;li&gt;call Claude&lt;/li&gt;
&lt;li&gt;record the generation and token cost&lt;/li&gt;
&lt;li&gt;increment &lt;code&gt;credits_used&lt;/code&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That looks fine until you remember the model call is the slow part.&lt;/p&gt;

&lt;p&gt;The old gate read &lt;code&gt;credits_used&lt;/code&gt;, saw the user was under the cap, then let the request continue. Only after the multi-second Claude call did it increment the counter.&lt;/p&gt;

&lt;p&gt;So a user sitting at &lt;strong&gt;29 / 30&lt;/strong&gt; credits could fire &lt;strong&gt;50 parallel requests&lt;/strong&gt;. All 50 read &lt;code&gt;29&lt;/code&gt;. All 50 passed the gate. All 50 spent tokens. Then the counter landed at &lt;strong&gt;79&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;No exotic attack. No clever prompt injection. Just check-then-act around a slow side effect.&lt;/p&gt;

&lt;p&gt;The fix was to stop treating the credit check as application logic and make Postgres do the one thing it is very good at: serialize writes to the row.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;create&lt;/span&gt; &lt;span class="k"&gt;or&lt;/span&gt; &lt;span class="k"&gt;replace&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="k"&gt;public&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reserve_ai_credit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;p_user_id&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;p_month&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;p_limit&lt;/span&gt; &lt;span class="nb"&gt;integer&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;returns&lt;/span&gt; &lt;span class="nb"&gt;integer&lt;/span&gt; &lt;span class="k"&gt;language&lt;/span&gt; &lt;span class="n"&gt;plpgsql&lt;/span&gt; &lt;span class="k"&gt;security&lt;/span&gt; &lt;span class="k"&gt;definer&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="err"&gt;$$&lt;/span&gt;
&lt;span class="k"&gt;declare&lt;/span&gt;
  &lt;span class="n"&gt;v_used&lt;/span&gt; &lt;span class="nb"&gt;integer&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;begin&lt;/span&gt;
  &lt;span class="k"&gt;insert&lt;/span&gt; &lt;span class="k"&gt;into&lt;/span&gt; &lt;span class="k"&gt;public&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ai_usage&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;month&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;credits_used&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;values&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p_user_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p_month&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;on&lt;/span&gt; &lt;span class="n"&gt;conflict&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;month&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="k"&gt;update&lt;/span&gt;
    &lt;span class="k"&gt;set&lt;/span&gt; &lt;span class="n"&gt;credits_used&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ai_usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;credits_used&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;where&lt;/span&gt; &lt;span class="n"&gt;ai_usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;credits_used&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;p_limit&lt;/span&gt;
  &lt;span class="n"&gt;returning&lt;/span&gt; &lt;span class="n"&gt;credits_used&lt;/span&gt; &lt;span class="k"&gt;into&lt;/span&gt; &lt;span class="n"&gt;v_used&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;v_used&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="err"&gt;$$&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That &lt;code&gt;where ai_usage.credits_used &amp;lt; p_limit&lt;/code&gt; on the conflict update is the whole trick. Concurrent requests now queue on the same row. The first one increments. The next sees the updated value. Once the cap is reached, &lt;code&gt;returning&lt;/code&gt; gives nothing back and the request stops before the model call.&lt;/p&gt;

&lt;p&gt;The Edge Function now reserves a credit before calling Claude:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;used&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;supabase&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rpc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;reserve_ai_credit&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;p_user_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;p_month&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;month&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;p_limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;allowance&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;used&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;used&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Credit limit reached&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That changes the failure shape, so the other half of the fix was refunds. If the model call fails, or the structured response violates the schema, the function calls &lt;code&gt;refund_ai_credit&lt;/code&gt;. Failed generations should not bill the user, but they also should not leave the gate open until after the expensive work is done.&lt;/p&gt;

&lt;p&gt;While I was in there, I removed a duplicate credit gate from one function and made it call the shared &lt;code&gt;runCreditGate&lt;/code&gt; path like the others. That cut about &lt;strong&gt;52 lines&lt;/strong&gt; and made the concurrency fix apply everywhere instead of only to the function I happened to be staring at.&lt;/p&gt;

&lt;p&gt;A few adjacent bugs fell out of the same pass:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;uncapped text fields could push huge strings into prompts&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;avgSleepHours.toFixed()&lt;/code&gt; could throw when the client sent a string&lt;/li&gt;
&lt;li&gt;an unknown Claude model name logged cost as &lt;code&gt;$0&lt;/code&gt; silently&lt;/li&gt;
&lt;li&gt;one suggestion endpoint was returning &lt;code&gt;401&lt;/code&gt; for “no subscription”, which signs the user out instead of showing the paywall&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of those needed a grand framework. Just input clamps at the boundary, one shared gate, and loud logs when accounting metadata drifts.&lt;/p&gt;

&lt;p&gt;The useful lesson: if an AI feature has a quota, the quota has to be enforced before the model call, in the database, atomically. Anything else is a polite suggestion with a cloud bill attached.&lt;/p&gt;

&lt;p&gt;I ran the Supabase Edge Function helper tests after the fix: &lt;strong&gt;62 Deno tests, 0 failed&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>supabase</category>
      <category>postgres</category>
      <category>devops</category>
    </item>
    <item>
      <title>The Tracking Bug Wasn't the Model. It Was the Dot Lying.</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Tue, 01 Sep 2026 08:07:11 +0000</pubDate>
      <link>https://dev.to/toddsullivan/the-tracking-bug-wasnt-the-model-it-was-the-dot-lying-3gk7</link>
      <guid>https://dev.to/toddsullivan/the-tracking-bug-wasnt-the-model-it-was-the-dot-lying-3gk7</guid>
      <description>&lt;p&gt;I spent part of this week debugging an on-device livestock counter in a field-video mode. The model was fine. The count was not.&lt;/p&gt;

&lt;p&gt;The symptom from the real device run was simple: a tracking dot would leave one animal, drift onto its neighbour for a few frames, then snap back. In a dense mob that is exactly the kind of tiny lie that ruins the user's trust. If the UI says &lt;em&gt;that&lt;/em&gt; sheep is being tracked, it had better mean it.&lt;/p&gt;

&lt;p&gt;The first version of the counter was deliberately boring:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;detect animals with a small CoreML model&lt;/li&gt;
&lt;li&gt;maintain short-lived tracks between frames&lt;/li&gt;
&lt;li&gt;count when a track crosses a tripwire line&lt;/li&gt;
&lt;li&gt;let missed tracks coast for a few frames so one bad detection does not drop the animal&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That worked in still-ish tests. It failed once the phone and the sheep were both moving.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prediction is not presentation
&lt;/h2&gt;

&lt;p&gt;The lazy assumption was that the predicted track position could also be the drawn dot position.&lt;/p&gt;

&lt;p&gt;Bad assumption.&lt;/p&gt;

&lt;p&gt;A tracker is allowed to guess. A UI marker is a claim. Those are different contracts.&lt;/p&gt;

&lt;p&gt;A track can go unseen for up to 15 frames, because dropping it too quickly loses animals behind brief occlusion. During that time prediction uses the last observed heading plus camera-motion compensation. Good for matching. Bad for display. On a slow pan, a neighbouring sheep is about one body length away, so a few frames of coasting was enough to draw the dot onto the wrong animal.&lt;/p&gt;

&lt;p&gt;The fix was smaller than the bug report made it sound:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="nv"&gt;visibleCoastLimit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="nv"&gt;isDisplayPrediction&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;track&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;missedFrames&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;track&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;missedFrames&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;visibleCoastLimit&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;drawDot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;at&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;track&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;predictedCenter&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="nv"&gt;opacity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;isDisplayPrediction&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mf"&gt;0.45&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="nv"&gt;scale&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;isDisplayPrediction&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mf"&gt;0.7&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Matching can keep the longer memory. Display gets the honest one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Centroid distance breaks in a flock
&lt;/h2&gt;

&lt;p&gt;The second bug was association cost.&lt;/p&gt;

&lt;p&gt;I had already moved past purely greedy matching, but the cost was still mostly centroid distance. That sounds reasonable until you put the phone in front of a packed group of sheep. Every animal has another animal a body length away. Centroids are not enough signal.&lt;/p&gt;

&lt;p&gt;The better signal is overlap.&lt;/p&gt;

&lt;p&gt;Two boxes on the same animal overlap heavily between adjacent frames. Two boxes on neighbouring animals usually do not. So the association cost now prefers overlap first and only falls back to distance when there is no meaningful overlap:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight swift"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;iou&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;cost&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;iou&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;cost&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;normalisedDistance&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That one change makes the tracker much less willing to jump to a neighbour just because its centre is nearby. It also means tracks need to keep the whole observed box, not just a centre point and longest side. Slightly more state, much better truth.&lt;/p&gt;

&lt;h2&gt;
  
  
  The useful lesson
&lt;/h2&gt;

&lt;p&gt;Most ML product bugs are not model bugs.&lt;/p&gt;

&lt;p&gt;The detector can be good enough and the product can still feel broken because the tracking layer lies, the UI overstates certainty, or the recovery path optimizes for the wrong thing.&lt;/p&gt;

&lt;p&gt;In this case the fix was not a new model, a heavier tracker, or a dependency. It was separating prediction from presentation, then using the geometry the video already had.&lt;/p&gt;

&lt;p&gt;Boring, local, testable. My favourite kind of AI fix.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>swift</category>
      <category>coreml</category>
      <category>computervision</category>
    </item>
    <item>
      <title>Development Notes: Navigation UX Is a State Problem</title>
      <dc:creator>Todd Sullivan</dc:creator>
      <pubDate>Mon, 31 Aug 2026 17:21:46 +0000</pubDate>
      <link>https://dev.to/toddsullivan/development-notes-navigation-ux-is-a-state-problem-10kl</link>
      <guid>https://dev.to/toddsullivan/development-notes-navigation-ux-is-a-state-problem-10kl</guid>
      <description>&lt;p&gt;Recent FlatNav development has been a useful reminder that navigation UX is mostly a state-management problem pretending to be a design problem.&lt;/p&gt;

&lt;p&gt;The visible part is the map, route, buttons, and labels.&lt;/p&gt;

&lt;p&gt;The real work is deciding what the user needs to know right now, what can wait, and what should never be shown at all.&lt;/p&gt;

&lt;p&gt;Good navigation software has to stay calm while the user is not. That means the interface needs to handle uncertainty without pushing it onto the person using it.&lt;/p&gt;

&lt;p&gt;The useful questions are simple:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;what state is the journey in?&lt;/li&gt;
&lt;li&gt;what is the next useful action?&lt;/li&gt;
&lt;li&gt;has the user changed context?&lt;/li&gt;
&lt;li&gt;does the UI need to interrupt, confirm, or stay quiet?&lt;/li&gt;
&lt;li&gt;is this information actionable now, or just noise?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most improvements come from reducing interpretation.&lt;/p&gt;

&lt;p&gt;If the user has to stop and decode the screen, the app has already leaked complexity. If a feature adds more choices than it removes, it probably made the system feel smarter while making the user slower.&lt;/p&gt;

&lt;p&gt;That is why the best development cycles often look small from the outside: clearer states, fewer edge-case dead ends, tighter feedback after actions, better defaults, less UI competing with the next step.&lt;/p&gt;

&lt;p&gt;There is a lazy engineering lesson in that.&lt;/p&gt;

&lt;p&gt;Before adding another control, another setting, or another explanation, ask whether the state model is doing its job. Often the fix is not more interface. It is making the existing state clearer, deleting an unnecessary decision, or moving logic into the product so the user does not have to carry it.&lt;/p&gt;

&lt;p&gt;Navigation products get better when they get quieter.&lt;/p&gt;

</description>
      <category>ux</category>
      <category>mobile</category>
      <category>engineering</category>
      <category>product</category>
    </item>
  </channel>
</rss>
