<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Marc</title>
    <description>The latest articles on DEV Community by Marc (@marc_kumiko).</description>
    <link>https://dev.to/marc_kumiko</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4001120%2F74d319bd-196a-45cb-843d-d70b2e6a54c5.png</url>
      <title>DEV Community: Marc</title>
      <link>https://dev.to/marc_kumiko</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/marc_kumiko"/>
    <language>en</language>
    <item>
      <title>From idea to both app stores in one week, built on Kumiko with AI</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Wed, 18 Nov 2026 13:23:19 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/from-idea-to-both-app-stores-in-one-week-built-on-kumiko-with-ai-522b</link>
      <guid>https://dev.to/marc_kumiko/from-idea-to-both-app-stores-in-one-week-built-on-kumiko-with-ai-522b</guid>
      <description>&lt;p&gt;On Friday, November 14, Timeline Murder was an idea and an empty repo. By Tuesday the Android build was in a closed test on Google Play and the iOS build was sitting in App Review. Google took hours. Apple is still thinking about it. If Apple weren't a turtle, you could already play it on your iPhone.&lt;/p&gt;

&lt;p&gt;This post covers what we built, what it cost, and which parts we didn't have to build because the backend already had them. "We" is me and Anne. Coding agents wrote most of the code; we decided what the game should be, read the diffs and played it a lot.&lt;/p&gt;

&lt;h2&gt;
  
  
  The game
&lt;/h2&gt;

&lt;p&gt;Timeline Murder is a logic crime puzzle. One house, five epochs: 1724, 1924, 1962, 1987 and 2046. Every case is a floor plan of the house with rooms, furniture and six people, one of them the victim. The clues say things like "Barnaby sat in an armchair" or "Eleanor stood next to a bookshelf". You place everyone on the plan until all clues hold, then make your accusation.&lt;/p&gt;

&lt;p&gt;Story mode runs in seasons. A season belongs to an epoch, has episodes with a running plot, scene images, music and voiced lines, and ends with a finale. There is also free play, a daily case, a multiplayer race (with a ghost opponent when nobody else is online) and co-op where friends share clues.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiy52b77dy155b1mw71y6.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiy52b77dy155b1mw71y6.jpg" alt="A solved 1924 case on iPhone" width="800" height="1739"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers
&lt;/h2&gt;

&lt;p&gt;From the repo, as of November 18:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;92 commits and 68 merged pull requests&lt;/li&gt;
&lt;li&gt;Commits per day: 4 on the 14th, 44 on the 15th, 23 on the 16th, 21 on the 17th&lt;/li&gt;
&lt;li&gt;About 64,900 lines of TypeScript and TSX&lt;/li&gt;
&lt;li&gt;108 test files&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;From the production database (read replica, counts only):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;5 epochs, 7 seasons, 50 episodes, 57 cases&lt;/li&gt;
&lt;li&gt;2,598 season variants and 59 asset packs&lt;/li&gt;
&lt;li&gt;7,397 events since November 15&lt;/li&gt;
&lt;li&gt;98 job runs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We also have accounts, played cases and one purchase. Nearly all of that is us and our testers, so I won't pretend it's traction.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the content cost
&lt;/h2&gt;

&lt;p&gt;Every scene image, episode text and voiced line came out of an AI pipeline that runs as a server job. Each model call is recorded as an event with the model, token usage, latency and the cost OpenRouter reports. So the bill is a database query:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;What&lt;/th&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Calls&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;th&gt;Avg. time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Images&lt;/td&gt;
&lt;td&gt;gemini-3.1-flash-image&lt;/td&gt;
&lt;td&gt;309&lt;/td&gt;
&lt;td&gt;$21.01&lt;/td&gt;
&lt;td&gt;10.2 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Texts&lt;/td&gt;
&lt;td&gt;gpt-6-luna&lt;/td&gt;
&lt;td&gt;876&lt;/td&gt;
&lt;td&gt;$5.79&lt;/td&gt;
&lt;td&gt;117.6 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Speech&lt;/td&gt;
&lt;td&gt;gpt-audio-mini&lt;/td&gt;
&lt;td&gt;1,477&lt;/td&gt;
&lt;td&gt;$0.34&lt;/td&gt;
&lt;td&gt;1.2 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2,662&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$27.14&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The OpenRouter key for the pipeline shows $27.60 for the same days, so the recording is close enough to trust.&lt;/p&gt;

&lt;p&gt;Images are 77% of the money. Texts take the longest by far: a season plot has to stay consistent across episodes, keep the clues solvable and match the epoch, and the model spends almost two minutes per call getting there. Speech is almost free.&lt;/p&gt;

&lt;h3&gt;
  
  
  $17 of trying models before the pipeline existed
&lt;/h3&gt;

&lt;p&gt;The table above starts on November 16. The two days before that don't show up in our database, because there was nothing to record into yet. We tried image models, generated the first title images by hand, tested a few voices and let a music model write the theme. That ran on a different OpenRouter key:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Image model: $15.10&lt;/li&gt;
&lt;li&gt;Text model: $0.95&lt;/li&gt;
&lt;li&gt;Music: $0.52&lt;/li&gt;
&lt;li&gt;Two speech models: $0.42&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;About $17 in total. From the first prompt to five epochs of finished content, we spent roughly $45 on AI generation. That excludes the coding agents, which are a separate bill.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we didn't build
&lt;/h2&gt;

&lt;p&gt;This is the part I care about most. Timeline Murder runs on Kumiko, our TypeScript backend framework. The game code is the puzzle engine, the content pipeline and about ten game features (timeline, progress, multiplayer, friends, moderation, purchases and so on). Everything below came from the framework and needed configuration, not code:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sign-in.&lt;/strong&gt; Sign in with Apple, Google and an email link. Apple insists on Sign in with Apple if you offer any other social login, so this had to be there before submission. It was one feature to enable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Account deletion.&lt;/strong&gt; Apple also wants users to delete their account inside the app. The framework's user data rights feature already does deletion and export, so the app just links to it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Push notifications.&lt;/strong&gt; Push notifications go out through the framework's push channel. We wrote the messages, not the delivery.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Background jobs.&lt;/strong&gt; Season generation runs as a job on a separate worker. When a job gets delivered twice, it resumes the unfinished season instead of starting over. That fix was ours; the job runner and the worker setup were not.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AI connections and cost tracking.&lt;/strong&gt; The model setup lives in an admin screen: an AI connection to OpenRouter and default models for text and image. Changing the image model is a form edit, not a deploy. Every call goes through the AI foundation package, which is where the cost events in the table come from.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Admin screens.&lt;/strong&gt; Reviewing a season before release, regenerating a single scene, giving testers early access to episodes: all of that sits in the admin UI the framework builds from the feature definitions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Event sourcing and audit.&lt;/strong&gt; Every change is an event. When a player says their progress vanished, we can see exactly what happened. That is also why the content numbers above were one query away.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tests against a real stack.&lt;/strong&gt; The integration tests start the server with a real Postgres and talk HTTP to it. Twenty test files use that stack. When an agent breaks the purchase flow, a test goes red before a human notices.&lt;/p&gt;

&lt;p&gt;The coding agents were fast because the hard and boring parts were already done and documented. They wrote game logic and wired features. They didn't invent auth.&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting into the stores
&lt;/h2&gt;

&lt;p&gt;A good part of the week went into the stores, not the game. A few things that bit us:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Product IDs.&lt;/strong&gt; We set up subscriptions in App Store Connect first, with descriptive IDs of 52 characters. Google Play caps subscription product IDs at 40 characters. So Android got short IDs, and the shared API contract now knows which IDs belong to which platform and rejects a purchase that names the wrong one. Purchases are verified on the server against Apple and Google before anything unlocks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;User-generated names.&lt;/strong&gt; Players pick display names and play against each other. App Review guideline 1.2 asks for a way to report and block other users when there's user-generated content. We built reporting and blocking on day two, before the first submission, which beat waiting for a rejection.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The closed test that stayed old.&lt;/strong&gt; Our build tooling submitted every Android build to the internal track as a draft. The closed test track, the one testers actually use, kept serving an older build for a day before we noticed. One line in the submit config fixed it. Checking which build the testers have, rather than which one you uploaded, is the lesson.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Store assets in two languages.&lt;/strong&gt; Screenshots for iPhone and iPad, a feature graphic, store texts, an age rating questionnaire and review notes, all in German and English. Screenshots come from a script that drives the app through ten scenes per device and locale.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F97bwtqssp8kq967eg3n9.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F97bwtqssp8kq967eg3n9.jpg" alt="Case resolution on iPad" width="800" height="600"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;p&gt;The Android closed test is open. The iOS build is in review, and judging by our past apps that takes one to two weeks. The website already has the daily case to play in the browser.&lt;/p&gt;

&lt;p&gt;You don't need either store to play. The daily case runs in the browser at &lt;a href="https://timelinemurder.cosmicdriftgamestudio.com/en/" rel="noopener noreferrer"&gt;timelinemurder.cosmicdriftgamestudio.com/en&lt;/a&gt;, no account needed. It's the same case for everyone, and the server checks every time before it lands on the leaderboard.&lt;/p&gt;

&lt;p&gt;So: who's faster? Solve today's case, hit share and paste your time and rank in the comments. And if you have an idea for the game, a sixth epoch, a new kind of clue, a mode we haven't thought of, put it there too.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>gamedev</category>
      <category>typescript</category>
      <category>buildinpublic</category>
    </item>
    <item>
      <title>An agent rebuilds your PDF letterhead: what Opus on high effort, Opus on low, a cheap model and a local 9B make of it</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Wed, 11 Nov 2026 20:29:10 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/an-agent-rebuilds-your-pdf-letterhead-what-opus-on-high-effort-opus-on-low-a-cheap-model-and-a-220</link>
      <guid>https://dev.to/marc_kumiko/an-agent-rebuilds-your-pdf-letterhead-what-opus-on-high-effort-opus-on-low-a-cheap-model-and-a-220</guid>
      <description>&lt;p&gt;Kumiko has a content editor for PDF and mail templates: invoices, letters, statements. Nobody wants to rebuild their letterhead by hand in a new editor, so we added an import. You upload the PDF you use today and an agent rebuilds it as a normal, editable template. It looks at the pages as images, puts header, footer, logo, margins, fonts and paper into a layout, writes the body with placeholders, renders a preview, compares it with the original and corrects up to three times. At the end it shows a card with the original next to the draft and two lists: what stayed fixed text, and what it couldn't reproduce.&lt;/p&gt;

&lt;p&gt;The agent doesn't write HTML. It calls the same write handlers the editor uses (create a layout, patch a draft, upload a resource), and everything it writes stays a draft until a person publishes it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;Three made-up fixtures, modelled on real templates, with invented names, IBANs and amounts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a club invoice with an embedded logo, address window, item table with sub-lines, subtotals and a three-column footer,&lt;/li&gt;
&lt;li&gt;a two-page bank tax statement with a vector logo, corner and fold marks, a monospace body next to a sans-serif footer, and a running header on page 2,&lt;/li&gt;
&lt;li&gt;a garage invoice delivered as a scan with no text layer, with coloured waves at the top and bottom edge and cream paper.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each run goes through the real stack: HTTP, the agent's turn stream, Postgres, and the puppeteer service that rasterises PDFs and renders previews. The benchmark may say "go on" twice and approves the logo upload the way a user would. It records tool calls, LLM calls, tokens, cost, wall time and SSIM, and writes the original and the draft side by side into one image. Two runs per template and model; Opus on high effort ran in two rounds, before and after the last editor fixes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Our first grades were too kind
&lt;/h2&gt;

&lt;p&gt;The first write-up of this measurement gave Opus grades between 1-2 and 2, on the German school scale where 1 is best and 6 is worst. Then we put the pictures next to each other properly. The coloured areas were missing, the waves were missing, table lines, the logo position, the header data on page 2. Nothing deserved better than a 3.&lt;/p&gt;

&lt;p&gt;Most of it wasn't the model. The editor couldn't express what a normal business letter has. An image had no print width and no alignment, so a right-aligned logo ended up on the left. Tables had no cell padding, no width and no line above a total. There were no fold or corner marks, no header that starts on page 2, no paper colour.&lt;/p&gt;

&lt;p&gt;So we did what the plan had asked for from the start and skipped: rebuild all three templates by hand in the editor, without any AI, and put each one next to its original. Every gap that showed up became an editor feature. Only when the hand-built versions matched did the models run again.&lt;/p&gt;

&lt;p&gt;From now on we only write down checklists per template: what is there, what is missing. The grade comes from the person who owns the product, looking at the pictures.&lt;/p&gt;

&lt;h2&gt;
  
  
  The models
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Claude Opus 5.5 on our own Anthropic key, with effort &lt;code&gt;high&lt;/code&gt; (the default) and with effort &lt;code&gt;low&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;gpt-6-luna through OpenRouter, as the cheap option&lt;/li&gt;
&lt;li&gt;Qwen 3.5 9B on a Mac through Ollama, as the self-hosted option&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Runs finished&lt;/th&gt;
&lt;th&gt;Cost per template&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Verdict&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Opus 5.5, effort high&lt;/td&gt;
&lt;td&gt;11 of 12&lt;/td&gt;
&lt;td&gt;$0.93 to $1.80&lt;/td&gt;
&lt;td&gt;130 to 340 s&lt;/td&gt;
&lt;td&gt;close to the originals, accepted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Opus 5.5, effort low&lt;/td&gt;
&lt;td&gt;5 of 6&lt;/td&gt;
&lt;td&gt;$0.29 to $0.97&lt;/td&gt;
&lt;td&gt;47 to 114 s&lt;/td&gt;
&lt;td&gt;letters fine, the scan loses its header wave and paper colour&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-6-luna&lt;/td&gt;
&lt;td&gt;5 of 6&lt;/td&gt;
&lt;td&gt;$0.03 to $0.10&lt;/td&gt;
&lt;td&gt;260 to 480 s&lt;/td&gt;
&lt;td&gt;rejected: header, footer and spacing wrong&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen 3.5 9B&lt;/td&gt;
&lt;td&gt;0 of 3, in two rounds&lt;/td&gt;
&lt;td&gt;$0&lt;/td&gt;
&lt;td&gt;-&lt;/td&gt;
&lt;td&gt;-&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Opus on high effort
&lt;/h2&gt;

&lt;p&gt;Opus rebuilds the club invoice with the logo on the right, the sender line, the info block, items with their sub-lines, subtotals, a bold total, the IBAN in bold in the payment text and the three-column footer. On the tax statement it places the bank address and the date block right-aligned, sets corner marks, rules the table and puts the page number into the footer. On the scan it cuts the waves out of the page image as two pictures, places them at the paper edge and reads the cream paper colour from the page margin.&lt;/p&gt;

&lt;p&gt;The waves and the logos needed a new tool option. Our image tool could only hand over images that sit in the PDF as embedded files. A scan is one big image and a vector logo is drawing commands. So the tool got a &lt;code&gt;crop&lt;/code&gt; option: a page and a box in pixels of the page image the model has already seen. The service cuts the box out and stores it as the logo.&lt;/p&gt;

&lt;p&gt;What's left: three fold marks on the tax statement where the original has one, table text a shade lighter than the original, and a logo a few millimetres too high on the club invoice.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8leb48mflj1hgc4bvrsn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8leb48mflj1hgc4bvrsn.png" alt="Scanned garage invoice next to the rebuild by Opus on high effort" width="799" height="557"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Left: the scanned original. Right: Opus on high effort. The preview data has three line items, the original five.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Opus on low effort
&lt;/h2&gt;

&lt;p&gt;Low costs about half. On the letters you'd hardly see a difference: titles and table headers come out a bit larger and bolder, and the running header on page 2 sits higher than it should. On the scan the difference is large. With low effort Opus made five tool calls, never rendered a preview and never compared. The header wave is missing, the paper is white, and the title overlaps the address block.&lt;/p&gt;

&lt;p&gt;The import is a one-off setup step, so we leave it on high. In Kumiko the import isn't a separate agent. It is a set of tools inside the normal assistant, so it runs with the effort the tenant has configured, and the factory default is high.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft3hn560in63sjoa92817.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft3hn560in63sjoa92817.png" alt="Same scan rebuilt by Opus on low effort" width="799" height="557"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Same scan, Opus on low effort: no waves, white paper, title on top of the address block.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  luna
&lt;/h2&gt;

&lt;p&gt;luna is cheap: 3 to 10 cents per template. It gets content and placeholders right and struggles with everything around them. Line spacing in the header doesn't match, footers fall apart, and on one run it asked questions until it ran out of turns. For a rebuild that should look like the original, that is not usable.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq4a7ic1o0mpnjtu4r4dz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq4a7ic1o0mpnjtu4r4dz.png" alt="Same scan rebuilt by gpt-6-luna" width="799" height="557"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Same scan, gpt-6-luna: the waves are there, but title, address and vehicle table land on top of each other.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Qwen 9B on a Mac
&lt;/h2&gt;

&lt;p&gt;0 of 3, in two rounds. Two runs hit the provider timeout as soon as the page images arrived. The others ran out of steps with text answers and malformed tool calls.&lt;/p&gt;

&lt;p&gt;A 9B model on a laptop is slow with images, and the task needs vision, a long prompt and about twenty correct tool calls in a row. At this size it doesn't work. Bigger self-hosted models with vision and tool use, for example on a GPU behind vLLM, we haven't measured.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two runs that ended in an error
&lt;/h2&gt;

&lt;p&gt;Two of 18 Opus runs ended with &lt;code&gt;agent.tool_not_allowed&lt;/code&gt;, both times after the draft was already done. Our agent doesn't hand the model every tool. It gets a directory of actions plus two fixed tools, &lt;code&gt;describe_action&lt;/code&gt; and &lt;code&gt;run_action&lt;/code&gt;, and &lt;code&gt;run_action&lt;/code&gt; is unwrapped into the named action. If the name isn't in the directory or the input isn't an object, the call stays &lt;code&gt;run_action&lt;/code&gt;, and the agent ends the turn rather than letting a model keep writing after it asked for something that doesn't exist.&lt;/p&gt;

&lt;p&gt;We don't know yet which of the two it was. &lt;code&gt;run_action&lt;/code&gt; now also accepts the input as a JSON string, which models sometimes send for large objects, and the benchmark logs the action name and the type of the input, so the next case will show the cause.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the money goes
&lt;/h2&gt;

&lt;p&gt;A high-effort import costs $1 to $1.80. Most of it is the cache: every LLM call sends the whole conversation again, page images included, and a run makes 15 to 25 calls. Against a tenant's monthly AI budget on our platform key, one import counts 120,000 to 170,000 tokens. For something you do once per template, that's fine.&lt;/p&gt;

&lt;h2&gt;
  
  
  SSIM doesn't help
&lt;/h2&gt;

&lt;p&gt;We hoped image similarity would do the grading. Finished rebuilds scored between 0.66 and 0.83. When a run failed and left the starting template untouched, that blank template scored 0.77 to 0.84, higher than most rebuilds. White paper dominates the score. The verdicts come from looking at the pictures.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this doesn't prove
&lt;/h2&gt;

&lt;p&gt;Three templates, two runs each, judged by eye, and the editor changed between rounds, so early and late rounds aren't a clean before and after. One limit sits in the product: placeholders in a layout header. One layout serves many templates, so "tax year 2025 · depot 0000" in the running header stays fixed text.&lt;/p&gt;

&lt;h2&gt;
  
  
  An open question
&lt;/h2&gt;

&lt;p&gt;The result card lists what stayed fixed text, so real customer data from the upload doesn't end up in the template unnoticed. How do you get a person to actually read that list before they publish?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>typescript</category>
      <category>pdf</category>
    </item>
    <item>
      <title>Our coding agents skipped the pre-push hook on 118 of 411 pushes</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Sat, 07 Nov 2026 10:10:58 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/our-coding-agents-skipped-the-pre-push-hook-on-118-of-411-pushes-27j0</link>
      <guid>https://dev.to/marc_kumiko/our-coding-agents-skipped-the-pre-push-hook-on-118-of-411-pushes-27j0</guid>
      <description>&lt;p&gt;This week a lot of people on X argued about "software factories" for AI coding. Some companies sell pipelines of agents, and their critics say most of it is marketing and the parts that work are the deterministic ones.&lt;/p&gt;

&lt;p&gt;We run something close to a factory. Coding agents write most of the code for our TypeScript framework, Kumiko, and the apps built on it, and an automated review has read 5,322 of our merged PRs since May 27. The rules for the agents are about 3,100 lines of Markdown in 32 files. I went through our agent transcripts to see which of those rules held. The critics are right, but "deterministic" covers less than I thought.&lt;/p&gt;

&lt;h2&gt;
  
  
  A note didn't hold
&lt;/h2&gt;

&lt;p&gt;On November 1 we wrote down that the main agent starts coding agents itself, with no Opus agent in between that only passes the prompt along. Between then and the evening of November 5, 22 general-purpose agents started a coding agent anyway, 53 times in total.&lt;/p&gt;

&lt;p&gt;That evening we added a hook that refuses the spawn when a subagent tries it. It has fired once since, in our own test. In the two days after, coding agents were started only by the main agent or by the one exception we allow.&lt;/p&gt;

&lt;p&gt;So the hook can't take the credit, at least not yet. The same commit rewrote the rule, and something the hook can't touch changed too: the main agent used to hand work to a general-purpose agent about 20 times a day, and since then it's about 3 times a day. The hook only checks spawns from subagents. The drop came from the rewritten Markdown, or from a different mix of work in those two days.&lt;/p&gt;

&lt;h2&gt;
  
  
  A hook didn't hold either
&lt;/h2&gt;

&lt;p&gt;Our pre-push hook runs typecheck, lint, unit tests and our own guards. Skipping it with &lt;code&gt;--no-verify&lt;/code&gt; or &lt;code&gt;PRE_PUSH_SKIP=1&lt;/code&gt; is on the short list of hard bans in our agent config, with one exception: the failure is outside your diff, you can show who caused it, and you ran the skipped checks on your own files.&lt;/p&gt;

&lt;p&gt;Since October 20 our transcripts contain 411 push commands, and 118 of them skipped the hook.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Repo&lt;/th&gt;
&lt;th&gt;Push commands&lt;/th&gt;
&lt;th&gt;Skipped the hook&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Infrastructure&lt;/td&gt;
&lt;td&gt;104&lt;/td&gt;
&lt;td&gt;69&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Framework&lt;/td&gt;
&lt;td&gt;135&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;One of the apps&lt;/td&gt;
&lt;td&gt;57&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Everything else&lt;/td&gt;
&lt;td&gt;115&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;I read the reasons the agents wrote next to the skips. Most point to red outside their diff, often with the three proofs listed. Some name tests that time out when the machine is busy, and the exception explicitly excludes flaky tests. Some give no reason at all.&lt;/p&gt;

&lt;p&gt;The infrastructure repo explains most of the skips. Its hook also scans the neighbouring repos, so it goes red on code the agent never touched. When a gate keeps failing for someone else's change, the exception gets used on every push, and after a while that's simply how pushing works in that repo. A memory note our agents keep for themselves even called &lt;code&gt;PRE_PUSH_SKIP=1&lt;/code&gt; the documented pattern. I corrected it today.&lt;/p&gt;

&lt;h2&gt;
  
  
  What held
&lt;/h2&gt;

&lt;p&gt;The checks on GitHub. In every repo those pushes went to, &lt;code&gt;main&lt;/code&gt; only takes a merge after the required checks pass, and the rule applies to admins too. All 118 skipped pushes still had to get through CI before anything merged. An agent on a laptop can't skip a check that runs somewhere else.&lt;/p&gt;

&lt;p&gt;Our cast audit shows the same thing. It counts suspicious &lt;code&gt;as&lt;/code&gt; casts in the framework against a frozen baseline and fails when a change adds one. It runs inside the same pre-push hook, so it got skipped along with everything else, but CI runs it too. The baseline stood at 349 in June, after we extended the audit's coverage, and it's at 69 now.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd tell someone building one
&lt;/h2&gt;

&lt;p&gt;Put the check that must hold on the server. Keep the local hook fast, and let it look only at the diff being pushed, or it will cry wolf until nobody listens. Write the Markdown anyway, as the reason behind a check, but don't count on it to be the check.&lt;/p&gt;

&lt;p&gt;Speed matters for the same reason. When we moved our per-file guards into Biome plugins on November 4, lint got two to three times slower. A slower hook is one more reason to skip it, and we haven't fixed that yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this doesn't prove
&lt;/h2&gt;

&lt;p&gt;The numbers come from our agent transcripts in this workspace, counted per tool call. A push that was retried counts twice.&lt;/p&gt;

&lt;p&gt;The reasons come from the short notes the agents wrote next to each push. I didn't replay the sessions, so I can't say how many of the claimed three proofs would hold up.&lt;/p&gt;

&lt;p&gt;The spawn hook has two days of data and hasn't been triggered by real work yet.&lt;/p&gt;

&lt;p&gt;We didn't compare bug rates before and after any of this, so I can't say what the skipped hooks cost us or how much CI caught in their place.&lt;/p&gt;

&lt;h2&gt;
  
  
  A question
&lt;/h2&gt;

&lt;p&gt;If you run agents on your repos, which check do you treat as the one that can't be skipped? And how do you keep the local one from going red on code the agent didn't touch?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>git</category>
      <category>devops</category>
      <category>discuss</category>
    </item>
    <item>
      <title>An AST fingerprint and an embedding model both look for duplicate code. Here's where each one is wrong</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Tue, 03 Nov 2026 12:14:06 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/an-ast-fingerprint-and-an-embedding-model-both-look-for-duplicate-code-heres-where-each-one-is-2noe</link>
      <guid>https://dev.to/marc_kumiko/an-ast-fingerprint-and-an-embedding-model-both-look-for-duplicate-code-heres-where-each-one-is-2noe</guid>
      <description>&lt;p&gt;We run two duplicate finders over our TypeScript repos. One is static: it parses every function, throws away the names and compares what is left. The other asks a local embedding model what each function means and compares the vectors. They disagree a lot, and most of what we learned came from the cases where they disagree.&lt;/p&gt;

&lt;p&gt;For this post I rebuilt both as small standalone scripts, with fourteen sample functions that reproduce the cases we kept running into. Everything below is real output from those scripts. The code is here: &lt;a href="https://github.com/CosmicDriftGameStudio/duplicate-finder-samples" rel="noopener noreferrer"&gt;https://github.com/CosmicDriftGameStudio/duplicate-finder-samples&lt;/a&gt;. You need Bun, and for the embedding part Ollama with &lt;code&gt;nomic-embed-text&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why we bother
&lt;/h2&gt;

&lt;p&gt;We had the static check for months, but nothing ran it. When we finally ran it over all our repos at the end of October, it reported 118 groups that span files or modules. Typically the same logic had been written a second time by someone who didn't find the first version. At one point we had six copies of an HTML escape function.&lt;/p&gt;

&lt;h2&gt;
  
  
  The static finder: a fingerprint per function
&lt;/h2&gt;

&lt;p&gt;The script uses the TypeScript compiler to find every function, method and arrow function with a body of at least 120 characters. Shorter bodies match each other all the time and are never worth extracting. Each body is then normalized: every identifier becomes &lt;code&gt;_&lt;/code&gt;, every string and number becomes &lt;code&gt;#&lt;/code&gt;, whitespace goes away.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/"&lt;/span&gt;&lt;span class="se"&gt;[^&lt;/span&gt;&lt;span class="sr"&gt;"&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;*"/g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;#&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/'&lt;/span&gt;&lt;span class="se"&gt;[^&lt;/span&gt;&lt;span class="sr"&gt;'&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;*'/g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;#&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/`&lt;/span&gt;&lt;span class="se"&gt;[^&lt;/span&gt;&lt;span class="sr"&gt;`&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;*`/g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;#&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\b\d&lt;/span&gt;&lt;span class="sr"&gt;+&lt;/span&gt;&lt;span class="se"&gt;(\.\d&lt;/span&gt;&lt;span class="sr"&gt;+&lt;/span&gt;&lt;span class="se"&gt;)?&lt;/span&gt;&lt;span class="sr"&gt;n&lt;/span&gt;&lt;span class="se"&gt;?\b&lt;/span&gt;&lt;span class="sr"&gt;/g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;#&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\b(&lt;/span&gt;&lt;span class="sr"&gt;true|false|null|undefined&lt;/span&gt;&lt;span class="se"&gt;)\b&lt;/span&gt;&lt;span class="sr"&gt;/g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;#&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\b[&lt;/span&gt;&lt;span class="sr"&gt;a-zA-Z_$&lt;/span&gt;&lt;span class="se"&gt;][\w&lt;/span&gt;&lt;span class="sr"&gt;$&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;*&lt;/span&gt;&lt;span class="se"&gt;\b&lt;/span&gt;&lt;span class="sr"&gt;/g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;KEYWORDS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;_&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The normalized body plus the parameter count is the fingerprint. Two functions with the same fingerprint are an exact group. An optional second pass compares the rest pairwise: 70% edit distance on the normalized body, 30% overlap of the identifiers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ bun ast-duplicates.ts samples --fuzzy 0.8
exact   1.00  accounts/disable-account.ts:disableAccount + accounts/enable-account.ts:enableAccount
exact   1.00  forms/is-field-row.ts:isFieldRow + scheduling/is-instant-like.ts:isInstantLike
similar 0.87  feed/escape-xml.ts:escapeXml + mail/escape-html.ts:escapeHtml
3 group(s)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  What it's good at: copies that only differ in names and values
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;enableAccount&lt;/code&gt; and &lt;code&gt;disableAccount&lt;/code&gt; are the same handler with &lt;code&gt;true&lt;/code&gt; swapped for &lt;code&gt;false&lt;/code&gt; and one audit string changed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;disableAccount&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;accountId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;actorId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;account&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;accounts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;accountId&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;account&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`account &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;accountId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; not found`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;account&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;enabled&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;account&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;updated&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;accounts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;accountId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;enabled&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;audit&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;accountId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;actorId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;account.disabled&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After normalization the two bodies are identical, so the fingerprint matches with a score of 1.0. There's no threshold and nothing to tune, and the same input always gives the same answer. That's why this is the check that runs automatically for us. We had this exact pair in our own code and replaced both with one factory that takes the flag.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where it's wrong: same shape, different meaning
&lt;/h3&gt;

&lt;p&gt;The second exact group is a false positive:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;isInstantLike&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;unknown&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="nx"&gt;InstantLike&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;object&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
        &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
        &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;epochMilliseconds&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
        &lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;epochMilliseconds&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;number&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
        &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;timeZone&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
        &lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;timeZone&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;string&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;isFieldRow&lt;/code&gt; in the forms code checks &lt;code&gt;fieldName&lt;/code&gt; and &lt;code&gt;fieldType&lt;/code&gt; in the same pattern. Every type guard over an object with two fields looks like this. Once names and strings are gone, nothing is left that tells them apart, and merging them would make no sense. We hit the same thing in our own code: a type guard for date values and one for form field rows, flagged as one group.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where it misses: same logic, different shape
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;orderTotal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;OrderLine&lt;/span&gt;&lt;span class="p"&gt;[]):&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;line&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;gross&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;unitPrice&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;quantity&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;discount&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;discountPercent&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;gross&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;discountPercent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="nx"&gt;total&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nx"&gt;gross&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;discount&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;total&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;invoiceTotal&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;items&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;InvoiceItem&lt;/span&gt;&lt;span class="p"&gt;[]):&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;sum&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;items&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reduce&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;acc&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;price&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;count&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;rebate&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sum&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both compute the same total. One uses a loop, the other &lt;code&gt;reduce&lt;/code&gt;, and the static finder doesn't see a group even in the fuzzy pass. A fingerprint only catches copies, and a second person solving the same problem rarely writes a copy.&lt;/p&gt;

&lt;h3&gt;
  
  
  A gap I found while writing this
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;escapeHtml&lt;/code&gt; and &lt;code&gt;escapeXml&lt;/code&gt; should be an exact group: the only difference is &lt;code&gt;&amp;amp;#39;&lt;/code&gt; against &lt;code&gt;&amp;amp;apos;&lt;/code&gt;. They only show up in the fuzzy pass, at 0.87. Here is what the normalizer makes of them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;escapeXml&lt;/span&gt;  &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;_&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&amp;amp;/&lt;/span&gt;&lt;span class="nx"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&amp;lt;/&lt;/span&gt;&lt;span class="nx"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&amp;gt;/&lt;/span&gt;&lt;span class="nx"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/#&amp;amp;_;#&amp;amp;_;"&lt;/span&gt;&lt;span class="se"&gt;)&lt;/span&gt;&lt;span class="sr"&gt;;&lt;/span&gt;&lt;span class="err"&gt;}
&lt;/span&gt;&lt;span class="nx"&gt;escapeHtml&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;_&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&amp;amp;/&lt;/span&gt;&lt;span class="nx"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&amp;lt;/&lt;/span&gt;&lt;span class="nx"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&amp;gt;/&lt;/span&gt;&lt;span class="nx"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;_&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/#&amp;amp;_;#&amp;amp;##;"&lt;/span&gt;&lt;span class="se"&gt;)&lt;/span&gt;&lt;span class="sr"&gt;;&lt;/span&gt;&lt;span class="err"&gt;}
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;"&lt;/code&gt; inside the regex literal &lt;code&gt;/"/g&lt;/code&gt; pairs with the quote of the next string, so the string regexes cut across the code. The &lt;code&gt;39&lt;/code&gt; in &lt;code&gt;&amp;amp;#39;&lt;/code&gt; then ends up outside a string and becomes &lt;code&gt;#&lt;/code&gt;. Our production check had the same normalizer and the same gap. While writing this post we switched it to the tokens of the already parsed syntax tree: string, template and JSX text tokens become &lt;code&gt;#&lt;/code&gt;, regex literals stay as they are, comments are gone. The first run with the new normalizer found 12 more groups across files, real duplicates that had been hidden all along, most of them because the bodies differed only in their comments. The sample here keeps the regex version, because it's shorter to read and it shows the gap.&lt;/p&gt;

&lt;h2&gt;
  
  
  The embedding finder: what does the function mean?
&lt;/h2&gt;

&lt;p&gt;The second script sends each function body to &lt;code&gt;nomic-embed-text&lt;/code&gt; through a local Ollama, with comments stripped and whitespace collapsed, and compares every pair by cosine similarity.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`search_document: &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;text&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt; &lt;span class="p"&gt;}),&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;search_document:&lt;/code&gt; prefix is what the model expects for documents.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ bun embedding-duplicates.ts samples --threshold 0.8
0.9881  accounts/disable-account.ts:disableAccount + accounts/enable-account.ts:enableAccount
0.9312  feed/escape-xml.ts:escapeXml + mail/escape-html.ts:escapeHtml
0.9304  reports/correlation.ts:correlation + stats/correlation.ts:pearsonCorrelation
0.8261  invoices/invoice-total.ts:invoiceTotal + orders/order-total.ts:orderTotal
0.8084  cache/redis-client.ts:createCacheClient + db/client.ts:createDbConnection
5 pair(s) at or above 0.8
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  What it's good at: the pair the fingerprint missed
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;orderTotal&lt;/code&gt; and &lt;code&gt;invoiceTotal&lt;/code&gt; are in the list, at 0.83. The model doesn't care about loop against &lt;code&gt;reduce&lt;/code&gt; or &lt;code&gt;discountPercent&lt;/code&gt; against &lt;code&gt;rebate&lt;/code&gt;. The two type guards that fooled the fingerprint score 0.78, just under the cut.&lt;/p&gt;

&lt;p&gt;That margin is the catch. With &lt;code&gt;--threshold 0&lt;/code&gt; the scale gets visible: &lt;code&gt;slugify&lt;/code&gt; and &lt;code&gt;parseDuration&lt;/code&gt;, which have nothing in common, still score 0.67. The real invoice pair is 0.05 above the type guards. At the script's default of 0.9 it drops out, and only the three pairs at 0.93 and above remain. The ranking is mostly sensible, but no single threshold separates real from false, and the scores depend on the model and its version.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where it's wrong: similar words, different code
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;createDbConnection&lt;/code&gt; and &lt;code&gt;createCacheClient&lt;/code&gt; sit at 0.81, almost as high as the real duplicate. Both read a URL from the environment, throw when it's missing, parse a number and build a client. One returns a Postgres pool with a listener connection, the other a Redis client with an error handler. There is nothing to extract.&lt;/p&gt;

&lt;p&gt;We saw this at a larger scale. Three database connection factories in our code scored 0.9961 and 1.0 against each other, and they return different shapes on different layers. Seven queries on the same JSON column clustered together because they share identifiers like the column and table name, not logic.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where it's dangerous: almost the same
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;pearsonCorrelation&lt;/code&gt; and &lt;code&gt;correlation&lt;/code&gt; score 0.93, higher than the invoice pair. They are the same formula, with one difference: &lt;code&gt;pearsonCorrelation&lt;/code&gt; returns 0 when a series has no variance, &lt;code&gt;correlation&lt;/code&gt; returns &lt;code&gt;NaN&lt;/code&gt;. An embedding can't see a missing early return. Merge them, and one caller changes behavior.&lt;/p&gt;

&lt;p&gt;We had this one for real. The correlation helper in a test suite stayed as it was, because the caller's tolerance check depends on getting &lt;code&gt;NaN&lt;/code&gt;. Another pair stayed because one of them sorts its input and the other doesn't, and a later step relies on the order.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we put the two together
&lt;/h2&gt;

&lt;p&gt;For each pair, the question is which finders saw it. &lt;code&gt;combine.ts&lt;/code&gt; runs both and sorts the pairs by that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ bun combine.ts samples
both           ast 1.00  emb 0.99  accounts/disable-account.ts:disableAccount + accounts/enable-account.ts:enableAccount
               -&amp;gt; extract one helper
both           ast 0.87  emb 0.93  feed/escape-xml.ts:escapeXml + mail/escape-html.ts:escapeHtml
               -&amp;gt; extract one helper
ast-only       ast 1.00  emb   -   forms/is-field-row.ts:isFieldRow + scheduling/is-instant-like.ts:isInstantLike
               -&amp;gt; same shape: check that both mean the same thing
embedding-only ast   -   emb 0.93  reports/correlation.ts:correlation + stats/correlation.ts:pearsonCorrelation
               -&amp;gt; same intent: run both on the same inputs before merging
embedding-only ast   -   emb 0.83  invoices/invoice-total.ts:invoiceTotal + orders/order-total.ts:orderTotal
               -&amp;gt; same intent: run both on the same inputs before merging
embedding-only ast   -   emb 0.81  cache/redis-client.ts:createCacheClient + db/client.ts:createDbConnection
               -&amp;gt; same intent: run both on the same inputs before merging
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A pair both finders report is the easy case. A pair only the static finder reports has the same shape, and someone has to check whether both functions are for the same thing. A pair only the embedding reports looks like the same idea, and someone has to check that the two functions behave the same. For that last group, &lt;code&gt;check-equivalence.ts&lt;/code&gt; feeds both functions the same inputs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ bun check-equivalence.ts
correlation [1,2,3,4] [2,4,6,8]: stats 1, reports 1
correlation [1,2,3,4] [4,3,2,1]: stats -1, reports -1
correlation [5,5,5,5] [1,2,3,4]: stats 0, reports NaN
total [{"price":19.99,"count":3}]: orders 59.97, invoices 59.97
total [{"price":19.99,"count":3,"rebate":15},{"price":4.5,"count":3}]: orders 64.47, invoices 64.47
total []: orders 0, invoices 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The totals agree on these three inputs, which makes the merge worth trying. The correlation differs on the third input. The connection factories don't get this far, because they don't even return the same type.&lt;/p&gt;

&lt;p&gt;In our repos the combining is less automatic than this script. Both finders read one shared ignore list for deliberate parallels, such as plugins that implement the same interface. For the static check, a function can also opt out with a &lt;code&gt;// guard:dup-ok&lt;/code&gt; comment. The static check runs after every review pass over merged pull requests and compares against a baseline: known groups pass, a new group becomes a finding in our review queue, and the baseline only ever goes down. Over three cleanup steps it went from 49 to 22, 3 and finally 0. The sample has a simpler version of the same idea behind &lt;code&gt;--baseline&lt;/code&gt; and exits with 1 on a new group, so you can also put it in CI.&lt;/p&gt;

&lt;p&gt;The embedding finder doesn't run automatically, because it needs Ollama. We run it by hand before a cleanup round and judge the pairs in bands. In one round, the band between 0.90 and 0.95 had 77 clusters across repos: 9 real duplicates, 23 partly, 13 already on our list, and 32 that weren't duplicates at all. A pair that both finders reported became one checklist item, which is what &lt;code&gt;combine.ts&lt;/code&gt; does automatically here.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we haven't solved
&lt;/h2&gt;

&lt;p&gt;The embedding band between about 0.8 and 0.95 mixes real duplicates with false positives, and the sample shows it on a small scale: the real invoice pair at 0.83 sits right next to the connection factories at 0.81. A threshold can't separate them. A few things we're considering:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Embed the signature and return type along with the body, so a function returning a Redis client and one returning a pool drift apart.&lt;/li&gt;
&lt;li&gt;Have a language model judge the pairs in the band instead of a person. That costs money for every pair, and we'd need to measure how often it's wrong.&lt;/li&gt;
&lt;li&gt;Make the equivalence check automatic: generate inputs from the parameter types and compare the outputs, instead of writing &lt;code&gt;check-equivalence.ts&lt;/code&gt; by hand for each pair.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you run duplicate detection on a real codebase: what did you add to get the false positives down? And has anyone automated the "do these two actually behave the same" step in a way you'd trust before a merge?&lt;/p&gt;

</description>
      <category>typescript</category>
      <category>ai</category>
      <category>refactoring</category>
      <category>testing</category>
    </item>
    <item>
      <title>Five models built the same app for $0.06 to $0.45. Most of the red was our fault</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Mon, 02 Nov 2026 07:45:00 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/five-models-built-the-same-app-for-006-to-045-most-of-the-red-was-our-fault-240a</link>
      <guid>https://dev.to/marc_kumiko/five-models-built-the-same-app-for-006-to-045-most-of-the-red-was-our-fault-240a</guid>
      <description>&lt;p&gt;We're building an app designer on Kumiko: you describe the app you need, a model designs it, and you get a running app back. The site is up at studio.kumiko.rocks, but the designer isn't open yet. Before it opens, I'm measuring how well it works on one case: a maintenance app for a metalworking shop, with machines, maintenance plans, inspection protocols with measured values, reminder mails, a PDF per protocol and an audit export. Each run has three steps. The model builds the app, then handles two change requests: "calibration every 6 months instead of 12" and "photos on the inspection protocol".&lt;/p&gt;

&lt;p&gt;The model doesn't write the app code. It submits a spec as JSON, and a generator turns it into entities, screens, navigation, translations, seed data and an integration test. The model only writes the bodies of custom handlers in TypeScript. After each submission a check runs: typecheck, boot, a dev-server smoke test in a browser, and a comparison against the design we approved for this case. That comparison has 63 points (entities, screens, actions, read and write rights per role). Every open point goes back to the model as a finding.&lt;/p&gt;

&lt;h2&gt;
  
  
  Green, and the app didn't work
&lt;/h2&gt;

&lt;p&gt;The first run in this mode, on Sonnet 5.5, passed all three steps for $1.16, with 63 of 63 points. Then the model listed what it had left out. The PDF actions only confirmed and never produced a file. A measurement outside its limits wasn't flagged. "Continued operation: no" didn't lock the machine. There were no overdue counters and no reminder mail. The 63 points only checked structure, and a maintenance manager would have found the gaps on the first day.&lt;/p&gt;

&lt;p&gt;So every check now also runs seven behavior tests taken from the approved design: deadlines, the limit check, the lock, the condition that must be written down when a machine keeps running with one, the reminder, and two PDF tests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Five models
&lt;/h2&gt;

&lt;p&gt;Same case, same prompt, same tests. The OpenAI models ran through OpenRouter's EU endpoint.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Build&lt;/th&gt;
&lt;th&gt;Change 1&lt;/th&gt;
&lt;th&gt;Change 2&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Sonnet 5.5&lt;/td&gt;
&lt;td&gt;red, 4/7, hit the check limit (6)&lt;/td&gt;
&lt;td&gt;green, 7/7&lt;/td&gt;
&lt;td&gt;green, 7/7&lt;/td&gt;
&lt;td&gt;$0.454&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-6.1-terra&lt;/td&gt;
&lt;td&gt;red, 6/7 after 5 checks&lt;/td&gt;
&lt;td&gt;red, 6/7&lt;/td&gt;
&lt;td&gt;red, 6/7&lt;/td&gt;
&lt;td&gt;$0.181&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-6.1-sol&lt;/td&gt;
&lt;td&gt;red, 3/7, hit the check limit (7)&lt;/td&gt;
&lt;td&gt;red, 3/7&lt;/td&gt;
&lt;td&gt;red, 3/7&lt;/td&gt;
&lt;td&gt;$0.342&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-6-luna&lt;/td&gt;
&lt;td&gt;red, hit the round limit (25)&lt;/td&gt;
&lt;td&gt;red, 0/7&lt;/td&gt;
&lt;td&gt;red, 0/7&lt;/td&gt;
&lt;td&gt;$0.056&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3.8-27b&lt;/td&gt;
&lt;td&gt;no answer to the first call&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The easy reading is that Sonnet wins and luna is cheap and useless. When I went through the findings one by one, most of the red came from our side.&lt;/p&gt;

&lt;h2&gt;
  
  
  What was ours
&lt;/h2&gt;

&lt;p&gt;The limit check failed for every model. A measurement outside min and max should give &lt;code&gt;notOk&lt;/code&gt;, and the test got &lt;code&gt;undefined&lt;/code&gt;. The model saw only the assertion text, with nothing pointing to the spec construct for exactly this case (&lt;code&gt;derived inRange&lt;/code&gt;). terra got everything else right and then stopped trying. Sonnet only fixed it during the first change request.&lt;/p&gt;

&lt;p&gt;The behavior tests create records with fixed values, like &lt;code&gt;asset.kind: "machine"&lt;/code&gt; and &lt;code&gt;continuedOperation: "withCondition"&lt;/code&gt;, and assume every other field is optional. None of that was in the prompt. sol made the inspector and the inspection date required and called the condition value &lt;code&gt;conditional&lt;/code&gt;, so the test could never create a protocol, and every test that needed one failed with it. luna had no &lt;code&gt;machine&lt;/code&gt; among the asset kinds and got 0 of 7 in both change requests.&lt;/p&gt;

&lt;p&gt;Two boot rules cost every model one or two checks. One fires when a list has an edit screen behind it but no action to open it. Our generator only looked for detail screens, so it never added that action to the maintenance plan list. The model got a boot error about code it hadn't written.&lt;/p&gt;

&lt;p&gt;Some of it was the models. Most of the 21 type errors across all runs were luna's, from &lt;code&gt;new Date()&lt;/code&gt; called on a date-only value in its own handler code. Qwen never answered: 20 minutes of silence with one provider, and the first call ran into the 10-minute limit twice with another, while short test calls came back in 1.6 seconds. I don't know yet whether the provider, the thinking phase or the 57k-token request is the cause.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fixes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The generator adds the open action itself and counts card columns with the framework's own function, so generated code can't trip either boot rule anymore.&lt;/li&gt;
&lt;li&gt;A failing behavior test now names the spec construct behind it: limit check to &lt;code&gt;derived inRange&lt;/code&gt;, lock to &lt;code&gt;approvals&lt;/code&gt; with finding and cascade, condition to &lt;code&gt;requiredWhen&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The prompt lists the test records with their fields and values.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Sonnet again
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Step&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;th&gt;Rounds&lt;/th&gt;
&lt;th&gt;Checks&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Build&lt;/td&gt;
&lt;td&gt;green, 63/63, 7/7&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;$0.247&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Change 1&lt;/td&gt;
&lt;td&gt;green, 63/63, 7/7&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;$0.043&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Change 2&lt;/td&gt;
&lt;td&gt;green, 63/63, 7/7&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;$0.032&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;$0.322 for all three steps, against $0.454 and a red build before. No boot errors, and the limit check was right in the first submission. The first two checks of the build still failed. The audit PDF action and the managers' write rights were missing, as they have been in almost every run since the first one. The condition test failed until the finding pointed at &lt;code&gt;requiredWhen&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;I haven't rerun the other four models yet. That's part 2.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this run doesn't prove
&lt;/h2&gt;

&lt;p&gt;The findings now name the construct, and for the limit check that's close to handing over the answer. These hints come from the design we approved for this one case. In the real designer nobody has approved a design behind the user's description, so nobody can write that hint. The second run shows that a spec, a check and precise findings get to green. It doesn't show how well a model gets the domain logic right on its own. Prompt and finding texts also changed between the two runs, so the comparison with the first table only goes so far.&lt;/p&gt;

&lt;h2&gt;
  
  
  An open question
&lt;/h2&gt;

&lt;p&gt;How specific do you make the feedback for an agent in a test loop? "Expected notOk, got undefined" left every model stuck. "Use derived inRange" fixed it at once, and it's almost the solution. If you run agents against tests: where did you draw that line, and how do you check that your tests aren't just asking for your own solution back?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>typescript</category>
      <category>testing</category>
    </item>
    <item>
      <title>Our AI assistant saw every feature as a tool. That cost $0.148 per turn</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Fri, 30 Oct 2026 10:34:49 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/our-ai-assistant-saw-every-feature-as-a-tool-that-cost-0148-per-turn-4gh7</link>
      <guid>https://dev.to/marc_kumiko/our-ai-assistant-saw-every-feature-as-a-tool-that-cost-0148-per-turn-4gh7</guid>
      <description>&lt;p&gt;Every app we build on Kumiko comes with an AI assistant. It can do what the logged-in user can do: create a campaign, look up a contact, edit a mail template. The framework turns each handler the user's role can reach into a tool, and the assistant got the whole list on every turn.&lt;/p&gt;

&lt;p&gt;In offlot, our marketing app, a tenant admin has 187 tools. I measured what goes into the prompt before the user types a word: about 418k characters. 143k of that was tool JSON. Another 260k was the app manifest, which tells the model what the app looks like (entities, screens, navigation, handlers). The handlers section alone was 154k characters, and it described the same handlers the tool list already described. We were sending most actions twice.&lt;/p&gt;

&lt;p&gt;About 60% of the 187 tools came from platform and admin features rather than from the marketing domain. Someone asking how many contacts signed up this week paid for all of them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four tools and a directory
&lt;/h2&gt;

&lt;p&gt;The model now gets four tools: &lt;code&gt;navigate&lt;/code&gt;, &lt;code&gt;ask_user&lt;/code&gt;, &lt;code&gt;describe_action&lt;/code&gt; and &lt;code&gt;run_action&lt;/code&gt;. The actions the role can use go into the system prompt as a directory, one line per action, with a summary capped at 160 characters. To use an action, the model calls &lt;code&gt;describe_action&lt;/code&gt; for the input schema and then &lt;code&gt;run_action&lt;/code&gt; with the name and the input.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;run_action&lt;/code&gt; doesn't execute anything itself. Right after the provider response we rewrite it into the call it names, so it takes the same path as before: dispatch, Zod validation, the approval step for risky writes. When a conversation is replayed, the history gets rewritten back, so the model sees the calls the way it made them. An unknown name and a name the role may not use get the same reply ("No action with this name is available"), so the assistant can't be used to probe for actions outside the role.&lt;/p&gt;

&lt;p&gt;The manifest lost its handlers section, since the directory covers it now. On the framework side, the manifest also lists only the features and entities the role can reach.&lt;/p&gt;

&lt;p&gt;We built a third mode for comparison, lazy, where the model sees less up front and loads action details on demand.&lt;/p&gt;

&lt;h2&gt;
  
  
  The run
&lt;/h2&gt;

&lt;p&gt;The same 11 end-to-end specs against offlot in all three modes, on Sonnet 5.5, with real API calls:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th&gt;Prompt prefix&lt;/th&gt;
&lt;th&gt;Cost per turn&lt;/th&gt;
&lt;th&gt;Model calls per turn&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Full tool catalog&lt;/td&gt;
&lt;td&gt;203k tokens&lt;/td&gt;
&lt;td&gt;$0.148&lt;/td&gt;
&lt;td&gt;1.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Directory&lt;/td&gt;
&lt;td&gt;85k tokens&lt;/td&gt;
&lt;td&gt;$0.062&lt;/td&gt;
&lt;td&gt;2.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lazy&lt;/td&gt;
&lt;td&gt;79k tokens&lt;/td&gt;
&lt;td&gt;$0.061&lt;/td&gt;
&lt;td&gt;2.6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All three modes passed 11 of 11. The directory costs 58% less per turn. It needs more model calls, because the model has to ask for a schema before it can use an action, but each call is cheap once the prefix is smaller and cached.&lt;/p&gt;

&lt;p&gt;Lazy saved another 6k tokens and a tenth of a cent per turn and needed half a call more for it. Every extra round is latency the user sits through, so we dropped lazy. The directory is now the only mode, in all our apps.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I got wrong
&lt;/h2&gt;

&lt;p&gt;Before the run I estimated the directory prefix at about 8k tokens. It came out at 85k. The tools were the obvious bulk, and I assumed the rest was small. The system rules, the directory itself and the rest of the manifest (entities, screens, navigation) add up, and about half of the 85k is still manifest.&lt;/p&gt;

&lt;p&gt;The measurement also cost more than it had to: $10.79 in total. Two runs went to the wrong model. In the first, the model setting only reached the demo tenant, and that run cost $3.90 on the old Sonnet. Then I found that our dev server ignores the environment variable that sets the model in production. By then the run script checked the model name in the first response and stopped the run after $1.10. Next time that check goes in before the first run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why not drop JSON?
&lt;/h2&gt;

&lt;p&gt;JSON is wordy, so a compact text format looked like the next lever. I ran the same manifest through Anthropic's &lt;code&gt;count_tokens&lt;/code&gt; endpoint as JSON and as a compact line format. The difference was 13 to 14%, not enough to justify a custom format that every consumer has to parse.&lt;/p&gt;

&lt;p&gt;The boring content was worth more. Every label went out in German, English and Spanish, although each conversation has one locale. Feature descriptions, written for the docs, went in at full length: 46.6k characters for 77 features. The framework now sends one label in the requested locale and only the first sentence of each feature description, capped at 300 characters. For offlot that takes the descriptions from 46.6k to 12.2k characters. That change shipped in the same release, but I haven't measured it in a full run yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  An open question
&lt;/h2&gt;

&lt;p&gt;The directory is still one line per action in every prompt. At 187 actions that works. If you run an agent with a lot more actions than that: where did a flat list stop working for you, and what replaced it? Grouping by feature, a search tool, a small model that picks the candidates first?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>typescript</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Our framework docs, three models, isolated agents: 99 seconds, 50 minutes, and a timeout</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Mon, 26 Oct 2026 19:53:45 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/our-framework-docs-three-models-isolated-agents-99-seconds-50-minutes-and-a-timeout-4jlf</link>
      <guid>https://dev.to/marc_kumiko/our-framework-docs-three-models-isolated-agents-99-seconds-50-minutes-and-a-timeout-4jlf</guid>
      <description>&lt;p&gt;A week ago we wrote about an eval for the agent docs that ship inside our framework package. A fresh coding agent passed all four tasks, but the run had a hole: the agents inherited our own Claude Code config, so nothing proved that the package docs alone were enough. The fix was to run each agent in isolation. That run is done now. We also gave the first task to two local models to see how far they get on a 48 GB Mac.&lt;/p&gt;

&lt;p&gt;Kumiko is our TypeScript framework for multi-tenant apps on Bun, Hono and Postgres. Since 0.364.0 the npm package contains an &lt;code&gt;AGENTS.md&lt;/code&gt; plus generated indexes of features, widgets and sample apps. The eval checks whether an agent that has never seen the framework source can build features from that.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;Each task gets a fresh app scaffolded from the published packages, version 0.393.0. The agent runs as &lt;code&gt;claude -p&lt;/code&gt; (Claude Code 2.1.310) in the project directory with &lt;code&gt;CLAUDE_CONFIG_DIR&lt;/code&gt; pointing at an empty temp directory, so none of our skills, rules or memory leak in. It sees the app, its &lt;code&gt;node_modules&lt;/code&gt; and the task text. Afterwards a grader copies in a hidden acceptance test and runs eight checks: typecheck, lint, unit tests, boot, schema validation, the acceptance test against a real Postgres, and two guard runs.&lt;/p&gt;

&lt;p&gt;The four tasks: an entity with CRUD and a list screen, custom write handlers, an overview screen with stats, and wiring in the bundled tags feature.&lt;/p&gt;

&lt;h2&gt;
  
  
  First, the grader was broken
&lt;/h2&gt;

&lt;p&gt;The first isolated Sonnet run failed every task. The agents' code was fine. Our grader had rotted.&lt;/p&gt;

&lt;p&gt;The acceptance tests imported &lt;code&gt;setupAppTestStack&lt;/code&gt; and &lt;code&gt;seedTenant&lt;/code&gt; from &lt;code&gt;@cosmicdrift/kumiko-testing&lt;/code&gt;. Those helpers moved to &lt;code&gt;@cosmicdrift/kumiko-dev-server/testing&lt;/code&gt; some releases ago, and the templates never followed. The guard check ran &lt;code&gt;bunx kumiko-guards@0.393.0&lt;/code&gt;, which installs the guards package on its own. That package imports &lt;code&gt;@cosmicdrift/kumiko-types&lt;/code&gt; without declaring it, so it crashed before checking anything. Inside an app the import resolves by luck, because the framework pulls the types package in.&lt;/p&gt;

&lt;p&gt;We fixed the templates, made the grader run the guards the project already has installed, and declared the missing dependency (kumiko-framework#4074). All results below come from that fixed grader.&lt;/p&gt;

&lt;p&gt;The funny part came from Sonnet's own report on task 02. Each agent ends with a list of things the docs didn't tell it. Sonnet wrote that &lt;code&gt;AGENTS.md&lt;/code&gt; names the wrong package for &lt;code&gt;setupAppTestStack&lt;/code&gt; and &lt;code&gt;seedTenant&lt;/code&gt;. It was right: our docs had the same stale import as our grader. The agent noticed, copied the working import from the scaffolded test, and moved on. That line is fixed in kumiko-framework#4075.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sonnet 5.5: four out of four
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Checks&lt;/th&gt;
&lt;th&gt;Wall time&lt;/th&gt;
&lt;th&gt;Responses / tool calls&lt;/th&gt;
&lt;th&gt;Median response&lt;/th&gt;
&lt;th&gt;p90 response&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;01 entity CRUD&lt;/td&gt;
&lt;td&gt;8/8&lt;/td&gt;
&lt;td&gt;99 s&lt;/td&gt;
&lt;td&gt;13 / 12&lt;/td&gt;
&lt;td&gt;4.0 s&lt;/td&gt;
&lt;td&gt;7.9 s&lt;/td&gt;
&lt;td&gt;$0.24&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02 custom handlers&lt;/td&gt;
&lt;td&gt;8/8&lt;/td&gt;
&lt;td&gt;60 s&lt;/td&gt;
&lt;td&gt;8 / 7&lt;/td&gt;
&lt;td&gt;3.6 s&lt;/td&gt;
&lt;td&gt;12.2 s&lt;/td&gt;
&lt;td&gt;$0.22&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;03 overview screen&lt;/td&gt;
&lt;td&gt;8/8&lt;/td&gt;
&lt;td&gt;35 s&lt;/td&gt;
&lt;td&gt;4 / 3&lt;/td&gt;
&lt;td&gt;5.8 s&lt;/td&gt;
&lt;td&gt;15.3 s&lt;/td&gt;
&lt;td&gt;$0.15&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;04 bundled tags&lt;/td&gt;
&lt;td&gt;8/8&lt;/td&gt;
&lt;td&gt;73 s&lt;/td&gt;
&lt;td&gt;7 / 6&lt;/td&gt;
&lt;td&gt;6.3 s&lt;/td&gt;
&lt;td&gt;21.4 s&lt;/td&gt;
&lt;td&gt;$0.20&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;"Response time" is measured from the moment the agent gets a prompt or a tool result to the last block of the model's answer, taken from the stream timestamps. All four tasks took under five minutes and $0.82 in total. The grader also scans the diff for patterns our rules forbid. It found one hit, an &lt;code&gt;any&lt;/code&gt; inside a string literal in task 04, which is a false positive.&lt;/p&gt;

&lt;p&gt;So the claim from last week holds with isolation: the package docs are enough for a strong model to finish these four tasks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two local models on task 01
&lt;/h2&gt;

&lt;p&gt;The local runs used the same Claude Code client against a local Anthropic-compatible endpoint. The machine is an M4 Pro with 48 GB of unified memory, of which about 37 GiB is available to the GPU.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Strata 0.2.6 with qwen3.8-flash-next at Q2_0 and 128K context.&lt;/strong&gt; The weights are 66 GB, so the model does not fit in GPU memory at all. We wanted to see whether it runs anyway. It did, at about 4.5 tokens per second for output and 9 tokens per second for reading the prompt. Every request carried 39k to 43k tokens of context, and almost all of that came from the prompt cache; without the cache this run would not have finished in any useful time.&lt;/p&gt;

&lt;p&gt;It passed. All eight checks green, no hits in the rule scan, in 50 minutes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Qwen3.5 9B on Ollama 0.31.2 with 64K context.&lt;/strong&gt; At 6.6 GB it fits entirely on the GPU and answers faster per turn. We gave it a 60-minute cap. It hit the cap with two of eight checks green, and those two were the guards, which pass on an app that barely changed. The app never booted ("Export named 'contactsFeature' not found"). On the way the model wrote feature code into a &lt;code&gt;.gitignore&lt;/code&gt; file and kept rewriting the same files, and Claude Code compacted its context twice.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Checks&lt;/th&gt;
&lt;th&gt;Wall time&lt;/th&gt;
&lt;th&gt;Responses / tool calls&lt;/th&gt;
&lt;th&gt;First response&lt;/th&gt;
&lt;th&gt;Median response&lt;/th&gt;
&lt;th&gt;p90 response&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Sonnet 5.5 (API)&lt;/td&gt;
&lt;td&gt;8/8&lt;/td&gt;
&lt;td&gt;99 s&lt;/td&gt;
&lt;td&gt;13 / 12&lt;/td&gt;
&lt;td&gt;4.5 s&lt;/td&gt;
&lt;td&gt;4.0 s&lt;/td&gt;
&lt;td&gt;7.9 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Strata Q2_0, 66 GB&lt;/td&gt;
&lt;td&gt;8/8&lt;/td&gt;
&lt;td&gt;50 min&lt;/td&gt;
&lt;td&gt;49 / 68&lt;/td&gt;
&lt;td&gt;276 s&lt;/td&gt;
&lt;td&gt;36.8 s&lt;/td&gt;
&lt;td&gt;141 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.5 9B, Ollama&lt;/td&gt;
&lt;td&gt;2/8&lt;/td&gt;
&lt;td&gt;60 min, stopped&lt;/td&gt;
&lt;td&gt;82 / 81&lt;/td&gt;
&lt;td&gt;67 s&lt;/td&gt;
&lt;td&gt;26.5 s&lt;/td&gt;
&lt;td&gt;66.9 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The first response is the slowest one for the local models because the whole system prompt and tool list have to be read once. For Strata that took four and a half minutes; its slowest answer took five.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the numbers don't say
&lt;/h2&gt;

&lt;p&gt;Every model ran each task once. One pass or one failure is an anecdote.&lt;/p&gt;

&lt;p&gt;The local runs had output capped at 4,096 tokens per response. Sonnet had no cap. The context windows differed too: 128K for Strata, 64K for the 9B.&lt;/p&gt;

&lt;p&gt;The local models only got task 01, the simplest one. A 50-minute pass on that says little about tasks 02 to 04.&lt;/p&gt;

&lt;p&gt;Strata's speed depends heavily on the prompt cache. A task that invalidates the cache more often would be much slower.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the docs still miss
&lt;/h2&gt;

&lt;p&gt;Each agent lists what it had to guess. These came up in the isolated runs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;personalData&lt;/code&gt; is described for writes, but not what happens on reads or sorting. Sonnet asked whether a PII field can be a default sort.&lt;/li&gt;
&lt;li&gt;How a screen gets registered in &lt;code&gt;client.tsx&lt;/code&gt; isn't documented.&lt;/li&gt;
&lt;li&gt;What a create result contains, and that a &lt;code&gt;ne&lt;/code&gt; filter excludes NULL rows.&lt;/li&gt;
&lt;li&gt;Which error class fits "this is already done".&lt;/li&gt;
&lt;li&gt;How to wire a tags section into an entity edit screen, and whether &lt;code&gt;entityName&lt;/code&gt; must be set. The tags recipe also uses the framework's internal test setup instead of the app's.&lt;/li&gt;
&lt;li&gt;Strata asked which &lt;code&gt;find&lt;/code&gt; level fits and which translation key a row action label uses.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these blocked a pass, which is the uncomfortable part. A strong model fills these gaps by reading compiled code or guessing well, and the eval stays green.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we take from it
&lt;/h2&gt;

&lt;p&gt;The isolated run confirms the docs work for Sonnet. It also showed that our grader and our docs had drifted from the code in the same place, and the agent caught it before we did. An eval that nobody runs for a week rots like any other code.&lt;/p&gt;

&lt;p&gt;For local models, a heavily quantized model that doesn't fit in memory finished task 01 correctly, and a small model that fits easily didn't. In this one run, model size mattered more than speed.&lt;/p&gt;

&lt;p&gt;If you've run coding agents on local models against a real codebase: did a bigger model at 2 bits beat a smaller one at 4 or 8 bits for you, or was our run a fluke?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>typescript</category>
      <category>llm</category>
    </item>
    <item>
      <title>Changing a mail template used to need a deploy. We built an editor that refuses to publish a broken one</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Sat, 24 Oct 2026 20:50:03 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/changing-a-mail-template-used-to-need-a-deploy-we-built-an-editor-that-refuses-to-publish-a-broken-281f</link>
      <guid>https://dev.to/marc_kumiko/changing-a-mail-template-used-to-need-a-deploy-we-built-an-editor-that-refuses-to-publish-a-broken-281f</guid>
      <description>&lt;p&gt;Until recently, changing the wording of a password-reset mail in our framework meant editing a TypeScript file and shipping a release. The same went for the PDF a rental app sends as an invoice. Nobody on the operations side could touch either one.&lt;/p&gt;

&lt;p&gt;Looking at why, we found two mail systems that did not know about each other. One stored the mail as a JSON list of sections (text, button) in code, with the branding hardcoded next to it. The other stored a Markdown document in the database and rendered it through MJML, and exactly one app used it. PDFs were a third thing: HTML built in code and printed with Puppeteer. The tenant's logo and colors lived in two places, the static mail branding in app code and the branding tokens per tenant. The mail message type had no concept of attachments or inline images at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  One model for mail and PDF
&lt;/h2&gt;

&lt;p&gt;Every mail and every PDF is now a template: a slug, a locale, a document, and a small JSON meta object. The document is structured content that tiptap edits in the browser. The server rebuilds it through a sanitizer, so only known node types survive a save. Meta is validated per kind: a mail needs a subject and may have preview text, a layout, a category and a sender name. A PDF has page format, margins and a file name. Images and files are linked resources, uploaded once and referenced by key.&lt;/p&gt;

&lt;p&gt;Layout is a separate template. A mail layout (MJML) and a PDF letterhead each have exactly one required slot for the content, and they read the tenant's branding tokens instead of carrying their own copy. Changing the layout changes every mail that uses it, without touching any sender.&lt;/p&gt;

&lt;p&gt;Features no longer send sections. They declare which variables a mail needs and seed a default template at boot. This is the declaration for the waitlist notice in the framework:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;WAITLIST_ADMIN_NOTICE_VARIABLES&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;TemplateVariables&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;templateVariable&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}),&lt;/span&gt;
  &lt;span class="na"&gt;details&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;templateVariable&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;templateVariable&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The variable types are text, link, money, date and list (the schema also knows rich text and percent). Each one carries a label and an example value, and the example is what the preview fills in.&lt;/p&gt;

&lt;h2&gt;
  
  
  Scopes and the four-step lookup
&lt;/h2&gt;

&lt;p&gt;A template can exist at two scopes: the system default that the feature ships, and a tenant customisation. When a handler asks for a template, the resolver walks four steps: the tenant in the requested locale, the system in the requested locale, the tenant in the fallback locale, the system in the fallback locale. A tenant overrides a default by having its own row, and no application code changes.&lt;/p&gt;

&lt;p&gt;Who may edit sits outside the model. By default the platform admin edits the system templates, and an app can open individual templates to its tenant admins. Mails that must stay as shipped can be marked system only, and tenants neither see nor override them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The editor
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj6g314yva60uz3s8j5ca.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj6g314yva60uz3s8j5ca.png" alt="Template overview with kind, origin, languages, status and where each template is used" width="799" height="512"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The overview lists mails, PDFs and layouts in one table. Each row shows where the template comes from (system or tenant customisation), which languages exist as filled chips, the status with its version, and which feature uses it. The usage column comes from the feature registry, so it is read from the code that sends the mail and nobody types it in.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faw4ary40qiairby8o01h.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faw4ary40qiairby8o01h.png" alt="Mail editor with the placeholder rail on the left, the paper in the middle and a blocked publish button below" width="799" height="512"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The mail editor has the placeholder rail on the left with type, required marker and a check for whether the placeholder is used. The middle holds subject, preview text and the content with placeholders as chips and blocks for buttons, images and repeated rows. The right side has layout, sender, category and attachments. In the screenshot, a required placeholder, the due date, appears nowhere in the subject, preview text or content. The footer says so and the publish button is disabled. Saving a draft still works.&lt;/p&gt;

&lt;p&gt;That rule is enforced on the server and not only in the editor. Saving rejects unknown placeholders. Saving an active template and publishing both check that every required variable is used. The editor only shows what the server would say.&lt;/p&gt;

&lt;p&gt;A draft over an active template does not replace it. The active version keeps being served while the edit sits beside it, and publish switches over. The first version got this wrong: a draft saved over an active tenant template overwrote the same row, the resolver fell back to the system text, and the tenant's active wording was gone. The draft now lives in its own column, and publish promotes it.&lt;/p&gt;

&lt;h2&gt;
  
  
  PDFs
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa2esfol9e0f7pzbjfjte.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa2esfol9e0f7pzbjfjte.png" alt="PDF template editor with the rendered preview next to page format, margins, file name and PDF metadata" width="799" height="512"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;PDFs use the same document model, rendered through Puppeteer in a separate service that runs Chromium behind an egress guard. The preview is the real render. It shows the letterhead from the layout, the repeating rows from a list variable and the page count, and the footer line reports the last test render, here 2 pages, 67 KB, 2.3 seconds. Page format, margins, the file name pattern and the PDF metadata sit in the side panel. A generated PDF is stored with the template id and version it came from, so you can say later which wording a customer received.&lt;/p&gt;

&lt;p&gt;Without a running Puppeteer service the PDF editor is locked, because a preview that cannot render would only mislead.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is not in this release
&lt;/h2&gt;

&lt;p&gt;Everything above is in the open framework, including the editors, the preview and the test mail. The AI assistant that edits templates from a sentence ("put the logo in the sign-up mail") is a separate piece that sits in our enterprise layer. It reads a template as a list of blocks and patches the draft through structured operations, and it never writes raw HTML. It shows the result as a preview first and saves it as a draft. Publishing stays a decision of the person using it. The same validation applies, so a suggestion that removes a required placeholder is rejected. Previewing a template with a real record is part of that layer too. This post is about the open part, so we only mention the assistant here.&lt;/p&gt;

&lt;p&gt;Also still missing: version history with restore, columns, PDFs without Puppeteer, and an editor control for conditional blocks. The renderer already shows or hides a paragraph or table row depending on whether a variable has a value; the editor just has no button for it yet.&lt;/p&gt;

&lt;p&gt;If you let non-developers edit transactional mail, what did you put between them and a broken invoice mail? We block publishing on a missing required placeholder and keep the active version served. We would like to hear what else you check before a template goes live.&lt;/p&gt;

</description>
      <category>typescript</category>
      <category>opensource</category>
      <category>webdev</category>
      <category>pdf</category>
    </item>
    <item>
      <title>Our comment section redesigned our agent permissions. Here's what we still can't solve</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Wed, 21 Oct 2026 07:08:14 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/our-comment-section-redesigned-our-agent-permissions-heres-what-we-still-cant-solve-236m</link>
      <guid>https://dev.to/marc_kumiko/our-comment-section-redesigned-our-agent-permissions-heres-what-we-still-cant-solve-236m</guid>
      <description>&lt;p&gt;In September I wrote about AI agents in our framework, Kumiko, and why a handler without a description never shows up in the agent's tool list: &lt;a href="https://dev.to/marc_kumiko/undocumented-handlers-dont-exist-to-the-agent-e71"&gt;Undocumented handlers don't exist to the agent&lt;/a&gt;. Two readers, phongdesigns and anp2network, went through the post more carefully than I had, and the thread under it ran to eleven comments over three days.&lt;/p&gt;

&lt;p&gt;Three changes to how our agent decides what it may do came out of that thread. We turned down two of the ideas, and there are three questions we haven't answered. This post covers all of it, because the open questions are the part I'd like help with.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup in two sentences
&lt;/h2&gt;

&lt;p&gt;Every write handler in a Kumiko app can carry an agent risk (&lt;code&gt;low&lt;/code&gt;, &lt;code&gt;mid&lt;/code&gt; or &lt;code&gt;high&lt;/code&gt;), and the user can set mid-risk tools to "always allow" while high-risk tools ask every time. Back then, the risk was whatever the handler declared, plus two tests that pinned the risk of specific handlers by name.&lt;/p&gt;

&lt;h2&gt;
  
  
  Change 1: the risk comes from what the handler does
&lt;/h2&gt;

&lt;p&gt;anp2network pointed out that pinning by name doesn't catch the hard delete someone adds in six months. It defaults to mid, and nothing goes red. Their suggestion was to watch what a handler actually does and compare that with what it declares.&lt;/p&gt;

&lt;p&gt;That fit our code better than I expected. A hard delete in Kumiko doesn't go through the normal event append. It calls a separate forget operation on the event store, so the framework can see it. Since 0.322.0, a handler that runs forget, or a hard delete on an entity without soft delete, has to resolve to &lt;code&gt;high&lt;/code&gt;. Otherwise the executor denies the call before it reads anything. The check looks at the handler the caller dispatched directly, so a mid-risk handler can't get around it by delegating to a high-risk one internally. Standard delete handlers on those entities default to &lt;code&gt;high&lt;/code&gt;, and declaring one lower fails at boot.&lt;/p&gt;

&lt;p&gt;My follow-up comment listed after-commit hooks, jobs and event consumers as a gap. Since then, writes dispatched inside those contexts go through the same check, with a short list of deliberate exceptions such as cron cleanups and pipelines started by a high-risk handler.&lt;/p&gt;

&lt;h2&gt;
  
  
  Change 2: the tool list and the dispatch table are tested against each other
&lt;/h2&gt;

&lt;p&gt;I had claimed that the model can't call a tool that isn't in its list, because the list and the dispatch table come from the same catalog. anp2network's reply was that this was only true because of how the code happened to be arranged, and no test checked it. Now a test asserts that every tool name has exactly one dispatch entry, and the other way round.&lt;/p&gt;

&lt;h2&gt;
  
  
  Change 3: some fields are dangerous to write because of who reads them
&lt;/h2&gt;

&lt;p&gt;This one I didn't see coming. Our prompt store keeps every revision of a feature's system prompt. Editing a prompt was a high-risk action. Reverting to an older revision was mid, and a revert appends a copy of the old revision, which then becomes the live prompt. A user who set revert to "always allow" had given the agent a way to swap another feature's instructions without asking.&lt;/p&gt;

&lt;p&gt;A floor on the handler can't see that, because on the storage side the revert is an ordinary, reversible write. anp2network put it this way: the risk is a property of the destination. A later run reads that row as instructions.&lt;/p&gt;

&lt;p&gt;Since 0.324.0, a field can say so:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nx"&gt;systemPrompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;createLongTextField&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;personal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;is_business_data&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;readAsInstruction&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}),&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Any create or update that writes such a field needs &lt;code&gt;high&lt;/code&gt; on the directly dispatched handler, through a generic update handler as well. The prompt store's content field carries the flag, so revert is high now.&lt;/p&gt;

&lt;p&gt;We also record how every agent write was approved: no approval needed, an "always" rule, or the user confirming it. "Always" isn't offered for high-risk tools, so the number of flagged writes approved by an "always" rule should be zero, and we can check that with a query.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we turned down
&lt;/h2&gt;

&lt;p&gt;anp2network noticed that a tool the user set to "never" vanishes from the model's view, so to the model "forbidden" looks exactly like "doesn't exist". The suggestion was a tombstone: keep the entry visible and have dispatch refuse it with its own reason. We didn't build it. "Never" is one of several filters that hide tools (roles, mode, an explicit deny list), so tombstones would either be inconsistent with the other filters or leak names those filters exist to hide. anp2network agreed with the reasoning. Instead, the agent's system prompt now tells it to say the action isn't available "rather than approximating it with a similar tool — a similar tool is not the same action."&lt;/p&gt;

&lt;p&gt;The other suggestion was a test that counts the high-risk handlers, so every change to that set shows up in a diff. Once the risk comes from the effect, the count adds little: a new hard delete is high whether anyone updates a number or not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Still open
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Irreversible effects outside our database
&lt;/h3&gt;

&lt;p&gt;The floor sees forget and hard deletes. It doesn't see a mail, a webhook or a payment. In our store, the write that queues a mail looks like any other reversible write, but the mail can't be unsent. Our webhooks now send an idempotency key, so the receiver can drop a redelivery, but that only protects against duplicates. Nothing decides whether the agent should trigger the first one without asking.&lt;/p&gt;

&lt;p&gt;Our half idea is to treat outbound channels like flagged fields: the channel declares itself irreversible, and any handler that hands work to it resolves to high. We haven't built that. I'm not sure the line is right either, because a status mail to the user themselves and a payment to a third party would land in the same class.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Who reads a field as instructions
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;readAsInstruction&lt;/code&gt; flag is set by hand next to the field, apart from the code that later puts the field into a prompt. If a new feature starts feeding an unflagged field into a system prompt, the flag set is stale and every check stays green. anp2network asked whether anything fails when that happens. Nothing does, and on the agent side the system prompt is still a plain string.&lt;/p&gt;

&lt;p&gt;The flag also has holes on the write side. A hand-written domain event whose projection writes the flagged field skips the gate. The gate checks whether the payload contains the field, not whether the value changed, which we chose deliberately so it fails closed. Reverting a prompt also doesn't undo a run that already read the bad version.&lt;/p&gt;

&lt;p&gt;Our half idea is a type. The system prompt parameter would only accept text that came from a flagged field or from code, so feeding it anything else wouldn't compile. I haven't checked whether that survives prompts assembled from several pieces.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Refusing instead of reaching for the neighbour
&lt;/h3&gt;

&lt;p&gt;When the model can't find a tool for what the user wants, it should say so. The risk is that it picks the closest-sounding tool instead, maybe one the user set to "always allow". Right now the only defence is the sentence in the system prompt quoted above.&lt;/p&gt;

&lt;p&gt;anp2network suggested a number: after a rejected call, record what the model calls next. In our loop, calling a tool that isn't in the list ends the run, so there's no "next" to count. The case we can't see at all is the model going straight to the neighbour without ever trying the missing tool.&lt;/p&gt;

&lt;p&gt;phongdesigns added the near-duplicate problem: two writes described in almost the same words, and the model confuses them. Our manifest snapshot still records only handler names, so a reworded description doesn't even show up in review. I said in the thread that adding descriptions and risk to it was the next step, and it still is.&lt;/p&gt;

&lt;p&gt;The only test I can think of for substitution is to give a real model pairs of similar tools and see where it goes wrong. That costs money on every run, and the result changes with the model.&lt;/p&gt;

&lt;p&gt;How do you test that an agent refuses instead of improvising, without paying for a model call on every pull request?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>security</category>
      <category>typescript</category>
      <category>discuss</category>
    </item>
    <item>
      <title>A fresh coding agent passed all four tasks against our framework docs. Here's what that result doesn't prove</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Mon, 19 Oct 2026 21:09:58 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/a-fresh-coding-agent-passed-all-four-tasks-against-our-framework-docs-heres-what-that-result-4l58</link>
      <guid>https://dev.to/marc_kumiko/a-fresh-coding-agent-passed-all-four-tasks-against-our-framework-docs-heres-what-that-result-4l58</guid>
      <description>&lt;p&gt;Kumiko is our TypeScript framework for multi-tenant apps on Bun, Hono and Postgres. More and more of the code written against it comes from coding agents, and until last week those agents saw very little of it. An agent working in an app sees the app and its &lt;code&gt;node_modules&lt;/code&gt;. Our framework package shipped &lt;code&gt;dist&lt;/code&gt;, a README and a license. The docs site exists, but it's outside what an agent sees in an app, and the instructions written for agents lived in skills inside our own workspace, so only our own sessions ever read them.&lt;/p&gt;

&lt;p&gt;So we moved the explanation into the package and then checked whether an agent that has never seen the framework source can work from it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What ships in the package now
&lt;/h2&gt;

&lt;p&gt;Since 0.364.0, &lt;code&gt;@cosmicdrift/kumiko-framework&lt;/code&gt; contains an &lt;code&gt;AGENTS.md&lt;/code&gt;, today about 350 lines. It explains the model (features, entities, handlers, screens, events, tenant scoping), lists rules in the form "never this, do that", and has a table of "I need X, use Y". Next to it sits an &lt;code&gt;agents/&lt;/code&gt; folder that the build generates from the source: an index of every bundled feature from the feature manifest, an index of every UI widget export, and one file per tested sample app with its README, source and tests. A drift guard fails CI when the generated files no longer match the code.&lt;/p&gt;

&lt;p&gt;A new app from &lt;code&gt;create-kumiko-app&lt;/code&gt; gets a short &lt;code&gt;AGENTS.md&lt;/code&gt; of its own for other agents and a &lt;code&gt;CLAUDE.md&lt;/code&gt; that imports the package file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;@node_modules/@cosmicdrift/kumiko-framework/AGENTS.md
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The docs always match the installed version and work offline. Boot errors and guard messages also carry a &lt;code&gt;docsUrl&lt;/code&gt; that points at a page for that error code.&lt;/p&gt;

&lt;h2&gt;
  
  
  The eval
&lt;/h2&gt;

&lt;p&gt;We wrote four tasks of the kind a real app needs:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;An entity with a list and a form.&lt;/li&gt;
&lt;li&gt;A write handler and a query handler with integration tests (complete a task, count open tasks per tenant).&lt;/li&gt;
&lt;li&gt;An overview screen built from existing widgets.&lt;/li&gt;
&lt;li&gt;Mounting the bundled &lt;code&gt;tags&lt;/code&gt; feature in an app.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Each task got its own fresh scaffold, installed from npm into a directory outside any workspace. Inside a workspace, bun resolves workspace links instead of the npm tarball, and the agent reads our source instead of the package. The prepare script refuses to run inside a directory that has a &lt;code&gt;package.json&lt;/code&gt; or &lt;code&gt;bun.lock&lt;/code&gt; above it.&lt;/p&gt;

&lt;p&gt;Each project got a fresh agent (Sonnet) with a prompt that contained the environment, the task text and the rules: work only in the project, read only the project and its &lt;code&gt;node_modules&lt;/code&gt;, no skills, no memory, no web search. At the end it had to report in 15 lines what it built, which docs it read and what was missing.&lt;/p&gt;

&lt;p&gt;We didn't grade on that report. A script copies a hidden acceptance test into the project that the agent never saw, then runs typecheck, lint, unit tests, boot validation, schema validation, the integration tests (the agent's own plus the hidden one) and our guards. It also scans the agent's diff for &lt;code&gt;any&lt;/code&gt;, raw SQL writes, inline styles and a test helper we don't allow in integration tests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Result: 4 of 4
&lt;/h2&gt;

&lt;p&gt;All eight checks passed in every project, including the hidden acceptance tests. In task 2 the integration run had 13 tests: 8 written by the agent and 5 from the acceptance file.&lt;/p&gt;

&lt;p&gt;Every task still showed a gap in the docs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;What the agent couldn't find in the docs&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Entity with list and form&lt;/td&gt;
&lt;td&gt;The personal-data setting a write handler needs when it writes personal fields. It found it through the boot error.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Handlers with tests&lt;/td&gt;
&lt;td&gt;How to count. &lt;code&gt;list&lt;/code&gt; with &lt;code&gt;totalCount&lt;/code&gt; and a filter was only in the &lt;code&gt;.d.ts&lt;/code&gt; files.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Overview screen&lt;/td&gt;
&lt;td&gt;Any example of a dashboard screen. It worked out the data contract from the compiled renderer.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mount &lt;code&gt;tags&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;That a newly mounted bundled feature needs a migration, and how to wire it into the client and the test stack.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The boot errors did their job in task 1. The agent made the mistake, the error named the fix, and it applied it. In the other three, the docs alone weren't enough, and the agents filled the gap by reading type declarations and compiled JavaScript in &lt;code&gt;node_modules&lt;/code&gt;. The rules allowed that, and a human developer would do the same. For this eval we count it as a gap anyway, because the next agent might not look there.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why 4 of 4 is weaker than it looks
&lt;/h2&gt;

&lt;p&gt;The task texts fixed names, payloads and IDs, so the hidden tests could find what they test. An open request like "add a way to see how many tasks are left" is harder.&lt;/p&gt;

&lt;p&gt;Task 3 passed on its declaration and its data. Nothing rendered a screen, so we don't know whether the title key resolves.&lt;/p&gt;

&lt;p&gt;The bigger problem is how the agents ran. To save time, we started them as subagents from an existing Claude Code session instead of in a clean CLI with an empty config directory. A subagent inherits the session's user and workspace instructions, and ours state the same rules the review checks: no &lt;code&gt;any&lt;/code&gt;, the real test stack instead of a test dispatcher, no direct writes to event tables. Our results table has a column for rule breaks, and it was empty for all four runs. That says nothing about whether the package docs teach those rules. Only the columns for which docs an agent read and what it missed measure the package docs on their own.&lt;/p&gt;

&lt;p&gt;We checked the transcripts for paths outside the project. None of the four agents touched our source tree or our config folder, and none used a skill or a web search.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the eval found that wasn't documentation
&lt;/h2&gt;

&lt;p&gt;The fresh scaffolds were broken in ways we never noticed, because we never start from a fresh scaffold. The prepare script had to patch some of this before an agent could start (version pins, the &lt;code&gt;.gitignore&lt;/code&gt;, the Biome config, a random master key and admin password in &lt;code&gt;.env&lt;/code&gt;), and none of those patches help with the tasks:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The scaffold wrote no &lt;code&gt;.gitignore&lt;/code&gt;. Biome is configured to read it, so &lt;code&gt;bun run lint&lt;/code&gt; failed in a brand new project.&lt;/li&gt;
&lt;li&gt;Biome was pinned with a caret range that now resolves to a newer minor version. The new project was lint-red on its first run.&lt;/li&gt;
&lt;li&gt;The generated production entry point seeded an admin user with a fixed password. Since 0.365.0 it requires &lt;code&gt;DEMO_ADMIN_PASSWORD&lt;/code&gt; (at least 12 characters) in the env schema, and the boot stops before seeding when it is missing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Upgrading the framework doesn't rewrite your &lt;code&gt;bin/main.ts&lt;/code&gt;. If you scaffolded an app with 0.364.0 or earlier and booted it in production mode, change that admin's password and add the env variable by hand.&lt;/p&gt;

&lt;p&gt;One of our own examples taught the wrong thing. The &lt;code&gt;counter:active&lt;/code&gt; query in the custom-handlers sample loaded a page with &lt;code&gt;list&lt;/code&gt; and filtered it in JavaScript. That only ever sees the first page, and it's the example an agent would copy when it needs to count. It now filters in the query and asks for &lt;code&gt;totalCount&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;While writing the error pages we also found that every &lt;code&gt;docsUrl&lt;/code&gt; led to a 404. The framework built &lt;code&gt;/errors/&amp;lt;code&amp;gt;&lt;/code&gt;, and the docs site served &lt;code&gt;/en/errors/&amp;lt;code&amp;gt;/&lt;/code&gt;. The site now redirects, and a check fails the docs build when an error code has no page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Next step
&lt;/h2&gt;

&lt;p&gt;The run the eval was designed for is still open: one agent per project, started in the project directory with an empty Claude Code config directory, so nothing from our setup leaks in. The README in &lt;code&gt;eval/agent-docs/&lt;/code&gt; describes it, and that run is what tells us whether the rules in &lt;code&gt;AGENTS.md&lt;/code&gt; reach an agent that has only the package.&lt;/p&gt;

&lt;p&gt;The tasks, the hidden acceptance tests and the runner are in the public repo under &lt;code&gt;eval/agent-docs/&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;How do you grade an eval like this? We count "had to read compiled code" as a docs gap even when the task passes. We're not sure that's the right line, or whether it should fail the task.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>typescript</category>
      <category>testing</category>
      <category>documentation</category>
    </item>
    <item>
      <title>Turning on MFA took one line. Making it true took three routes and a test kit</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Thu, 15 Oct 2026 13:38:50 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/turning-on-mfa-took-one-line-making-it-true-took-three-routes-and-a-test-kit-4a5g</link>
      <guid>https://dev.to/marc_kumiko/turning-on-mfa-took-one-line-making-it-true-took-three-routes-and-a-test-kit-4a5g</guid>
      <description>&lt;p&gt;We are preparing Veridom, a property management app built on Kumiko, for a closed launch. One of the launch requirements was simple to state: every admin signs in with a second factor. In Kumiko that is a config value on the MFA feature:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;issuer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;BRAND_NAME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;setupTokenSecret&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;challengeTokenSecret&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;requiredPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;admins&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The policy has three values: &lt;code&gt;optional&lt;/code&gt;, &lt;code&gt;admins&lt;/code&gt; and &lt;code&gt;all&lt;/code&gt;. With &lt;code&gt;admins&lt;/code&gt;, a password login by an admin who has no factor gets a setup step instead of a session, and an admin with a factor gets a challenge. The login handler enforces that.&lt;/p&gt;

&lt;p&gt;The change was one line. Everything after it is what this post is about.&lt;/p&gt;

&lt;h2&gt;
  
  
  89 red tests, one message
&lt;/h2&gt;

&lt;p&gt;The first effect was that the whole end-to-end suite went red. 89 Playwright specs, all failing with 401 on their first API call.&lt;/p&gt;

&lt;p&gt;Every spec seeds a tenant and signs in its admin through a helper from our test kit. The helper posted the password, got back &lt;code&gt;{ mfaRequired: true, challengeToken }&lt;/code&gt; instead of a session cookie, and returned as if everything was fine. The 401 came one request later, from code that had nothing to do with login.&lt;/p&gt;

&lt;p&gt;The fix for the suite was to teach the kit MFA. The more useful fix was to make the helper fail where the problem is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;mfaRequired&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;mfaSetupRequired&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;`loginViaApi(&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;email&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;): the account needs MFA but no mfaTotpSecret was given; `&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
      &lt;span class="s2"&gt;`use seedTenant({ mfa: "totp" }) or addUser(roles, { mfa: "totp" })`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A test helper that swallows an unexpected state turns one clear failure into 89 confusing ones.&lt;/p&gt;

&lt;h2&gt;
  
  
  Teaching the kit TOTP
&lt;/h2&gt;

&lt;p&gt;The kit now enrolls a real TOTP factor through the real endpoints. Under a policy that blocks the login, that is the pre-auth enrollment flow. Without one, it is the normal enrollment from a signed-in session. &lt;code&gt;seedTenant()&lt;/code&gt; does this on its own when the admin's login answers with a setup step, so an app that switches on the policy does not have to touch every spec. Users created later can opt in with &lt;code&gt;addUser(roles, { mfa: "totp" })&lt;/code&gt;, which matters when a spec creates a member and promotes them to admin afterwards.&lt;/p&gt;

&lt;p&gt;One detail cost some thought. The server burns an accepted code for its whole validity window of one step either side, which is correct: a code seen once should not log anyone in twice. A suite that signs the same user in three times within 30 seconds then needs the next counter, and only codes up to one step ahead are valid. So the kit keeps the last counter per secret and, when it runs out of valid counters, waits until the clock catches up. That wait is tied to the TOTP window. It is not a test timeout, and we do not raise timeouts to make tests pass. In practice the full suite of 89 specs ran in 55 seconds with MFA on.&lt;/p&gt;

&lt;h2&gt;
  
  
  The policy had three doors around it
&lt;/h2&gt;

&lt;p&gt;With the suite green, a security review of the release went through the diff. It found nothing blocking in the app. It found two problems in the framework and pointed at a third.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Switching tenants.&lt;/strong&gt; A user can belong to several tenants. Picture someone who is a plain member in tenant A and an admin in tenant B, with no second factor. They sign in to A, where the policy does not apply to them, and get a session. Then they call &lt;code&gt;POST /auth/switch-tenant&lt;/code&gt; for B. That route minted a fresh session for B without asking the MFA question. An admin session without a second factor, from a stolen password.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Accepting an invitation.&lt;/strong&gt; Completing an invite signup issued the first session directly. That included admin invitations. The account holder could have enrolled a factor themselves, so this was hard to exploit, but it meant the first admin session of every invited admin had no second factor.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Signing up.&lt;/strong&gt; Self-service signup creates a new tenant and makes the person who signed up its admin. Confirming the signup issued that admin session directly, again without the gate. Veridom launches without self-registration, so this one did not block us, but it is the same hole.&lt;/p&gt;

&lt;p&gt;All three had the same root. The policy was enforced in the login handler, and these routes mint sessions without going through it. A policy flag is only as strong as the set of routes that create sessions, and nobody had made a list of that set.&lt;/p&gt;

&lt;h2&gt;
  
  
  The agent's first fix was worse than the bug
&lt;/h2&gt;

&lt;p&gt;I gave the fix to a coding agent with a clear description: put the same MFA check in front of the switch-tenant session.&lt;/p&gt;

&lt;p&gt;There is a layering wrinkle. The route lives in the framework core, the MFA gate lives in a bundled feature that the core cannot import. The agent solved that by creating a write handler for the gate and dispatching it from the route. Its first version took &lt;code&gt;tenantId&lt;/code&gt; and &lt;code&gt;roles&lt;/code&gt; from the request payload and was open to every signed-in user. Since all write handlers are reachable through the public write endpoint, anyone could have asked for an MFA challenge for any tenant, with any roles they liked. The second version restricted the handler to signed-in users and took the &lt;code&gt;userId&lt;/code&gt; from the payload, which made it possible to trigger the gate for other people.&lt;/p&gt;

&lt;p&gt;An automated security check flagged both versions while the agent was still working. I stopped it each time with the same rule, written out explicitly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;only the system identity may call the handler, and the route dispatches it as the system user;&lt;/li&gt;
&lt;li&gt;the payload carries &lt;code&gt;userId&lt;/code&gt; and &lt;code&gt;tenantId&lt;/code&gt;, nothing else;&lt;/li&gt;
&lt;li&gt;the handler resolves the membership and the roles itself, on the server, and returns an error when there is no membership;&lt;/li&gt;
&lt;li&gt;a test calls the handler directly as a normal user and expects a 403.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is the version we kept. The useful lesson for me: "put the same check here" is not a spec for auth code. The agent did what it was asked and filled the gaps with the most convenient assumption, which in auth code is the client's word. The rules that matter (who can call this, which inputs are trusted) have to be in the task, and a reviewer has to check them in the diff.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tests that fail when the gate is gone
&lt;/h2&gt;

&lt;p&gt;Each new test for the first two routes was checked by removing the thing it protects. Taking the gate out of the switch-tenant route turned two tests red. Taking it out of the invite signup turned one red. Opening the gate handler to normal users again turned four red, including the direct call.&lt;/p&gt;

&lt;p&gt;The new integration tests run over real HTTP against a real stack and cover: switching to an admin tenant without a factor (setup step, no session cookie), with a factor (challenge, then a session after verifying), switching to a tenant where the user is only a member (direct session), switching to a tenant without membership (error, no token), the invite signup for an admin and for a member, and the signup confirmation under the policy, including a direct call through the public write endpoint.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making the list
&lt;/h2&gt;

&lt;p&gt;Each fix closes one route. To catch the next route that mints a session, the framework now keeps the list we were missing. Every production route mints its session through one function, and a guard test lists each of those routes together with how the MFA gate applies to it. A new route that mints a session without an entry fails the build, and so does a token signed anywhere else.&lt;/p&gt;

&lt;p&gt;All of this shipped in Kumiko 0.345.0. Since 0.347.0, signing up under the policy also keeps the page the user wanted to land on after they enroll their factor. If you run Kumiko with an MFA policy, upgrade to at least 0.345.0.&lt;/p&gt;

&lt;p&gt;If you run a multi-tenant app with per-tenant roles: does your MFA policy hold when a user switches tenants? We found out ours did not when a security review read the release that turned the policy on.&lt;/p&gt;

</description>
      <category>security</category>
      <category>ai</category>
      <category>testing</category>
      <category>typescript</category>
    </item>
    <item>
      <title>We moved OCR out of the app so egress could be zero</title>
      <dc:creator>Marc</dc:creator>
      <pubDate>Wed, 30 Sep 2026 08:15:00 +0000</pubDate>
      <link>https://dev.to/marc_kumiko/we-moved-ocr-out-of-the-app-so-egress-could-be-zero-17m6</link>
      <guid>https://dev.to/marc_kumiko/we-moved-ocr-out-of-the-app-so-egress-could-be-zero-17m6</guid>
      <description>&lt;p&gt;Our document ingest path used to run OCR inside the app pods. The library wanted glibc. The app images were Alpine. Missing language packs got pulled from github.com at parse time. That last part is the one that kept me up: a PDF upload could trigger an outbound fetch from a pod that otherwise had no business talking to the public internet.&lt;/p&gt;

&lt;p&gt;So we moved parsing into its own service, baked the traineddata into the image, and closed egress entirely. Including DNS.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the app was the wrong place
&lt;/h2&gt;

&lt;p&gt;Three separate failures stacked in one process:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The native liteparse binary didn't load on Alpine. You discover that in the target environment, not in a glibc laptop build.&lt;/li&gt;
&lt;li&gt;Tessdata that wasn't already on disk downloaded at runtime. OCR "just worked" until the network policy, the rate limit, or the broken mirror made it not work.&lt;/li&gt;
&lt;li&gt;The parse library accepted path-shaped options (&lt;code&gt;tessdataPath&lt;/code&gt;, &lt;code&gt;imageOutputDir&lt;/code&gt;, &lt;code&gt;ocrServerUrl&lt;/code&gt;). A caller that could set those could point the parser at another filesystem or another host.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;None of those are model problems. They're packaging and trust-boundary problems that show up the first time you treat document upload as untrusted input, which it is.&lt;/p&gt;

&lt;h2&gt;
  
  
  One HTTP service, three caller knobs
&lt;/h2&gt;

&lt;p&gt;The service is a small Bun HTTP app: multipart upload in, pages of text out. Callers may pass only:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strictObject&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;ocrEnabled&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;boolean&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;optional&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
  &lt;span class="na"&gt;ocrLanguage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;string&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;regex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/^&lt;/span&gt;&lt;span class="se"&gt;[&lt;/span&gt;&lt;span class="sr"&gt;a-z&lt;/span&gt;&lt;span class="se"&gt;]{3}(\+[&lt;/span&gt;&lt;span class="sr"&gt;a-z&lt;/span&gt;&lt;span class="se"&gt;]{3})&lt;/span&gt;&lt;span class="sr"&gt;*$/&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;optional&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
  &lt;span class="na"&gt;maxPages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;number&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;positive&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;optional&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;strictObject&lt;/code&gt; rejects every other field by name. &lt;code&gt;tessdataPath&lt;/code&gt; is not an option; it comes from &lt;code&gt;LITEPARSE_TESSDATA_PATH&lt;/code&gt; and nowhere else. Languages are checked against the &lt;code&gt;.traineddata&lt;/code&gt; files present at startup. Ask for &lt;code&gt;fra&lt;/code&gt; when only &lt;code&gt;eng&lt;/code&gt; and &lt;code&gt;deu&lt;/code&gt; are baked in and you get &lt;code&gt;400&lt;/code&gt;, not a download.&lt;/p&gt;

&lt;p&gt;The image pins &lt;code&gt;deu&lt;/code&gt; and &lt;code&gt;eng&lt;/code&gt; from &lt;code&gt;tesseract-ocr/tessdata_best&lt;/code&gt; at a fixed commit SHA with sha256 verification. Build time is when we fetch. Runtime is when we refuse to.&lt;/p&gt;

&lt;p&gt;Auth is a shared bearer token, sixteen characters minimum. Health checks stay unauthenticated so Kubernetes can probe without holding the secret. There is no &lt;code&gt;/metrics&lt;/code&gt; scrape path, because nothing scrapes it yet and an open metrics port would be another thing to reason about.&lt;/p&gt;

&lt;h2&gt;
  
  
  Closed egress is the point
&lt;/h2&gt;

&lt;p&gt;In the cluster the service sits in its own namespace. The NetworkPolicy sets &lt;code&gt;egress: []&lt;/code&gt; — no outbound traffic, DNS included. Ingress is an allowlist of caller namespaces from stack config. Empty list means deny-all until the first consumer is declared.&lt;/p&gt;

&lt;p&gt;That shape only works because the container never needs the network. Puppeteer, sitting next to it in our stack, cannot do the same: it renders user HTML with external subresources. OCR can, once the models live in the image.&lt;/p&gt;

&lt;p&gt;App pods call &lt;code&gt;http://liteparse-service…:3000&lt;/code&gt; over the cluster network with the shared token. They never embed the native binary. Alpine stays Alpine. The parse blast radius is one dedicated Deployment with a memory tier and no way out.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we'd do again
&lt;/h2&gt;

&lt;p&gt;Pin traineddata in the image and fail closed on unknown languages. Whitelist request options so a multipart field cannot redirect filesystem or network paths. Put the NetworkPolicy next to the Deployment in the same PR, with ingress empty by default so a forgotten consumer config doesn't open the service to the whole cluster.&lt;/p&gt;

&lt;p&gt;If your OCR still downloads language packs on first use, you don't have an offline parser. You have a deferred &lt;code&gt;curl&lt;/code&gt; with a PDF as the trigger.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>kubernetes</category>
      <category>security</category>
    </item>
  </channel>
</rss>
