<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Neeraj Sharma</title>
    <description>The latest articles on DEV Community by Neeraj Sharma (@neeraj_sharma_757fbb49aa7).</description>
    <link>https://dev.to/neeraj_sharma_757fbb49aa7</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4156727%2Fa4e3050c-181e-455b-8a50-2398b0f2b950.png</url>
      <title>DEV Community: Neeraj Sharma</title>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/neeraj_sharma_757fbb49aa7"/>
    <language>en</language>
    <item>
      <title>How We Made a 55-Second AI Action Reel on One GPU: Open Models, an MCP Studio and an ffmpeg Finishing Pass</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Wed, 11 Nov 2026 08:09:58 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/how-we-made-a-55-second-ai-action-reel-on-one-gpu-open-models-an-mcp-studio-and-an-ffmpeg-22c3</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/how-we-made-a-55-second-ai-action-reel-on-one-gpu-open-models-an-mcp-studio-and-an-ffmpeg-22c3</guid>
      <description>&lt;p&gt;We posted a 55-second Instagram reel this week: a skydive over Palm Jumeirah, a jet ski past Burj Al Arab, a Lamborghini Urus launching off a dune, four recurring characters with their own voices, and a twist ending where the hero tells the camera the whole thing was AI. Every frame came out of a studio we self-host on one GPU, and Claude drove the whole production through tool calls.&lt;/p&gt;

&lt;p&gt;This post is the engineering side: how the studio is wired so an agent can use it, the one-GPU queue, and the ffmpeg finishing pass that turns raw clips into something you can post. The code below is simplified to the essentials.&lt;/p&gt;

&lt;h2&gt;
  
  
  The models
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;License&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Stills, character sheets&lt;/td&gt;
&lt;td&gt;Qwen-Image-2512, Qwen-Image-Edit&lt;/td&gt;
&lt;td&gt;Apache 2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Video with native audio&lt;/td&gt;
&lt;td&gt;MiniMax H3 (open weights since Aug 3, 2026)&lt;/td&gt;
&lt;td&gt;MiniMax H3 Community License&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Voices&lt;/td&gt;
&lt;td&gt;Qwen3-TTS&lt;/td&gt;
&lt;td&gt;Apache 2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Music&lt;/td&gt;
&lt;td&gt;ACE-Step 1.5&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A licensing note up front: the H3 license asks users in the US, UK, EU and South Korea to apply to MiniMax, and commercial products must name the model. Keep the video model behind an interface so you can swap it.&lt;/p&gt;

&lt;p&gt;H3's reference mode is what makes a series possible. You pass up to nine images (our character sheets plus a location still) and the dialogue, and it returns a 4 to 15 second clip at 768p with lip movement and stereo sound. Same faces, every shot.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: expose the studio as MCP tools
&lt;/h2&gt;

&lt;p&gt;The agent never touches the GPU. It calls tools, gets a job id, and polls. With the official Python MCP SDK that is a few lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;mcp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;server&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fastmcp&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;FastMCP&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;  &lt;span class="c1"&gt;# thin wrapper over Postgres&lt;/span&gt;

&lt;span class="n"&gt;mcp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;FastMCP&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"studio"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nd"&gt;@mcp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;generate_video&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;character_ids&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reference_images&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                   &lt;span class="n"&gt;duration_s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;aspect&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"9:16"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;quality&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"full"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""Queue a video clip. Returns a job id; poll job_status for the result."""&lt;/span&gt;
    &lt;span class="n"&gt;job_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enqueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"video"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;character_ids&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;character_ids&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                                      &lt;span class="n"&gt;reference_images&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;reference_images&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;duration_s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;duration_s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                                      &lt;span class="n"&gt;aspect&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;aspect&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;quality&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;quality&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"job_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;job_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"eta_s"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;estimate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"video"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;duration_s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;quality&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nd"&gt;@mcp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;job_status&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job_ids&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;wait_seconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""State, progress and (when done) presigned download URLs."""&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;status&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job_ids&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;wait_seconds&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wait_seconds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"__main__"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;mcp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details matter more than they look:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Return an ETA with every job.&lt;/strong&gt; The agent plans around it: it queues all stills first, reviews them while videos render, and does not sit polling a six-minute job every five seconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Let &lt;code&gt;job_status&lt;/code&gt; block for up to 50 seconds.&lt;/strong&gt; One long poll replaces a dozen chatty ones and saves a lot of tokens.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Step 2: one GPU, one heavy job, smart ordering
&lt;/h2&gt;

&lt;p&gt;On an RTX PRO 6000 (96 GB), a 10 second clip at full quality takes about 346 seconds and a 5 second clip about 125. Swapping between the video, image and image-edit models costs 35 to 40 seconds each time. So the worker does two things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Runs &lt;strong&gt;one heavy job at a time&lt;/strong&gt; (video and image models do not share the card well).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pulls images before videos&lt;/strong&gt; when both are waiting, so the model swaps as rarely as possible.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- claim the next job; images first, then oldest&lt;/span&gt;
&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'active'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;started_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;
  &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'waiting'&lt;/span&gt;
  &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'image'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;created_at&lt;/span&gt;
  &lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="no"&gt;SKIP&lt;/span&gt; &lt;span class="no"&gt;LOCKED&lt;/span&gt;
  &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;RETURNING&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The box is a spot instance. If it is off, the first job boots it (about 5 minutes plus a minute to load weights) and it can shut itself down when the queue stays empty. Spot pricing for this instance type is listed around $1.43 an hour, so a reel with all its retakes lands in the single-digit dollars.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: the finishing pass in ffmpeg
&lt;/h2&gt;

&lt;p&gt;Raw clips are not a reel. Ours go through one Python script per cut:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# segment: (file, start, end, keep_clip_audio, captions)&lt;/span&gt;
&lt;span class="n"&gt;vf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"trim={s}:{e},setpts=PTS-STARTPTS,scale=1080:1920:flags=lanczos,unsharp=5:5:0.8,fps=30"&lt;/span&gt;
&lt;span class="n"&gt;af&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"atrim={s}:{e},asetpts=PTS-STARTPTS,volume={1.0 if dialogue else 0.0}"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three lessons from this reel:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Mute the clips' own sound except dialogue.&lt;/strong&gt; H3 generates ambient audio, which is great in isolation and muddy under a song. We keep clip audio only on talking shots.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Duck music with an envelope, not a compressor.&lt;/strong&gt; Our first mix used &lt;code&gt;sidechaincompress&lt;/code&gt; and the song vanished under the final pitch. A plain volume envelope keeps the song about 9 dB under the voices and always audible:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"clip((t-{a0:.2f})/0.25,0,1)*clip(({b0:.2f}-t)/0.4,0,1)"&lt;/span&gt;   &lt;span class="c1"&gt;# 0 outside dialogue, 1 inside&lt;/span&gt;
&lt;span class="n"&gt;fc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"[mus]volume='1-0.72*{k}':eval=frame[duck];[dlg][duck]amix=inputs=2:normalize=0,loudnorm=I=-14"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Watch out for filter graph surprises.&lt;/strong&gt; Chaining &lt;code&gt;acrossfade&lt;/code&gt; on segments split from the same input silently dropped a whole chorus in our build (31 seconds of music for a 48 second video). &lt;code&gt;asplit&lt;/code&gt; plus &lt;code&gt;concat&lt;/code&gt; fixed it. We only caught it because we measure loudness per three-second window after every render.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: automatic QA before a human sees anything
&lt;/h2&gt;

&lt;p&gt;Every cut gets two checks:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Frames:&lt;/strong&gt; a contact sheet with one frame every 0.5 seconds, reviewed for anatomy, faces, captions and safe zones. This caught a character's head going through a car roof twice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Words:&lt;/strong&gt; a speech-to-text pass on the final mix. One line the voice model rendered as "it's all a AI" got cut at the word level and re-checked until it read "It's AI."
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;faster_whisper&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;WhisperModel&lt;/span&gt;
&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;WhisperModel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"small"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;compute_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"int8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;seg&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transcribe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;audio&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;word_timestamps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What it adds up to
&lt;/h2&gt;

&lt;p&gt;Four open models, a queue, a worker and an ffmpeg script. The agent does the directing, the GPU does the rendering, and the edit layer is where quality actually comes from. For context, Higgsfield, which sells this kind of capability as a hosted product, says it crossed a $1 billion annualized revenue run rate in September 2026. The opening for smaller teams is a private or niche version: one brand's characters, one industry's ads, one kids channel.&lt;/p&gt;

&lt;p&gt;If you want a studio like this built for your product or brand, that is what we do at Axionry. Book a call at &lt;a href="https://axionry.com/book" rel="noopener noreferrer"&gt;https://axionry.com/book&lt;/a&gt; or see our builds at &lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;https://axionry.com&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>python</category>
      <category>mcp</category>
    </item>
    <item>
      <title>Dub any video into another language in the speaker's own voice: Whisper, Claude and Chatterbox in 70 lines of Python</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Tue, 10 Nov 2026 07:09:16 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/dub-any-video-into-another-language-in-the-speakers-own-voice-whisper-claude-and-chatterbox-in-2ifk</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/dub-any-video-into-another-language-in-the-speakers-own-voice-whisper-claude-and-chatterbox-in-2ifk</guid>
      <description>&lt;p&gt;ElevenLabs passed $500M in annual recurring revenue this year, and a big part of what people pay for is simple to describe: take a video, keep the speaker's voice, and make them speak Spanish, Hindi or German. We made a reel about it this week (an uncle whose car starts talking in his own voice, in Spanish), and people asked how the pipeline actually works.&lt;/p&gt;

&lt;p&gt;So here it is, end to end, on three open pieces: Whisper for speech to text, Claude for the translation, and Chatterbox Multilingual for the cloned voice. It runs on one rented GPU.&lt;/p&gt;

&lt;p&gt;A rule before any code: only clone a voice with the owner's written permission. Store that consent next to the voice sample. Everything below assumes you have it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pipeline
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Whisper transcribes the video and gives you each line with a start and end time.&lt;/li&gt;
&lt;li&gt;Claude translates every line, keeping names, jokes and tone, and keeping each line short enough to say in the same slot.&lt;/li&gt;
&lt;li&gt;Chatterbox Multilingual speaks each translated line in the cloned voice, from a 10 to 20 second reference sample.&lt;/li&gt;
&lt;li&gt;ffmpeg places each new line at its original timestamp, nudges the speed if a line runs long, and muxes it back onto the video.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Setup
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install openai-whisper chatterbox-tts anthropic torchaudio
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;ANTHROPIC_API_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;...   &lt;span class="c"&gt;# your own key&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You need ffmpeg on the PATH and a CUDA GPU for reasonable speed. Chatterbox runs on CPU too, just slowly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: transcribe with timestamps
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;whisper&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;torchaudio&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;ta&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;anthropic&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Anthropic&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;chatterbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mtl_tts&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ChatterboxMultilingualTTS&lt;/span&gt;

&lt;span class="no"&gt;VIDEO&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;VOICE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;LANG&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;   &lt;span class="c1"&gt;# in.mp4 consented_sample.wav es&lt;/span&gt;

&lt;span class="n"&gt;asr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;whisper&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"small"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;segments&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;asr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transcribe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;VIDEO&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"segments"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;lines&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"end"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"end"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
         &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;segments&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Whisper's segments are usually one sentence or a natural phrase, which is exactly the unit you want to dub. If a speaker talks very fast, the "medium" model splits lines more cleanly than "small".&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: translate with Claude, line by line
&lt;/h2&gt;

&lt;p&gt;The trick is to translate for the ear, not the page. A literal translation of a joke is rarely funny, and a long translation will not fit the slot.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"""You are dubbing a video into the language with code "{LANG}".
Translate each line so it sounds natural when spoken. Keep names, jokes and tone.
Each translation must be speakable in about the same time as the original line.
Return only JSON: a list of objects {{"id": int, "text": str}}, one per input line.
Lines: {json.dumps(lines, ensure_ascii=False)}"""&lt;/span&gt;

&lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"claude-sonnet-4-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                             &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;
&lt;span class="n"&gt;translated&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"["&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rfind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"]"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sending the whole transcript in one call matters. Claude sees the context, so a pun set up in line 3 still lands in line 7, and names stay consistent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: speak every line in the cloned voice
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;tts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ChatterboxMultilingualTTS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;device&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"cuda"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;parts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;wav&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;translated&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;language_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="no"&gt;LANG&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;audio_prompt_path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="no"&gt;VOICE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"line_{l['id']:03d}.wav"&lt;/span&gt;
    &lt;span class="n"&gt;ta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;wav&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;wav&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;tts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# file, length in seconds&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;audio_prompt_path&lt;/code&gt; is the consented reference sample. A clean 10 to 20 second recording with no music under it gives a far better clone than a longer, noisy one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: put the lines back on the timeline
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;inputs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;filters&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;secs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;slot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"end"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;speed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;secs&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;slot&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.35&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# only speed up, never above 1.35x&lt;/span&gt;
    &lt;span class="n"&gt;ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;inputs&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"-i"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;filters&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"[{n + 1}:a]atempo={speed:.3f},adelay={ms}:all=1[a{n}]"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;mix&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"[a{n}]"&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;filters&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{mix}amix=inputs={len(parts)}:normalize=0[dub]"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"ffmpeg"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-y"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-i"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;VIDEO&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;inputs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-filter_complex"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;";"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;filters&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s2"&gt;"-map"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"0:v"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-map"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"[dub]"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-c:v"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"copy"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-c:a"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"aac"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"dubbed_{LANG}.mp4"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python dub.py talk.mp4 consented_sample.wav es
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the whole thing, under 70 lines.&lt;/p&gt;

&lt;h2&gt;
  
  
  What breaks first, and the fixes
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Lines that run long.&lt;/strong&gt; Spanish and German are often 15 to 25 percent longer than English. The 1.35x speed cap keeps voices natural. If a line still overflows, send it back to Claude with "shorter, same meaning" and the target seconds. One retry fixes almost all of them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Background music disappears.&lt;/strong&gt; This script replaces the whole audio track. For real videos, split the original into voice and background with a source separation model first, then mix the new voice over the original background.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Numbers and names.&lt;/strong&gt; TTS models read "$49" and "H3" in odd ways. Ask Claude to write numbers as words in the target language, and keep a small pronunciation list per creator.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lip sync.&lt;/strong&gt; This pipeline does not move the lips. For talking-head content, a lip-sync model is the next layer. For courses, screen recordings, podcasts and voiceover-led videos, you do not need it.&lt;/p&gt;

&lt;h2&gt;
  
  
  From script to product
&lt;/h2&gt;

&lt;p&gt;The script is the easy 20 percent. A product people pay for monthly adds a consent flow and voice vault, a GPU job queue so ten uploads do not fight over one card, a review screen where the creator can fix any line before export, and per-minute metering. That is the part that decides margins.&lt;/p&gt;

&lt;p&gt;The rough maths from our reel: 5,000 creators at $49 a month is about $3M a year, and the models cost nothing to license. You pay for GPU time and the Claude calls.&lt;/p&gt;

&lt;p&gt;We wrote up the full open-source stack, the build order and the niches worth selling to here: &lt;a href="https://axionry.com/open-source-ai-products/ai-voice-cloning-dubbing-app" rel="noopener noreferrer"&gt;https://axionry.com/open-source-ai-products/ai-voice-cloning-dubbing-app&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you want a product like this built for your own niche without hiring a team, that is what we do at Axionry: &lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;https://axionry.com&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>ai</category>
      <category>opensource</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Catching wrong-voice lines in AI-generated video with 60 lines of Python</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Thu, 05 Nov 2026 11:33:35 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/catching-wrong-voice-lines-in-ai-generated-video-with-60-lines-of-python-32f9</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/catching-wrong-voice-lines-in-ai-generated-video-with-60-lines-of-python-32f9</guid>
      <description>&lt;p&gt;Every few clips, our video model gives a line to the wrong character. The uncle speaks with the investor's voice, or a narrator turns up mid-scene. Watching every clip doesn't scale once you publish every day, so we built a check that does it for us. It's about 60 lines of Python.&lt;/p&gt;

&lt;p&gt;The setup: we run two animated series, &lt;a href="https://www.youtube.com/@hummingtreekids" rel="noopener noreferrer"&gt;Hummingtree&lt;/a&gt;, a nursery rhyme channel, and a 3D comedy cast on &lt;a href="https://www.instagram.com/axionry_/" rel="noopener noreferrer"&gt;Instagram&lt;/a&gt; with three recurring characters: Ethan, Uncle Rick and Chad, an investor. Clips come out of an image-to-video model that renders dialogue and lip-sync together. Faces stay on model. Voices don't always.&lt;/p&gt;

&lt;p&gt;Here's the pipeline that catches the wrong-voice lines and swaps them out before the edit.&lt;/p&gt;

&lt;h2&gt;
  
  
  The idea
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Keep one clean reference recording per character (10 to 20 seconds of that character talking).&lt;/li&gt;
&lt;li&gt;Transcribe each generated clip with word timestamps, so you know when each line starts and ends.&lt;/li&gt;
&lt;li&gt;Embed every line and every reference with a speaker encoder, then compare with cosine similarity.&lt;/li&gt;
&lt;li&gt;If a line is closer to another character's reference than to the expected speaker's, flag it.&lt;/li&gt;
&lt;li&gt;Mute the flagged span and drop in a clean TTS line, loudness matched to the clip.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Nothing here is model-specific. It works on any clip with speech in it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: references
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# pip install resemblyzer faster-whisper soundfile numpy
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;resemblyzer&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;VoiceEncoder&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;preprocess_wav&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;

&lt;span class="n"&gt;enc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;VoiceEncoder&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;REFS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ethan&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;embed_utterance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;preprocess_wav&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;refs/ethan.wav&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rick&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;embed_utterance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;preprocess_wav&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;refs/rick.wav&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chad&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;embed_utterance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;preprocess_wav&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;refs/chad.wav&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;embed_utterance&lt;/code&gt; returns a 256-dimension vector that is already L2-normalised, so cosine similarity is just a dot product.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: find the lines
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;faster_whisper&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;WhisperModel&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;soundfile&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;sf&lt;/span&gt;

&lt;span class="n"&gt;asr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;WhisperModel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;small&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;compute_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;int8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clip_wav&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;segs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;asr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transcribe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clip_wav&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;word_timestamps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;language&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;en&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;segs&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We convert every clip to 16 kHz mono first (&lt;code&gt;ffmpeg -i clip.mp4 -ac 1 -ar 16000 clip.wav&lt;/code&gt;). Whisper and the encoder both expect that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: score each line
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;who_said_it&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clip_wav&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;audio&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clip_wav&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;span&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;audio&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;sr&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;sr&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;sr&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.6&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;          &lt;span class="c1"&gt;# too short to judge reliably
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
    &lt;span class="n"&gt;emb&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;embed_utterance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;preprocess_wav&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;source_sr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;sr&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;scores&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;emb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ref&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;REFS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;scores&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;audit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clip_wav&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;expected: list of character names, one per line in script order&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;report&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="nf"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;want&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clip_wav&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;got&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;scores&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;who_said_it&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clip_wav&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;got&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;got&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;want&lt;/span&gt;
        &lt;span class="n"&gt;report&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;t&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                       &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;want&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;want&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;got&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;got&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ok&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                       &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;margin&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;want&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;report&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things we learned the hard way:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Skip spans under about 0.6 seconds.&lt;/strong&gt; A one-word "Whoa!" doesn't carry enough voice for a stable embedding, so you get noise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look at the margin, not only at the winner.&lt;/strong&gt; If the expected speaker loses by 0.01, it's usually a voice that moved slightly between clips, which is fine. If it loses by 0.1 or more, someone else is talking.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 4: the second failure mode, invented lines
&lt;/h2&gt;

&lt;p&gt;The same check finds a quieter bug. If you ask for an 8-second clip and the script line takes 3 seconds, the model fills the rest. Sometimes that's a smile or a nod. Sometimes it's a line nobody wrote: a mumble, a repeated name, or an extra "Uncle Rick!" at the start of a scene.&lt;/p&gt;

&lt;p&gt;The fix is simple. Whisper returns more segments than the script has lines, so anything outside the scripted spans gets cut in the edit or muted:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;extra_speech&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;report_lines&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;scripted_count&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;report_lines&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;scripted_count&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt;   &lt;span class="c1"&gt;# everything past the script is invented
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's even better to stop it happening: keep each generated clip close to the length of its dialogue, and write a visual beat into the prompt for any silence ("he grins and raises his glass, no dialogue").&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: patch the line
&lt;/h2&gt;

&lt;p&gt;For a flagged line, we synthesise the correct voice and splice it in at the same loudness as the surrounding clip:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;patch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clip_mp4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out_mp4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tts_wav&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# zero the bad span, then mix the new line in at `start`
&lt;/span&gt;    &lt;span class="n"&gt;filt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[0:a]volume=enable=&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;between(t,&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;:volume=0[a0];&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[1:a]adelay=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;[a1];&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[a0][a1]amix=inputs=2:normalize=0:duration=first[a]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ffmpeg&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-y&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-i&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;clip_mp4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-i&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tts_wav&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-filter_complex&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;filt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-map&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;0:v&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-map&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[a]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-c:v&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;copy&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-c:a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;aac&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out_mp4&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before mixing, scale the TTS file so its RMS matches the clip's speech. Otherwise the patched line jumps out. Lip-sync on a replaced line is never perfect, so we prefer to replace lines where the character is facing away, or a reaction shot is on screen.&lt;/p&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;On our last two comedy episodes the audit flagged wrong voices in 5 of 14 clips. In one clip every line came out in the wrong voice. In the others it was a single line, including the closing call to action. All five were fixed in the edit without regenerating any video. Regenerating a clip takes minutes of GPU time and often introduces a new problem. A 2-second audio patch costs nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this sits in the pipeline
&lt;/h2&gt;

&lt;p&gt;Voice audit is one of four automatic checks we run on every clip before a person watches it:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Frame sheet:&lt;/strong&gt; 12 evenly spaced frames on one image, to catch warped hands, extra characters or the camera wandering off.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Transcript:&lt;/strong&gt; word timestamps against the script, to find missing and invented lines.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Voice match:&lt;/strong&gt; the check above.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Layout check:&lt;/strong&gt; every caption and overlay drawn on the real frames, so no text box ever sits on a face.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Each check takes a few seconds. Together they're why a daily schedule is possible at all.&lt;/p&gt;

&lt;p&gt;If you'd like this kind of series without building the pipeline yourself, it's what our &lt;a href="https://axionry.com/services/animation-studio" rel="noopener noreferrer"&gt;AI animation studio&lt;/a&gt; does: every episode as a YouTube video, a Short and a Reel, with music, voices, retakes, covers and captions included.&lt;/p&gt;

</description>
      <category>python</category>
      <category>ai</category>
      <category>ffmpeg</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Architecture of a Multi-Tenant Private ChatGPT SaaS on LibreChat: Silo Tenants, Seat Billing and Token Caps</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Wed, 04 Nov 2026 10:23:09 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/architecture-of-a-multi-tenant-private-chatgpt-saas-on-librechat-silo-tenants-seat-billing-and-33hl</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/architecture-of-a-multi-tenant-private-chatgpt-saas-on-librechat-silo-tenants-seat-billing-and-33hl</guid>
      <description>&lt;p&gt;&lt;em&gt;This post is AI-assisted.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;ChatGPT Business costs $20 per user per month on annual billing and $25 on monthly billing. Claude Team's standard seat is the same $20 and $25. Microsoft 365 Copilot Business is $25.20 on monthly billing, on top of a Microsoft 365 license. That is a price umbrella, and a focused product can live under it: a private, branded company ChatGPT, connected to each customer's own tools, sold at $12 a seat.&lt;/p&gt;

&lt;p&gt;The open-source core already exists. &lt;strong&gt;LibreChat&lt;/strong&gt; (MIT, about 46.6K GitHub stars) gives you the chat UI, agents, skills, MCP, multi-user auth, an admin panel and token balances. Shopify runs its internal AI on it, with thousands of custom agents and more than 30 internal MCP servers, according to ClickHouse's announcement when it acquired the project in November 2025.&lt;/p&gt;

&lt;p&gt;What LibreChat does not give you is the SaaS around it: signup, provisioning, isolation between customers, seat billing and cost control. This post is the architecture for that layer.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fra3ev96jubfhxfeca4zq.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fra3ev96jubfhxfeca4zq.jpg" alt="Two founders sketching a multi-tenant private ChatGPT architecture: control plane, tenants, Claude and MCP connectors" width="799" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Requirements
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Functional&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A customer signs up, picks a plan and gets &lt;code&gt;acme.yourapp.com&lt;/code&gt; within minutes&lt;/li&gt;
&lt;li&gt;Their staff sign in with the customer's own identity provider (Google Workspace, Microsoft Entra ID, Okta)&lt;/li&gt;
&lt;li&gt;Claude as the model, MCP connectors to the customer's tools, shared agents per customer&lt;/li&gt;
&lt;li&gt;Billing per seat, updated automatically as people join and leave&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Non-functional&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Hard isolation of chats, files and connector tokens between customers&lt;/li&gt;
&lt;li&gt;Bounded model spend per user and per customer&lt;/li&gt;
&lt;li&gt;Upgrades that can be rolled to one customer first&lt;/li&gt;
&lt;li&gt;Target cost per seat of about $3 to keep a 75% gross margin at $12&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Silo, not pool
&lt;/h2&gt;

&lt;p&gt;LibreChat is designed as one deployment per organization: one &lt;code&gt;librechat.yaml&lt;/code&gt;, one &lt;code&gt;.env&lt;/code&gt;, one user base, one admin panel. You could try to bolt tenant IDs onto every collection, but you would be fighting the codebase on every upgrade.&lt;/p&gt;

&lt;p&gt;So the data plane is &lt;strong&gt;one stack per customer&lt;/strong&gt; (the silo model), and the shared parts live in a small &lt;strong&gt;control plane&lt;/strong&gt; you write yourself.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concern&lt;/th&gt;
&lt;th&gt;Silo (one stack per tenant)&lt;/th&gt;
&lt;th&gt;Pool (shared stack)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Data isolation&lt;/td&gt;
&lt;td&gt;Separate database per tenant&lt;/td&gt;
&lt;td&gt;Tenant ID on every query&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Per-tenant config, branding, MCP servers&lt;/td&gt;
&lt;td&gt;Native: separate yaml and env&lt;/td&gt;
&lt;td&gt;Needs a custom config layer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Noisy neighbor&lt;/td&gt;
&lt;td&gt;Contained&lt;/td&gt;
&lt;td&gt;Shared blast radius&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Upgrades&lt;/td&gt;
&lt;td&gt;Canary per tenant&lt;/td&gt;
&lt;td&gt;All at once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Infra cost per tenant&lt;/td&gt;
&lt;td&gt;Higher&lt;/td&gt;
&lt;td&gt;Lower&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fit with upstream LibreChat&lt;/td&gt;
&lt;td&gt;Clean&lt;/td&gt;
&lt;td&gt;Fork pressure&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Start silo. When you have hundreds of small tenants, pack several tenant containers per host, but keep separate databases.&lt;/p&gt;

&lt;h2&gt;
  
  
  The architecture
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 Control plane (yours)
  +================================================+
  | Signup + plan    Tenant registry (Postgres)    |
  | Stripe webhooks  Provisioner   Seat sync job   |
  | Usage + alerts   Upgrade orchestrator          |
  +================================================+
           | creates / updates
           v
  Data plane: one stack per tenant (acme.yourapp.com)
  +================================================+
  | LibreChat API + web (pinned image tag)         |
  | MongoDB database: acme (users, chats, balance) |
  | Meilisearch index (chat search)                |
  | RAG API + pgvector (file search)               |
  | MCP servers for this tenant's tools            |
  +================================================+
           |                          |
   Anthropic API                Customer IdP (OpenID)
   (one Workspace per tenant,   Customer tools via MCP
    own key + spend limit)       (per-user OAuth)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three decisions do most of the work:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;One Anthropic Workspace per tenant.&lt;/strong&gt; The Anthropic Console lets you group API keys into workspaces, with a monthly spend limit and rate limits per workspace. Each tenant stack gets its own workspace-scoped key. Spend reporting per customer comes for free, and a runaway tenant hits its own ceiling, not yours.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;OpenID per tenant.&lt;/strong&gt; Each tenant's &lt;code&gt;.env&lt;/code&gt; points at the customer's identity provider. People leave the company, they lose access. No passwords for you to store.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-user OAuth for MCP.&lt;/strong&gt; LibreChat supports &lt;code&gt;stdio&lt;/code&gt;, &lt;code&gt;sse&lt;/code&gt;, &lt;code&gt;websocket&lt;/code&gt; and &lt;code&gt;streamable-http&lt;/code&gt; MCP servers, and with &lt;code&gt;requiresOAuth: true&lt;/code&gt; each person connects their own account. The assistant can only see what that person can see in the underlying tool.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The tenant config template
&lt;/h2&gt;

&lt;p&gt;The provisioner renders this per tenant. The values that change are the tenant name, the spend cap and the MCP servers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1.3.18&lt;/span&gt;

&lt;span class="na"&gt;interface&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;customWelcome&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;Welcome&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;to&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Acme&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;AI.&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Ask&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;about&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;your&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;clients,&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;files&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;and&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;projects.'&lt;/span&gt;

&lt;span class="na"&gt;balance&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;enabled&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="na"&gt;startBalance&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;6000000&lt;/span&gt;        &lt;span class="c1"&gt;# 1,000,000 credits = $1, so $6 of tokens&lt;/span&gt;
  &lt;span class="na"&gt;autoRefillEnabled&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="na"&gt;refillIntervalValue&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;30&lt;/span&gt;
  &lt;span class="na"&gt;refillIntervalUnit&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;days'&lt;/span&gt;
  &lt;span class="na"&gt;refillAmount&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;6000000&lt;/span&gt;
  &lt;span class="na"&gt;refillMode&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;reset'&lt;/span&gt;          &lt;span class="c1"&gt;# replace the allowance each period, no rollover&lt;/span&gt;
  &lt;span class="na"&gt;display&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;currency'&lt;/span&gt;

&lt;span class="na"&gt;mcpServers&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;documents&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;streamable-http&lt;/span&gt;
    &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https://mcp.acme.yourapp.com/documents/mcp&lt;/span&gt;
    &lt;span class="na"&gt;requiresOAuth&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;timeout&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;60000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the branding in the tenant &lt;code&gt;.env&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;APP_TITLE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Acme AI"&lt;/span&gt;
&lt;span class="nv"&gt;CUSTOM_FOOTER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Acme AI. Private assistant for Acme staff."&lt;/span&gt;
&lt;span class="nv"&gt;ALLOW_REGISTRATION&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false
&lt;/span&gt;&lt;span class="nv"&gt;ALLOW_SOCIAL_LOGIN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true
&lt;/span&gt;&lt;span class="nv"&gt;OPENID_ISSUER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;https://login.acme-idp.example.com
&lt;span class="nv"&gt;OPENID_CLIENT_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;from-the-customer
&lt;span class="nv"&gt;OPENID_BUTTON_LABEL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Sign in with Acme"&lt;/span&gt;
&lt;span class="nv"&gt;ANTHROPIC_API_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;key-from-the-acme-workspace
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;refillMode: 'reset'&lt;/code&gt; line matters for a seat business. With the older &lt;code&gt;add&lt;/code&gt; behavior, unused allowance rolls over and a quiet user can bank a large balance. Reset caps every person at the same monthly ceiling. Check that every replica runs a version that supports reset before you switch, as the example config warns.&lt;/p&gt;

&lt;h2&gt;
  
  
  Provisioning flow
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// control-plane/provision.ts (sketch)&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;provisionTenant&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Tenant&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;ws&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;createAnthropicWorkspace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;monthlyLimitUsd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;seats&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;createMongoDatabase&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;writeConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;renderYaml&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;renderEnv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;ws&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;apiKey&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;deployStack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`librechat:&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;PINNED_TAG&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt; &lt;span class="c1"&gt;// ECS service or compose on a host&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;createDnsRecord&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.yourapp.com`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;waitForHealthy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`https://&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.yourapp.com/health`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;registry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;markActive&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;createAnthropicWorkspace&lt;/code&gt; is a placeholder for however you manage workspaces and keys; whether you do it by API or once by hand in the Console, keep the key in a secrets manager and inject it at deploy time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Seat billing that follows reality
&lt;/h2&gt;

&lt;p&gt;Bill on seats provisioned, sync once a day, and let Stripe prorate.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// control-plane/seat-sync.ts&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;Stripe&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;stripe&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Stripe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;STRIPE_SECRET_KEY&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;syncSeats&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Tenant&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;users&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;tenantDb&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;collection&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;users&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;countDocuments&lt;/span&gt;&lt;span class="p"&gt;({});&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;seats&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;users&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;minimumSeats&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;seats&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;billedSeats&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;subscriptionItems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stripeSeatItemId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;quantity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;seats&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;proration_behavior&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;create_prorations&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;});&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;registry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setBilledSeats&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;seats&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Stripe Billing costs 0.7% of billing volume on pay as you go, plus the standard US card fee of 2.9% plus 30 cents. On a $600 monthly invoice for 50 seats that is about $0.44 per seat.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cost control in three fences
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Per user:&lt;/strong&gt; LibreChat balances, as configured above. When a user's allowance is spent, their requests stop until the next reset.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per tenant:&lt;/strong&gt; the Anthropic Workspace spend limit, set to roughly seats times the per-user cap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per model:&lt;/strong&gt; route by task. Claude Haiku 5.5 is $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100K tokens. Claude Sonnet 5.5 is $2 and $10. Make Haiku the default for quick questions and send agents and long documents to Sonnet.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The model mix is the biggest lever in the whole business. With 300 messages per seat per month, 4,000 input and 600 output tokens per message:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mix&lt;/th&gt;
&lt;th&gt;Token cost per seat per month&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;100% Sonnet 5.5&lt;/td&gt;
&lt;td&gt;$4.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;30% Sonnet, 70% Haiku&lt;/td&gt;
&lt;td&gt;$1.41&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100% Haiku 5.5&lt;/td&gt;
&lt;td&gt;$0.21&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Add about $1.20 per seat for a $60 monthly hosting budget on a 50-seat stack (the t4g.medium that runs the app lists at about $24.53 a month on demand in us-east-1) and the $0.44 payments line. The routed mix lands at about $3.05 per seat, a 75% gross margin at $12. Prompt caching, which these numbers ignore, only improves it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Upgrades without fear
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Pin the LibreChat image tag per tenant in the registry.&lt;/li&gt;
&lt;li&gt;Keep one internal tenant on the newest tag, then one friendly customer, then everyone.&lt;/li&gt;
&lt;li&gt;Render configs from templates so a new config key is one change, not 50.&lt;/li&gt;
&lt;li&gt;Run a smoke test after each tenant upgrade: login, one Haiku message, one Sonnet message, one MCP tool call.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What fails first
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;MCP OAuth tokens&lt;/strong&gt; expire or get revoked. Make reconnecting a one-click path for users.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long conversations&lt;/strong&gt; grow context with every turn. Nudge people to start a new chat per task.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Slow MCP servers&lt;/strong&gt; make the whole answer slow. Set per-server timeouts and alert on them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Seat disputes.&lt;/strong&gt; Show every customer admin the exact list of users being billed.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Build it with us
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;Axionry&lt;/a&gt; is an AI product development studio. We build private company assistants and multi-tenant AI products like this one, with Claude, MCP connectors, SSO and spend controls, and we ship them in weeks. The product page is at &lt;a href="https://axionry.com/build/private-chatgpt" rel="noopener noreferrer"&gt;axionry.com/build/private-chatgpt&lt;/a&gt;, and you can &lt;a href="https://axionry.com/book" rel="noopener noreferrer"&gt;book a call&lt;/a&gt; to talk through your version.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;ChatGPT Business pricing: &lt;a href="https://openai.com/business/pricing/" rel="noopener noreferrer"&gt;https://openai.com/business/pricing/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Claude Team pricing update (Jan 2026): &lt;a href="https://claude.com/it/blog/claude-team-updates" rel="noopener noreferrer"&gt;https://claude.com/it/blog/claude-team-updates&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Microsoft 365 Copilot Business pricing: &lt;a href="https://www.microsoft.com/en-us/microsoft-365/copilot/business" rel="noopener noreferrer"&gt;https://www.microsoft.com/en-us/microsoft-365/copilot/business&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;ClickHouse acquires LibreChat, Shopify figures (4 Nov 2025): &lt;a href="https://www.businesswire.com/news/home/20251104505230/en" rel="noopener noreferrer"&gt;https://www.businesswire.com/news/home/20251104505230/en&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LibreChat repo: &lt;a href="https://github.com/danny-avila/LibreChat" rel="noopener noreferrer"&gt;https://github.com/danny-avila/LibreChat&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LibreChat MCP server config: &lt;a href="https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/mcp_servers" rel="noopener noreferrer"&gt;https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/mcp_servers&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LibreChat balance config: &lt;a href="https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/balance" rel="noopener noreferrer"&gt;https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/balance&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic API pricing: &lt;a href="https://platform.claude.com/docs/en/about-claude/pricing" rel="noopener noreferrer"&gt;https://platform.claude.com/docs/en/about-claude/pricing&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic Console workspaces: &lt;a href="https://claude.com/resources/articles/workspaces" rel="noopener noreferrer"&gt;https://claude.com/resources/articles/workspaces&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Stripe Billing pricing: &lt;a href="https://stripe.com/billing/pricing" rel="noopener noreferrer"&gt;https://stripe.com/billing/pricing&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;AWS t4g.medium price (Holori): &lt;a href="https://calculator.holori.com/aws/ec2/t4g.medium/us-east-1" rel="noopener noreferrer"&gt;https://calculator.holori.com/aws/ec2/t4g.medium/us-east-1&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>architecture</category>
      <category>opensource</category>
      <category>saas</category>
    </item>
    <item>
      <title>How to Build Your Own Team ChatGPT With LibreChat, Claude and MCP (Pay per Token, Not per Seat)</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Tue, 03 Nov 2026 18:56:20 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/how-to-build-your-own-team-chatgpt-with-librechat-claude-and-mcp-pay-per-token-not-per-seat-1fma</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/how-to-build-your-own-team-chatgpt-with-librechat-claude-and-mcp-pay-per-token-not-per-seat-1fma</guid>
      <description>&lt;p&gt;Most companies buy AI as seats. ChatGPT Business is $20 per user per month on annual billing and $25 on monthly billing, and OpenAI says more than 9 million paying business users now rely on ChatGPT for work. For a 200-person company that is $48,000 to $60,000 a year, whether people use it or not.&lt;/p&gt;

&lt;p&gt;This guide builds the alternative that a lot of engineering teams are moving to: your own "team ChatGPT" on &lt;strong&gt;LibreChat&lt;/strong&gt; (open source, MIT, about 46.6K GitHub stars), with &lt;strong&gt;Claude&lt;/strong&gt; as the model, &lt;strong&gt;MCP connectors&lt;/strong&gt; for Google Drive and Gmail, shared &lt;strong&gt;skills and agents&lt;/strong&gt;, company &lt;strong&gt;SSO&lt;/strong&gt;, and &lt;strong&gt;per-user spend caps&lt;/strong&gt;. It runs in your own cloud account and you pay per token instead of per seat.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcil53buzagu5ses1vybi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcil53buzagu5ses1vybi.png" alt="A team using a self-hosted team ChatGPT built on LibreChat and Claude" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Everything below comes from the current LibreChat example config (&lt;code&gt;librechat.yaml&lt;/code&gt; version 1.3.18) and docs. Check the docs for your release before you copy anything into production.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you are building
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Browser (SSO login)
   |
LibreChat (web + API)  -&amp;gt;  MongoDB (users, chats, balances)
   |                   -&amp;gt;  Meilisearch (chat search)
   |                   -&amp;gt;  RAG API + pgvector (file search)
   |
   +-&amp;gt; Anthropic API (Claude Sonnet 5.5 / Haiku 5.5)
   +-&amp;gt; MCP servers (Google Workspace: Drive, Gmail, Calendar)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Linux VM with Docker and Docker Compose (start with 2 vCPU and 8 GB RAM for a pilot)&lt;/li&gt;
&lt;li&gt;A domain and TLS (any reverse proxy or a cloud load balancer)&lt;/li&gt;
&lt;li&gt;An Anthropic API key&lt;/li&gt;
&lt;li&gt;Optional: an OpenID Connect app in Google Workspace, Okta or Microsoft Entra ID for SSO&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1: Clone and configure
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/danny-avila/LibreChat.git
&lt;span class="nb"&gt;cd &lt;/span&gt;LibreChat
&lt;span class="nb"&gt;cp&lt;/span&gt; .env.example .env
&lt;span class="nb"&gt;cp &lt;/span&gt;librechat.example.yaml librechat.yaml
&lt;span class="nb"&gt;cp &lt;/span&gt;docker-compose.override.yml.example docker-compose.override.yml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In &lt;code&gt;docker-compose.override.yml&lt;/code&gt;, uncomment the block that mounts your config into the API container:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;api&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;bind&lt;/span&gt;
      &lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./librechat.yaml&lt;/span&gt;
      &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/app/librechat.yaml&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 2: Point it at Claude
&lt;/h2&gt;

&lt;p&gt;In &lt;code&gt;.env&lt;/code&gt;, set a company-owned key (the default &lt;code&gt;user_provided&lt;/code&gt; asks each user for their own key, which is not what you want for a team):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;ANTHROPIC_API_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;sk-ant-your-company-key
&lt;span class="nv"&gt;ANTHROPIC_MODELS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;claude-sonnet-5-5,claude-haiku-5-5
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why two models: on Anthropic's current price list, Sonnet 5.5 is $2 / $10 per million input / output tokens and Haiku 5.5 is $0.10 / $0.50 for prompts up to 100K tokens. Offer Haiku for quick rewrites and summaries, and keep Sonnet for real reasoning. That one choice can cut the token bill roughly in half.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Turn on spend caps before anyone logs in
&lt;/h2&gt;

&lt;p&gt;Per-token pricing has one risk: a single power user running long agent sessions. LibreChat has a built-in token balance. In &lt;code&gt;librechat.yaml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1.3.18&lt;/span&gt;

&lt;span class="na"&gt;balance&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;enabled&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="na"&gt;startBalance&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;15000000&lt;/span&gt;       &lt;span class="c1"&gt;# 1,000,000 credits = $1, so about $15&lt;/span&gt;
  &lt;span class="na"&gt;autoRefillEnabled&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="na"&gt;refillIntervalValue&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;30&lt;/span&gt;
  &lt;span class="na"&gt;refillIntervalUnit&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;days'&lt;/span&gt;
  &lt;span class="na"&gt;refillAmount&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;15000000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;LibreChat counts 1,000 credits as $0.001, so 15,000,000 credits is roughly $15 of model usage per person per month, still under a $20 to $25 seat. Tune the numbers after the first two weeks of real usage. The point is that no single account can surprise you.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: Company SSO
&lt;/h2&gt;

&lt;p&gt;Use OpenID Connect so people sign in with their work account and leave when they leave. In &lt;code&gt;.env&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;ALLOW_SOCIAL_LOGIN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true
&lt;/span&gt;&lt;span class="nv"&gt;OPENID_CLIENT_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;your-client-id
&lt;span class="nv"&gt;OPENID_CLIENT_SECRET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;your-client-secret
&lt;span class="nv"&gt;OPENID_ISSUER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;https://your-idp.example.com
&lt;span class="nv"&gt;OPENID_SESSION_SECRET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;a-long-random-string
&lt;span class="nv"&gt;OPENID_SCOPE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"openid profile email"&lt;/span&gt;
&lt;span class="nv"&gt;OPENID_CALLBACK_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/oauth/openid/callback
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once SSO works, set &lt;code&gt;ALLOW_REGISTRATION=false&lt;/code&gt; so nobody can create a local account around it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: MCP connectors for Drive and Gmail
&lt;/h2&gt;

&lt;p&gt;This is what turns a chat window into a work tool. LibreChat speaks the Model Context Protocol and supports &lt;code&gt;stdio&lt;/code&gt;, &lt;code&gt;sse&lt;/code&gt;, &lt;code&gt;websocket&lt;/code&gt; and &lt;code&gt;streamable-http&lt;/code&gt; servers. Each user's connection and OAuth tokens are kept per user and never shared.&lt;/p&gt;

&lt;p&gt;For Google Workspace, one option is the open-source &lt;a href="https://github.com/taylorwilsdon/google_workspace_mcp" rel="noopener noreferrer"&gt;google_workspace_mcp&lt;/a&gt; server (MIT), which covers Gmail, Drive, Calendar, Docs, Sheets and more, runs over streamable HTTP and supports OAuth 2.1 for multiple users. Run it next to LibreChat and register it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;mcpServers&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;google-workspace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;streamable-http&lt;/span&gt;
    &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http://workspace-mcp:8000/mcp&lt;/span&gt;
    &lt;span class="na"&gt;requiresOAuth&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;timeout&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;60000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When someone first uses a Drive or Gmail tool, LibreChat asks them to sign in with their own Google account, so the assistant only sees what that person can already see.&lt;/p&gt;

&lt;p&gt;You can also pass the signed-in user to your own internal MCP servers with placeholders:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;  &lt;span class="na"&gt;internal-crm&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;streamable-http&lt;/span&gt;
    &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https://crm-mcp.internal.example.com/mcp&lt;/span&gt;
    &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;X-User-Email&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;{{LIBRECHAT_USER_EMAIL}}'&lt;/span&gt;
      &lt;span class="na"&gt;X-API-Key&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;${CRM_MCP_KEY}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 6: Shared skills and agents
&lt;/h2&gt;

&lt;p&gt;Two features make the tool feel like it belongs to your company.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Skills.&lt;/strong&gt; LibreChat can mirror a GitHub repo of &lt;code&gt;SKILL.md&lt;/code&gt; files, so your team keeps prompts and procedures (brand voice, proposal format, support macros) in version control:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;skillSync&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;github&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;enabled&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;intervalMinutes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;60&lt;/span&gt;
    &lt;span class="na"&gt;runOnStartup&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;sources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;company-skills&lt;/span&gt;
        &lt;span class="na"&gt;owner&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;your-org&lt;/span&gt;
        &lt;span class="na"&gt;repo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ai-skills&lt;/span&gt;
        &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;main&lt;/span&gt;
        &lt;span class="na"&gt;paths&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;skills&lt;/span&gt;
        &lt;span class="na"&gt;skillDiscoveryDepth&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;
        &lt;span class="na"&gt;token&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;${GITHUB_SKILLS_TOKEN}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Agents.&lt;/strong&gt; In the Agent Builder, create agents for real jobs, add MCP tools from the Tool Library, and share them with a group. Three that pay off quickly:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Proposal drafter&lt;/strong&gt;: reads the client folder in Drive, writes the first draft in your format.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Policy answerer&lt;/strong&gt;: answers HR and IT questions from your handbook files, with sources.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Inbox triage&lt;/strong&gt;: summarises unread threads and drafts replies for a human to send.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Step 7: Start it and set roles
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker compose up &lt;span class="nt"&gt;-d&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use the admin panel to manage users, groups and roles: who can create agents, who can add MCP servers, and which groups see which agents.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;Assume 200 people, 25 messages each per working day, 4,000 input and 600 output tokens per message:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Setup&lt;/th&gt;
&lt;th&gt;Monthly cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;ChatGPT Business, 200 seats&lt;/td&gt;
&lt;td&gt;$4,000 to $5,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LibreChat + Sonnet 5.5 only (+ ~$300 hosting)&lt;/td&gt;
&lt;td&gt;about $1,770&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LibreChat + half Haiku 5.5 (+ ~$300 hosting)&lt;/td&gt;
&lt;td&gt;about $1,070&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These are estimates from the stated assumptions, not quotes. Run them with your own usage numbers.&lt;/p&gt;

&lt;h2&gt;
  
  
  When not to do this
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Under about 50 daily users, the seat plan is cheaper than the engineering time.&lt;/li&gt;
&lt;li&gt;If nobody on the team can own upgrades, backups and connector issues, do not self-host.&lt;/li&gt;
&lt;li&gt;If the only reason is cost, check the numbers twice. The strongest reasons are data in your own cloud, multiple model vendors, and connecting internal systems.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Need it built?
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;Axionry&lt;/a&gt; is an AI product development studio. We build private team assistants like this, with your SSO, your connectors and Claude or open-weights models, in a couple of weeks. Here is the product page: &lt;a href="https://axionry.com/build/private-chatgpt" rel="noopener noreferrer"&gt;axionry.com/build/private-chatgpt&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;OpenAI, 9 million paying business users (27 Feb 2026): &lt;a href="https://openai.com/index/scaling-ai-for-everyone/" rel="noopener noreferrer"&gt;https://openai.com/index/scaling-ai-for-everyone/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;ChatGPT Business pricing: &lt;a href="https://openai.com/business/pricing/" rel="noopener noreferrer"&gt;https://openai.com/business/pricing/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LibreChat repo: &lt;a href="https://github.com/danny-avila/LibreChat" rel="noopener noreferrer"&gt;https://github.com/danny-avila/LibreChat&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LibreChat MCP server config: &lt;a href="https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/mcp_servers" rel="noopener noreferrer"&gt;https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/mcp_servers&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;LibreChat balance config: &lt;a href="https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/balance" rel="noopener noreferrer"&gt;https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/balance&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic pricing: &lt;a href="https://platform.claude.com/docs/en/about-claude/pricing" rel="noopener noreferrer"&gt;https://platform.claude.com/docs/en/about-claude/pricing&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Google Workspace MCP server: &lt;a href="https://github.com/taylorwilsdon/google_workspace_mcp" rel="noopener noreferrer"&gt;https://github.com/taylorwilsdon/google_workspace_mcp&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>mcp</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>How to Build a Manus-Style AI Employee With Claude, OpenClaw, Lightpanda and Mem0</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Mon, 02 Nov 2026 12:11:31 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/how-to-build-a-manus-style-ai-employee-with-claude-openclaw-lightpanda-and-mem0-4mno</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/how-to-build-a-manus-style-ai-employee-with-claude-openclaw-lightpanda-and-mem0-4mno</guid>
      <description>&lt;p&gt;Meta closed its acquisition of Manus in December 2025 for a reported price above $2 billion (&lt;a href="https://www.techradar.com/pro/meta-buys-manus-for-usd2-billion-to-power-high-stakes-ai-agent-race" rel="noopener noreferrer"&gt;TechRadar&lt;/a&gt;). Manus sold AI agents that each get their own virtual computer, and it reached about $100M in annualized revenue roughly eight months after its March 2025 launch (&lt;a href="https://www.trendingtopics.eu/meta-acquires-ai-agent-startup-manus-which-recently-reached-100-million-arr/" rel="noopener noreferrer"&gt;Trending Topics&lt;/a&gt;). It did not train its own model.&lt;/p&gt;

&lt;p&gt;That last fact is the interesting one for developers. The value was in the harness: computer, browser, memory, permissions. Most of that harness now exists as open source. This post walks through how the pieces fit if you want to build a narrow "AI employee" for one type of small business.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdda5vpbs26ynim7mei3g.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdda5vpbs26ynim7mei3g.jpg" alt="AI employee stack on a whiteboard" width="800" height="448"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The four layers
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Open source option&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Brain&lt;/td&gt;
&lt;td&gt;Plan steps, choose tools, check results&lt;/td&gt;
&lt;td&gt;Claude via API (paid per use)&lt;/td&gt;
&lt;td&gt;The only paid part of the stack&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hands&lt;/td&gt;
&lt;td&gt;Files, shell, scripts, chat apps, cron&lt;/td&gt;
&lt;td&gt;&lt;a href="https://www.digitalocean.com/resources/article/what-is-openclaw" rel="noopener noreferrer"&gt;OpenClaw&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Self-hosted, model agnostic, can run sandboxed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Browser&lt;/td&gt;
&lt;td&gt;Log in, fill forms, read pages&lt;/td&gt;
&lt;td&gt;&lt;a href="https://themenonlab.blog/blog/lightpanda-headless-browser-ai-agents" rel="noopener noreferrer"&gt;Lightpanda&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Zig headless browser, speaks CDP, AGPL-3.0, beta&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Memory&lt;/td&gt;
&lt;td&gt;Remember clients, rules, preferences&lt;/td&gt;
&lt;td&gt;Mem0&lt;/td&gt;
&lt;td&gt;Memory layer with an API&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;OpenClaw had about 392K GitHub stars and Mem0 about 67K by mid October 2026 according to this &lt;a href="https://dev.to/rosgluk/trending-open-source-ai-projects-on-github-october-2026-4d42"&gt;DEV roundup&lt;/a&gt;. Lightpanda's own benchmark claims about 11x faster execution and 9x less memory than headless Chrome. Treat that as a vendor number and measure on your own workload.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Owner (WhatsApp / Slack / web dashboard)
        |
   Task intake  -&amp;gt;  Task queue (per customer)
                        |
               Agent worker (sandbox per customer)
               +----------------------------------+
               | OpenClaw runtime                 |
               |   brain: Claude API              |
               |   tools: files, scripts, email   |
               |   browser: Lightpanda over CDP   |
               |   memory: Mem0 (customer scope)  |
               +----------------------------------+
                        |
     Policy check (approve pay / delete / new recipient)
                        |
        Action log + screenshots -&amp;gt; owner's morning report
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step by step
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Pick one job with a clear finish line.&lt;/strong&gt; "Send yesterday's unpaid invoice reminders from the billing portal" is a good first job. "Run my business" is not.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Give each customer an isolated worker.&lt;/strong&gt; One container or VM per customer, with its own browser profile and credentials vault. Never share browser sessions across customers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Point the browser at Lightpanda.&lt;/strong&gt; Because it speaks the Chrome DevTools Protocol, existing Puppeteer or Playwright scripts can connect to it instead of launching Chrome. Keep a Chrome fallback for sites that Lightpanda's beta does not render correctly yet.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Playwright connecting to a CDP endpoint (Lightpanda or Chrome)&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;chromium&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;playwright&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;chromium&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connectOverCDP&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;CDP_URL&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;newPage&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;goto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;https://billing.example.com/invoices?status=unpaid&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;4. Scope memory per customer.&lt;/strong&gt; Store facts like "invoices go out Fridays" or "never book before 9 a.m." in Mem0 under a customer ID, and load them into the system prompt for each run.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Put a policy layer between the brain and the world.&lt;/strong&gt; The model proposes actions. Deterministic code decides whether they run, ask for approval, or are blocked. Payments, deletions and messages to new recipients always ask.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Log everything.&lt;/strong&gt; Every tool call, URL, screenshot and model response with token cost. The morning report the owner reads is generated from this log, and so is your eval set.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Cap spend per customer.&lt;/strong&gt; Set a monthly model budget in code. When a customer hits it, the agent pauses and notifies you instead of silently burning tokens.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where these projects usually break
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Login and 2FA.&lt;/strong&gt; Plan a human handoff for one-time codes from day one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sites that change.&lt;/strong&gt; Prefer stable selectors and text over coordinates, and alert on repeated failures.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runaway loops.&lt;/strong&gt; Limit steps per task and stop on repeated identical actions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Licensing.&lt;/strong&gt; Lightpanda is AGPL-3.0. Check what that means for how you deploy it before you sell a hosted product.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foadk76ybc5di9dunvm9w.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foadk76ybc5di9dunvm9w.jpg" alt="Bakery owner checking her AI employee's overnight work" width="800" height="448"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Pricing it
&lt;/h2&gt;

&lt;p&gt;A niche agent at $99 a month across 300 small offices would be $29,700 a month. That is a goal to plan against, and your gross margin depends almost entirely on model calls and how many browser sessions you can pack per server. That is why the memory footprint of the browser matters more than it looks.&lt;/p&gt;

&lt;p&gt;At &lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;Axionry&lt;/a&gt;, an AI product development studio, we build agent products like this, with sandboxing, approvals and audit logs, from demo to launch in a few weeks. If you are building one, &lt;a href="https://axionry.com/book" rel="noopener noreferrer"&gt;book a call&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What job would you give an AI employee first? Tell me in the comments.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>agents</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Cursor Sold for $60B and Lovable Hit $13.3B: What It Means for How We Build AI Products</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Fri, 30 Oct 2026 16:51:09 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/cursor-sold-for-60b-and-lovable-hit-133b-what-it-means-for-how-we-build-ai-products-3i68</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/cursor-sold-for-60b-and-lovable-hit-133b-what-it-means-for-how-we-build-ai-products-3i68</guid>
      <description>&lt;p&gt;In August 2026 SpaceX closed its acquisition of Anysphere, the company behind Cursor, for an implied $60 billion in stock (&lt;a href="https://en.wikipedia.org/wiki/Cursor_(company)" rel="noopener noreferrer"&gt;Wikipedia&lt;/a&gt;, &lt;a href="https://www.techzine.eu/news/devops/142197/spacex-acquires-cursor-for-60-billion/" rel="noopener noreferrer"&gt;Techzine&lt;/a&gt;). The same month Lovable raised $400 million at a $13.3 billion valuation (&lt;a href="https://dealroom.co/news/144597-lovable-raises-400m-at-13-3b-valuation-eyes-600m-revenue-run-rate/" rel="noopener noreferrer"&gt;Dealroom&lt;/a&gt;). In March, Replit raised $400 million at $9 billion (&lt;a href="https://www.trendingtopics.eu/replit-raises-400m-tripling-its-valuation-to-9-billion-in-six-months/" rel="noopener noreferrer"&gt;Trending Topics&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Those are headlines. What matters for developers is what the numbers say about where value is moving, and what that means for the things we choose to build.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcxxtcuu16myclswjhw7a.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcxxtcuu16myclswjhw7a.png" alt="Valuation and deal value of Cursor, Lovable and Replit" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers, with sources
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Company&lt;/th&gt;
&lt;th&gt;Revenue signal&lt;/th&gt;
&lt;th&gt;Latest capital event&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Cursor&lt;/td&gt;
&lt;td&gt;ARR of $100M in Jan 2025, $500M in Jun 2025, over $1B around Nov 2025, about $3B in May 2026&lt;/td&gt;
&lt;td&gt;Acquired by SpaceX, closed 14 Aug 2026, implied $60B&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lovable&lt;/td&gt;
&lt;td&gt;$200M ARR in Nov 2025; run rate near $600M expected by end of Aug 2026&lt;/td&gt;
&lt;td&gt;$400M Series C at $13.3B, Aug 2026&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Replit&lt;/td&gt;
&lt;td&gt;About $265M ARR at end of 2025; company targets $1B by end of 2026&lt;/td&gt;
&lt;td&gt;$400M at $9B, Mar 2026&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sources: Cursor figures from &lt;a href="https://en.wikipedia.org/wiki/Cursor_(company)" rel="noopener noreferrer"&gt;Wikipedia&lt;/a&gt; (citing Bloomberg and company statements); Lovable from &lt;a href="https://en.wikipedia.org/wiki/Lovable_(company)" rel="noopener noreferrer"&gt;Wikipedia&lt;/a&gt; and &lt;a href="https://dealroom.co/news/144597-lovable-raises-400m-at-13-3b-valuation-eyes-600m-revenue-run-rate/" rel="noopener noreferrer"&gt;Dealroom&lt;/a&gt;; Replit from &lt;a href="https://www.trendingtopics.eu/replit-raises-400m-tripling-its-valuation-to-9-billion-in-six-months/" rel="noopener noreferrer"&gt;Trending Topics&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl5oqdxznu3nykzrermy8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl5oqdxznu3nykzrermy8.png" alt="ARR growth of Cursor, Lovable and Replit" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two details stand out.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cursor went from $100M to roughly $3B ARR in about 16 months&lt;/strong&gt; with a team Wikipedia put at roughly 300 people in August 2025. That is an unusual revenue per head for a developer tool.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lovable started as an open source project.&lt;/strong&gt; Anton Osika first built GPT Engineer, an open source code generation tool, before turning it into the commercial product in late 2024 (&lt;a href="https://en.wikipedia.org/wiki/Lovable_(company)" rel="noopener noreferrer"&gt;Wikipedia&lt;/a&gt;). Open source core, product layer on top, then distribution. That pattern keeps repeating.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this tells us as builders
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Writing code is no longer the scarce part
&lt;/h3&gt;

&lt;p&gt;When the tools that generate code are worth tens of billions, the code itself is getting cheaper. A CRUD dashboard, an auth flow, a Stripe integration: these are now hours, not sprints. If your product's main value is "we wrote the software", you are competing with a prompt.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. The moat moved up the stack
&lt;/h3&gt;

&lt;p&gt;What is still hard, and therefore still valuable:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A specific workflow in a specific industry.&lt;/strong&gt; "AI for dental clinic billing" beats "AI app builder #47".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Proprietary context.&lt;/strong&gt; The documents, integrations and data your users already have, wired in properly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reliability.&lt;/strong&gt; Evals, fallbacks, observability and cost control for LLM calls. Demos are easy; the 95th percentile request is not.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Distribution.&lt;/strong&gt; Owning a channel to your buyer.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. Generic builders get you to a prototype, not always to production
&lt;/h3&gt;

&lt;p&gt;Lovable, Replit and Cursor are excellent for getting something working. The gaps usually show up in the same places:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;What tends to break&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Auth and roles&lt;/td&gt;
&lt;td&gt;Multi-tenant permissions, SSO for B2B buyers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data&lt;/td&gt;
&lt;td&gt;Migrations, backups, row-level security&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Payments&lt;/td&gt;
&lt;td&gt;Usage-based billing, proration, tax&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LLM features&lt;/td&gt;
&lt;td&gt;Prompt drift, no evals, runaway token cost&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ops&lt;/td&gt;
&lt;td&gt;Logging, alerting, background jobs, rate limits&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;None of these are exotic. They are just the boring 20% that decides whether a paying customer stays.&lt;/p&gt;

&lt;h2&gt;
  
  
  A practical architecture for an AI product in 2026
&lt;/h2&gt;

&lt;p&gt;Here is the shape we see working for small teams shipping fast:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Frontend (Next.js / React)  -&amp;gt;  API layer (typed, versioned)
                                   |
             +---------------------+----------------------+
             |                     |                      |
      Postgres (RLS)      LLM gateway (Claude +      Job queue
      + object store      fallback model, caching,   (retries,
                          per-tenant cost limits)    webhooks)
                                   |
                         Eval suite on real traces
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three choices that matter more than framework debates:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Put an LLM gateway in front of every model call.&lt;/strong&gt; One place for retries, caching, logging and per-customer spend caps. It turns model swaps into config changes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Store traces from day one.&lt;/strong&gt; Every prompt, response, latency and cost. Your eval set is built from these, and so is your answer when a customer says "it got worse".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep the AI feature narrow.&lt;/strong&gt; One job done reliably beats ten done at 70%.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Where to point your next build
&lt;/h2&gt;

&lt;p&gt;If code generation keeps getting cheaper, the opportunities are where AI meets a real workflow that someone pays for today: intake and scheduling, document review, outbound follow up, reporting, internal search over company knowledge. Pick one buyer, one painful job, and ship a version they can pay for in weeks.&lt;/p&gt;

&lt;p&gt;At &lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;Axionry&lt;/a&gt;, an AI product development studio, this is the work we do every day: taking an idea or a prototype from a tool like Lovable or Cursor and turning it into a production product with auth, billing, evals and monitoring. If you are sitting on a prototype that needs to become a business, &lt;a href="https://axionry.com/book" rel="noopener noreferrer"&gt;book a call&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What is the hardest part of taking your prototype to production? I read every comment.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>startup</category>
      <category>architecture</category>
      <category>saas</category>
    </item>
    <item>
      <title>Build an AI Receptionist With Whisper, Claude, Chatterbox and LiveKit: Architecture, Latency Budget and Starter Code</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Thu, 29 Oct 2026 14:06:44 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/build-an-ai-receptionist-with-whisper-claude-chatterbox-and-livekit-architecture-latency-budget-5eka</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/build-an-ai-receptionist-with-whisper-claude-chatterbox-and-livekit-architecture-latency-budget-5eka</guid>
      <description>&lt;p&gt;&lt;em&gt;This post is AI-assisted.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Most "AI receptionist" tutorials stop at a chatbot that talks. A receptionist a dental clinic will actually pay for has to pick up a real phone line, understand messy speech, follow the clinic's rules, book into a real calendar and do all of it fast enough that the caller never notices a gap.&lt;/p&gt;

&lt;p&gt;This is the architecture we use as a starting point, built from four pieces you can run yourself:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;License&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Phone line, streaming, turn taking&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/livekit/agents" rel="noopener noreferrer"&gt;LiveKit Agents&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Apache 2.0 (turn detector model under the LiveKit Model License)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Speech to text&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/openai/whisper" rel="noopener noreferrer"&gt;Whisper&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reasoning and tool calls&lt;/td&gt;
&lt;td&gt;Claude (Anthropic API)&lt;/td&gt;
&lt;td&gt;Paid API&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Text to speech, voice cloning&lt;/td&gt;
&lt;td&gt;&lt;a href="https://huggingface.co/ResembleAI/chatterbox" rel="noopener noreferrer"&gt;Chatterbox by Resemble AI&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Why bother
&lt;/h2&gt;

&lt;p&gt;Patient Prism's analysis of 11.5 million calls to 8,280 dental locations found 31% of callers hung up before reaching anyone (&lt;a href="https://www.patientprism.com/report/dental-patient-access-report/" rel="noopener noreferrer"&gt;report&lt;/a&gt;). Only 42% of consumers say they leave a voicemail (&lt;a href="https://www.callrail.com/blog/missed-calls-cost-businesses-more-than-ever" rel="noopener noreferrer"&gt;CallRail survey&lt;/a&gt;). Missed calls are lost bookings, which makes "answer every call" an easy thing to sell.&lt;/p&gt;

&lt;p&gt;The business math people quote is 100 clinics × $300 a month = $30,000 MRR. The engineering below is what makes that number survivable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PSTN caller
   │  SIP trunk
   ▼
LiveKit server (room per call)
   │  audio frames
   ▼
Agent worker (Python, LiveKit Agents)
   ├─ VAD + turn detector  → decides "caller finished"
   ├─ STT adapter          → faster-whisper on GPU (streaming chunks)
   ├─ LLM                  → Claude with tools: check_slots, book, take_message, transfer
   └─ TTS adapter          → Chatterbox (clinic's consented voice), streamed sentence by sentence
   │
   ▼
Clinic systems: calendar / practice software / CRM / SMS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Key design choices:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;One room per call.&lt;/strong&gt; LiveKit treats a phone call like any other room participant, so the agent joins the room the SIP call lands in. Scaling is "more worker processes", not a rewrite.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stream everything.&lt;/strong&gt; Do not wait for the full transcript, the full LLM answer or the full audio file. Start speaking on the first complete sentence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tools, not prompts, for anything that changes state.&lt;/strong&gt; The model never "says" it booked an appointment. It calls &lt;code&gt;book_appointment&lt;/code&gt;, your code books it, and the result goes back to the model.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Latency budget
&lt;/h2&gt;

&lt;p&gt;Callers start feeling awkward somewhere past a second of silence. Rough budget from "caller stops talking" to "agent audio starts":&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Target&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;End of turn detection&lt;/td&gt;
&lt;td&gt;200 to 400 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Final transcript chunk&lt;/td&gt;
&lt;td&gt;100 to 300 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude first tokens (short system prompt, cached rules)&lt;/td&gt;
&lt;td&gt;300 to 700 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;First sentence of TTS audio&lt;/td&gt;
&lt;td&gt;150 to 400 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These are targets to measure against, not guarantees. Measure each stage per call and log it. The biggest wins usually come from shorter prompts, streaming TTS by sentence and keeping the GPU workers warm.&lt;/p&gt;

&lt;h2&gt;
  
  
  Starter code shape
&lt;/h2&gt;

&lt;p&gt;LiveKit's starter template (&lt;code&gt;lk agent init my-agent --template agent-starter-python&lt;/code&gt;) gives you a working agent with hosted models. The API evolves quickly, so treat the snippet below as the shape of the code and check the current &lt;a href="https://docs.livekit.io/agents/start/voice-ai/" rel="noopener noreferrer"&gt;voice AI quickstart&lt;/a&gt; for exact signatures.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;livekit&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;agents&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;livekit.agents&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Agent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;AgentServer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;AgentSession&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;function_tool&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;adapters&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;WhisperSTT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ClaudeLLM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ChatterboxTTS&lt;/span&gt;  &lt;span class="c1"&gt;# your thin adapters
&lt;/span&gt;
&lt;span class="n"&gt;CLINIC_RULES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;You are the receptionist for Bright Smile Dental.
Hours: Mon to Sat, 9am to 6pm. Book cleanings, checkups and consultations only.
Never give medical advice. Transfer emergencies, billing disputes and upset callers to a human.
Keep answers under two sentences.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Receptionist&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Agent&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="nf"&gt;super&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;instructions&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;CLINIC_RULES&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nd"&gt;@function_tool&lt;/span&gt;
    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;check_slots&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Return free appointment slots for a date (YYYY-MM-DD).&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;calendar&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;free_slots&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nd"&gt;@function_tool&lt;/span&gt;
    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;book_appointment&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;phone&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;slot&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Book a slot and send an SMS confirmation.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
        &lt;span class="n"&gt;booking&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;calendar&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;book&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;phone&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;slot&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sms&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;phone&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Confirmed: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;slot&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; at Bright Smile Dental&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;booking&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;

&lt;span class="n"&gt;server&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AgentServer&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="nd"&gt;@server.rtc_session&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agent_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;receptionist&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;entrypoint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;agents&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;JobContext&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AgentSession&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;stt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;WhisperSTT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;large-v3&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;device&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cuda&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;ClaudeLLM&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
        &lt;span class="n"&gt;tts&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;ChatterboxTTS&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;voice_ref&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;voices/clinic_consented.wav&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;room&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;room&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;Receptionist&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate_reply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;instructions&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Greet the caller and ask how you can help.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;agents&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cli&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run_app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;server&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The adapters are small classes that implement LiveKit's STT, LLM and TTS interfaces around faster-whisper, the Anthropic SDK and the Chatterbox Python package. Writing them yourself keeps every model swappable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Production checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;SIP and numbers:&lt;/strong&gt; connect a SIP trunk to LiveKit and port or forward the clinic's existing number, so patients dial what they already know.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Barge-in:&lt;/strong&gt; let callers interrupt. Stop TTS immediately when the caller speaks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Transfer path:&lt;/strong&gt; a warm transfer to the front desk is a feature, not a failure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Voice consent:&lt;/strong&gt; only clone voices you have written permission for. Chatterbox watermarks every output, which helps with provenance.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Observability:&lt;/strong&gt; store transcript, tool calls and per-stage latency for every call. Your support tickets will thank you.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Data handling:&lt;/strong&gt; health and personal details need careful storage and retention rules in many countries. Decide this before onboarding the first clinic.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where the money actually is
&lt;/h2&gt;

&lt;p&gt;The models are commodities now. The value is in the boring parts: integrations with each niche's software, reliable transfers, and reporting that shows the owner how many calls were saved. Pick one niche, nail those three, and the $30K MRR math stops being a slide.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Axionry is an AI product development studio that builds production voice agents. See the &lt;a href="https://axionry.com/build/ai-receptionist" rel="noopener noreferrer"&gt;AI receptionist build&lt;/a&gt;, our &lt;a href="https://axionry.com/services/voice-ai-development" rel="noopener noreferrer"&gt;voice AI development&lt;/a&gt; service, or price your setup with the &lt;a href="https://axionry.com/tools/voice-ai-cost-calculator" rel="noopener noreferrer"&gt;voice AI cost calculator&lt;/a&gt; at &lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;axionry.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>voiceai</category>
      <category>opensource</category>
      <category>python</category>
    </item>
    <item>
      <title>n8n or Activepieces? Building a Multi-Tenant AI Automation Product Without Breaking the License</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Tue, 27 Oct 2026 13:10:56 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/n8n-or-activepieces-building-a-multi-tenant-ai-automation-product-without-breaking-the-license-5gh7</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/n8n-or-activepieces-building-a-multi-tenant-ai-automation-product-without-breaking-the-license-5gh7</guid>
      <description>&lt;p&gt;Everyone wants to build "Zapier for X" on an open-source engine right now. n8n just raised at a $2.5B valuation, AI agents need somewhere to run their tool calls, and every vertical (clinics, realtors, agencies) is begging for automations they don't have to configure themselves.&lt;/p&gt;

&lt;p&gt;I've shipped a few of these. The engineering is the easy half. The half that bites is a decision most tutorials skip: &lt;strong&gt;will your customers build workflows, or only benefit from them?&lt;/strong&gt; That single answer decides your license, your architecture and your margins.&lt;/p&gt;

&lt;p&gt;This post walks through both architectures, the license line for n8n and Activepieces, and the parts of multi-tenancy nobody mentions until production.&lt;/p&gt;

&lt;h2&gt;
  
  
  The license line, in one table
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;n8n&lt;/th&gt;
&lt;th&gt;Activepieces CE&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;License&lt;/td&gt;
&lt;td&gt;Sustainable Use License (fair-code)&lt;/td&gt;
&lt;td&gt;MIT (paid features in &lt;code&gt;packages/ee/&lt;/code&gt; are commercial)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Run your own business on it&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Build workflows for clients on your instance, charge for it&lt;/td&gt;
&lt;td&gt;Yes, if clients get outputs, not access&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hidden backend, end users never configure workflows&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Customers build or edit their own workflows in your product&lt;/td&gt;
&lt;td&gt;Needs an Enterprise license&lt;/td&gt;
&lt;td&gt;MIT core allows it; embedding their builder UI is a paid deal&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fork and rebrand as your own automation product&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;MIT allows forking; keep the notice; paid features excluded&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sources: the &lt;a href="https://docs.n8n.io/n8n-community-license/community-license/license-faq" rel="noopener noreferrer"&gt;n8n license FAQ&lt;/a&gt; states that free use applies when "workflows are only created or modified by you or people in your organization," and that letting external users build or configure workflows (including through your own UI, API, MCP or an AI agent) is not allowed without an Enterprise license. Activepieces documents its &lt;a href="https://www.activepieces.com/blog/activepieces-open-source-license-what-is-free-2026" rel="noopener noreferrer"&gt;MIT core and commercial &lt;code&gt;ee&lt;/code&gt; folder&lt;/a&gt;. Not legal advice; get it in writing from the vendor for anything customer-facing.&lt;/p&gt;

&lt;p&gt;So:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pattern A, "Outcomes as a service":&lt;/strong&gt; you design the workflows, customers just turn them on. n8n works great here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern B, "Customers build":&lt;/strong&gt; customers drag their own steps. Start from Activepieces CE or budget for an n8n Enterprise license.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Pattern A: n8n as a hidden engine
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiw9sr0eugt3kpbdlutqt.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiw9sr0eugt3kpbdlutqt.png" alt="Pattern A architecture" width="799" height="330"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Your app owns the UI, tenants, billing and credentials. n8n only ever talks to your backend.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Customer UI (your app)
   |  toggles "Missed-call text back" for Clinic #42
   v
Your API  ──&amp;gt;  Template registry (versioned workflow JSON)
   |                 |
   |   instantiate with tenant params (phone, calendar id, timezone)
   v                 v
n8n (queue mode: main + workers + Redis)  &amp;lt;──  webhook in from Twilio/Calendar
   |
   v
Your API callback  ──&amp;gt;  tenant DB (Postgres, row-level security)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Key implementation points:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Templates, not ad-hoc workflows.&lt;/strong&gt; Store each automation as versioned workflow JSON in your repo. When a tenant enables it, your API creates (or updates) a workflow through the n8n API with tenant-specific values injected. Never let tenant-specific edits drift outside version control.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Tenant context travels in every execution.&lt;/strong&gt; Pass &lt;code&gt;tenant_id&lt;/code&gt; in the trigger payload and keep it on every downstream call. Your callbacks validate it against a signed token, not trust it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Credentials never live in n8n per tenant if you can avoid it.&lt;/strong&gt; Keep OAuth tokens in your own vault (encrypted columns or a secrets manager) and have the workflow call your API to perform the action. n8n then needs one credential: your internal API key. This keeps tenant offboarding to a single &lt;code&gt;DELETE&lt;/code&gt; in your database.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Run queue mode from day one.&lt;/strong&gt; A single n8n process will stall when one tenant's workflow loops. Main instance plus workers plus Redis lets you scale workers horizontally and isolate noisy tenants onto their own worker pool later.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Idempotency on every webhook.&lt;/strong&gt; Phone systems and calendars retry. Hash the event ID and drop duplicates before the workflow runs, or you'll text a patient three times.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern B: customers build their own flows
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5jf8lqf9z90lnog5cptb.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5jf8lqf9z90lnog5cptb.png" alt="Pattern B architecture" width="799" height="268"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If the product &lt;em&gt;is&lt;/em&gt; the builder, you inherit everything Zapier solves:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tenant isolation:&lt;/strong&gt; separate projects or namespaces per tenant, and an enforcement layer in your API so one tenant can't trigger another's flows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rate limits per tenant:&lt;/strong&gt; executions per minute and per month, enforced before the engine, not after.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Connector sandboxing:&lt;/strong&gt; custom code steps run untrusted code. Run them in isolated workers with CPU, memory and network egress limits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Usage metering for billing:&lt;/strong&gt; count task runs per tenant from execution logs, push them to your billing system nightly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is real platform engineering. Price it accordingly, or start with Pattern A and graduate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where AI agents fit
&lt;/h2&gt;

&lt;p&gt;The fun part: an LLM step is just another node. The patterns that hold up in production:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Classify, then act.&lt;/strong&gt; Use a small, cheap model call to classify the incoming event ("reschedule request", "complaint", "new lead"), then route to deterministic steps. Don't let a model free-run your customer's calendar.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tool calls go through your API.&lt;/strong&gt; If an agent can book, cancel or refund, those are your endpoints with your validation and audit log, never direct third-party calls from the model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Human-in-the-loop for money and messages.&lt;/strong&gt; Anything that sends money or an outbound message on a customer's behalf gets an approval step until the error rate is proven.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Log the prompt, the output and the decision&lt;/strong&gt; per tenant. When a clinic asks "why did it text that patient?", you need the answer in one query.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Cost sketch for 50 tenants (Pattern A)
&lt;/h2&gt;

&lt;p&gt;Rough monthly numbers for a modest deployment, illustrative only:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;n8n main + 2 workers + Redis on small instances: low hundreds of dollars.&lt;/li&gt;
&lt;li&gt;Managed Postgres: tens to low hundreds.&lt;/li&gt;
&lt;li&gt;LLM classification calls: cents per tenant per day if you use a small model and cache aggressively.&lt;/li&gt;
&lt;li&gt;SMS and telephony: usually your biggest variable cost, pass it through.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At $199 per tenant per month, 50 tenants is roughly $10K MRR on a stack with no license fees for Pattern A. The expensive line item is engineering time for onboarding integrations, not compute.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist before you write code
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Decide Pattern A or B, and confirm the license in writing for B.&lt;/li&gt;
&lt;li&gt;[ ] Versioned workflow templates in Git.&lt;/li&gt;
&lt;li&gt;[ ] Tenant credentials in your vault, not the engine.&lt;/li&gt;
&lt;li&gt;[ ] Queue mode and per-tenant rate limits.&lt;/li&gt;
&lt;li&gt;[ ] Idempotent webhooks.&lt;/li&gt;
&lt;li&gt;[ ] LLM steps classify, your API acts, humans approve money and messages.&lt;/li&gt;
&lt;li&gt;[ ] Per-tenant execution logs for support and billing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Get those right and "Zapier for clinics" stops being a weekend demo and becomes a business.&lt;/p&gt;




&lt;p&gt;I run &lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;Axionry&lt;/a&gt;, an AI product development studio. We build exactly these products: &lt;a href="https://axionry.com/build/custom-ai-agent" rel="noopener noreferrer"&gt;custom AI agents&lt;/a&gt;, multi-tenant automation platforms and vertical SaaS MVPs. If you want a second pair of eyes on your architecture or license choice, &lt;a href="https://axionry.com/book" rel="noopener noreferrer"&gt;book a call&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>automation</category>
      <category>saas</category>
    </item>
    <item>
      <title>Your Voice Agent Needs a Flight Simulator Before It Gets a Phone Number</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Mon, 26 Oct 2026 10:16:43 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/your-voice-agent-needs-a-flight-simulator-before-it-gets-a-phone-number-4cnd</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/your-voice-agent-needs-a-flight-simulator-before-it-gets-a-phone-number-4cnd</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvnh30yge54khxi3rbxmv.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvnh30yge54khxi3rbxmv.gif" alt="Animated: a voice agent simulator running 240 test calls and catching one forbidden promise" width="720" height="405"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Pilots spend hours in a simulator before they ever carry a passenger. Most voice agents go live after the founder makes five test calls from a quiet room.&lt;/p&gt;

&lt;p&gt;That gap is turning into a product category. This week The Times of India profiled Cekura, a Y Combinator-backed startup that tests and monitors voice agents for banks and lenders, catching agents that mishear callers or make promises they are not allowed to make (&lt;a href="https://timesofindia.indiatimes.com/city/bhubaneswar/ai-calling-ensuring-reliability-when-a-voice-agent-calls/articleshow/135193575.cms" rel="noopener noreferrer"&gt;Times of India&lt;/a&gt;). A week earlier, monday.com launched Harmony, a voice platform for sales calls that ships with tools to simulate changes and test alternatives before rollout (&lt;a href="https://www.unite.ai/monday-com-harmony-voice-ai-sales-agents/" rel="noopener noreferrer"&gt;Unite.AI&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;You can buy this, and at scale you probably should. But the core idea fits in a weekend, and building it once teaches you what to look for. Here is the simulator I would build.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why voice is harder to test than chat
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Errors compound.&lt;/strong&gt; Speech to text hears "fifty" as "fifteen", the model reasons perfectly on the wrong number, and the caller gets a confident wrong answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Timing is part of correctness.&lt;/strong&gt; Talking over the caller, or leaving four seconds of silence, fails the call even if every word was right.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Calls are long and branchy.&lt;/strong&gt; A bug on turn nine only shows up if the caller says the right thing on turn three.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nothing is deterministic.&lt;/strong&gt; The same prompt can pass ten times and fail on the eleventh.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So a handful of manual calls tells you almost nothing. You need volume, variety and a score.&lt;/p&gt;

&lt;h2&gt;
  
  
  The simulator in four parts
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;A &lt;strong&gt;scenario matrix&lt;/strong&gt;: who is calling, what they want, and what makes it hard.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;simulated caller&lt;/strong&gt;: a model that plays the customer, speaking through real audio.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scoring&lt;/strong&gt;: hard checks first, then a judge model with a rubric.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;regression gate&lt;/strong&gt; in CI, fed by real production calls.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  1. The scenario matrix
&lt;/h2&gt;

&lt;p&gt;Write scenarios as data, then multiply them. Ten intents, six personas and four conditions give you 240 distinct calls from a few dozen lines. Even the short lists below already produce 64.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;itertools&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;product&lt;/span&gt;

&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Scenario&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;intent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;        &lt;span class="c1"&gt;# what the caller wants
&lt;/span&gt;    &lt;span class="n"&gt;persona&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;       &lt;span class="c1"&gt;# how they behave
&lt;/span&gt;    &lt;span class="n"&gt;condition&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;     &lt;span class="c1"&gt;# what makes it hard
&lt;/span&gt;    &lt;span class="n"&gt;must&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;    &lt;span class="c1"&gt;# outcomes the agent must reach
&lt;/span&gt;    &lt;span class="n"&gt;must_not&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;  &lt;span class="c1"&gt;# things the agent must never do
&lt;/span&gt;
&lt;span class="n"&gt;INTENTS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;reschedule payment&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ask about late fee&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dispute a charge&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ask for a human&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;PERSONAS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;calm and brief&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;angry and interrupts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;elderly and slow&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;non-native speaker&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;CONDITIONS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;quiet line&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;street noise&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;says numbers fast&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;changes mind halfway&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;build_matrix&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;intent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;persona&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;condition&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;product&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;INTENTS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;PERSONAS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;CONDITIONS&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="nc"&gt;Scenario&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;intent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;intent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;persona&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;persona&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;condition&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;condition&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;must&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;confirm identity&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;summarize next step&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="n"&gt;must_not&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;promise a fee waiver&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;quote a rate not in policy&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;must_not&lt;/code&gt; list is where most real damage hides. Write it with whoever owns the business rules, not just the engineers.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. The simulated caller
&lt;/h2&gt;

&lt;p&gt;The caller is another model with a hidden goal and a personality. Its text goes through text to speech, gets noise mixed in for the condition, and reaches your agent over the &lt;strong&gt;same transport your real calls use&lt;/strong&gt;: your WebSocket or SIP endpoint, not a shortcut into the prompt. If you skip the audio path, you skip half the bugs.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;CALLER_PROMPT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;You are a customer phoning a lender.
Goal: {intent}. Personality: {persona}.
Stay in character. Reveal details only when asked.
If the agent asks something unclear, say so like a real person would.
End the call when your goal is met or after 12 turns.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scenario&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agent_endpoint&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;caller&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LLM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;CALLER_PROMPT&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="nf"&gt;vars&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scenario&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
    &lt;span class="n"&gt;call&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agent_endpoint&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;            &lt;span class="c1"&gt;# same path as production
&lt;/span&gt;    &lt;span class="n"&gt;transcript&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="n"&gt;agent_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;listen&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;                &lt;span class="c1"&gt;# agent greets first
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;turn&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;transcript&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agent_text&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;reply&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;caller&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;respond&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agent_text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;audio&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;add_noise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;tts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reply&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;scenario&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;condition&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;sent_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;agent_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;say_and_listen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;audio&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;timings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;sent_at&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;             &lt;span class="c1"&gt;# caller stops, agent starts
&lt;/span&gt;        &lt;span class="n"&gt;transcript&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;caller&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reply&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;caller&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;break&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;transcript&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timings&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  3. Scoring: hard checks before the judge
&lt;/h2&gt;

&lt;p&gt;Run cheap, deterministic checks first. They are fast, exact and impossible to argue with.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;FORBIDDEN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;waive&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;guarantee&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;definitely approved&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no fee&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;hard_checks&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scenario&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;transcript&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timings&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;agent_lines&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;who&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;transcript&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;who&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no_forbidden_promises&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;agent_lines&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;FORBIDDEN&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;human_when_asked&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;scenario&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;intent&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ask for a human&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;transfer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;agent_lines&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;p95_response_ms_ok&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;p95&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timings&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;1200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ended_cleanly&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;transcript&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then a judge model scores what string matching cannot: did the agent actually solve the caller's problem, did it confirm the right numbers, was it polite under pressure. Ask for a score and a quote from the transcript as evidence, so you can audit the judge.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;JUDGE_RUBRIC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Score the agent from 1 to 5 on each item and quote the line that justifies it:
- goal_met: did the caller get what they called for?
- facts_confirmed: were names, dates and amounts read back correctly?
- recovery: when misunderstood, did the agent recover gracefully?
- tone: calm, clear and respectful throughout?
Return JSON.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A call passes only if every hard check passes and every judge score is 4 or above.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. The regression gate
&lt;/h2&gt;

&lt;p&gt;Every prompt edit, model swap or voice change is a deploy. Run the matrix on each pull request and block the merge if the pass rate drops.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# .github/workflows/voice-sim.yml&lt;/span&gt;
&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pull_request&lt;/span&gt;
&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;simulate&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v4&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;python sim/run.py --scenarios 240 --concurrency 20 --out results.json&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;python sim/gate.py results.json --min-pass-rate 0.92 --max-drop 0.02 --critical no_forbidden_promises&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Treat &lt;code&gt;no_forbidden_promises&lt;/code&gt; as critical: a single failure fails the build, regardless of the overall pass rate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Close the loop with real calls
&lt;/h2&gt;

&lt;p&gt;Simulated callers are only as creative as you made them. Real callers are more creative. Sample a few percent of production calls, run the same hard checks and judge on their transcripts, and turn every failure into a new scenario. After a month, your matrix reflects what customers actually say, not what you guessed they would say.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to watch
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Why it matters&lt;/th&gt;
&lt;th&gt;A sane starting target&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Pass rate on the matrix&lt;/td&gt;
&lt;td&gt;Overall health per release&lt;/td&gt;
&lt;td&gt;92% or higher&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Critical rule failures&lt;/td&gt;
&lt;td&gt;Promises, wrong amounts&lt;/td&gt;
&lt;td&gt;Zero&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p95 time to first response&lt;/td&gt;
&lt;td&gt;Feels human or robotic&lt;/td&gt;
&lt;td&gt;Under 1.2 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Talk-over events per call&lt;/td&gt;
&lt;td&gt;Interrupting the caller&lt;/td&gt;
&lt;td&gt;Under 1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Escaped failures&lt;/td&gt;
&lt;td&gt;Bugs found in production first&lt;/td&gt;
&lt;td&gt;Falling every week&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The agents that survive real customers are not the ones with the cleverest prompt. They are the ones that have already taken ten thousand bad calls in a simulator before taking their first real one.&lt;/p&gt;

&lt;p&gt;We build and test voice agents at &lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;Axionry&lt;/a&gt;, an AI product development studio. See how we approach &lt;a href="https://axionry.com/services/voice-ai-development" rel="noopener noreferrer"&gt;voice AI development&lt;/a&gt;, or estimate what your agent will cost per minute with the &lt;a href="https://axionry.com/tools/voice-ai-cost-calculator" rel="noopener noreferrer"&gt;voice AI cost calculator&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>python</category>
      <category>voiceai</category>
    </item>
    <item>
      <title>Stop Spending a Full LLM Call on a Yes or No Question: OpenAI's Decisions API, Explained</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Sun, 25 Oct 2026 21:05:52 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/stop-spending-a-full-llm-call-on-a-yes-or-no-question-openais-decisions-api-explained-31c1</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/stop-spending-a-full-llm-call-on-a-yes-or-no-question-openais-decisions-api-explained-31c1</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj18m9ejbqxbstualkhe2.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj18m9ejbqxbstualkhe2.jpg" alt="An AI agent choosing one path from a short list with a confidence score" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Count the decisions inside any agent you have shipped. Is this ticket billing or technical? Should the agent retry, continue or ask for clarification? Is this image damaged? Is the caller angry enough to escalate?&lt;/p&gt;

&lt;p&gt;None of these need a paragraph of reasoning. They need one answer from a short list. Yet most of us send them through the same chat completion endpoint we use for writing emails, pay for the output tokens, parse the JSON, and add a retry for when the model gets creative.&lt;/p&gt;

&lt;p&gt;On October 22, OpenAI made the Decisions API generally available, and it is built for exactly this (&lt;a href="https://openai.com/index/decisions-api/" rel="noopener noreferrer"&gt;OpenAI&lt;/a&gt;). Here is what it does, the three answer types, and the patterns worth stealing even if you never call it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it is
&lt;/h2&gt;

&lt;p&gt;Decisions runs on GPT-6 Luna, but instead of generating text it answers questions with bounded answers. OpenAI says it is up to 10x faster than asking Luna the same thing through the Responses API. Since the beta it has gained prompt caching, native JSON inputs, images inside instructions, up to 400 questions per request, and US and EU inference residency.&lt;/p&gt;

&lt;p&gt;The pricing is the interesting part: &lt;strong&gt;$0.10 per million input tokens, $0.01 cached, and no output token charges at all.&lt;/strong&gt; For a router that reads a 600 token ticket, that is about six thousandths of a cent per decision before caching.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three answer types
&lt;/h2&gt;

&lt;p&gt;You define a question and its allowed answers, pass context as text, images or JSON, and get back one of three result shapes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Use it for&lt;/th&gt;
&lt;th&gt;You get&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;predicate&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Is a condition true? ("Does this photo show visible damage?")&lt;/td&gt;
&lt;td&gt;A probability from 0 to 1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;choice&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Pick one of your options ("continue", "retry", "ask")&lt;/td&gt;
&lt;td&gt;The choice, plus probabilities and confidence&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;score&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Rate against an ordered rubric ("damage severity 0 to 4")&lt;/td&gt;
&lt;td&gt;A probability-weighted average of the level index&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The shape matters more than it looks. A choice with probabilities is something you can &lt;strong&gt;threshold&lt;/strong&gt;. A score that is a weighted average is something you can &lt;strong&gt;sort&lt;/strong&gt;. A free-text answer from a chat model is neither until you parse it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern 1: the confidence gate
&lt;/h2&gt;

&lt;p&gt;The most useful thing you get back is not the answer. It is how sure the model is. Box shared that in its evaluation the Decisions API picked routes about 5x faster on certificate of insurance documents with the same accuracy as Responses, and sent lower-confidence cases to human reviewers.&lt;/p&gt;

&lt;p&gt;That is the pattern: cheap decision first, expensive path only when needed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ticket&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;decide&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;  &lt;span class="c1"&gt;# thin wrapper around the Decisions endpoint
&lt;/span&gt;        &lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;choice&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;queue&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;instructions&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Which team should handle this support ticket?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;choices&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;billing&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;technical&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;account&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sales&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ticket&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# native JSON, no prompt templating
&lt;/span&gt;    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;confidence&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.85&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choice&lt;/span&gt;                  &lt;span class="c1"&gt;# fast path, most traffic
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;confidence&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.55&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;escalate_to_large_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ticket&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hint&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choice&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# slow path
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;human_review&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;                &lt;span class="c1"&gt;# the model knows it does not know
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tune the two thresholds on a labelled sample, not by feel. The first one controls cost, the second controls how much lands on people.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern 2: batch the questions you were asking one by one
&lt;/h2&gt;

&lt;p&gt;Up to 400 questions per request means you can stop making serial calls for related checks. Moderating a listing? Ask "contains contact details?", "price looks like a typo?", "photo matches category?" and "language is English?" in one request against one cached context. One round trip instead of four, and the shared context is billed once.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;checks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;predicate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;has_contact_info&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;instructions&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Does the listing text include a phone number or email?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;predicate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price_suspicious&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;instructions&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Is the price implausible for this item category?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;score&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;photo_quality&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;instructions&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Rate photo quality.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
     &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;levels&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unusable&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;poor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fine&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;good&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;excellent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]},&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;decide_many&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;checks&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;listing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Pattern 3: decisions inside a live voice call
&lt;/h2&gt;

&lt;p&gt;This is the part I find most exciting. Decisions now plugs into GPT-Live sessions as a trigger on transcript events. While the caller is still speaking, the app can decide whether to ask a clarifying question, route, or escalate, without pausing the conversation to run a full model turn. OpenAI's demo uses it to pick an avatar's facial expression from the live transcript:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"decisions"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"event_type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"session.input_transcript.delta"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"decisions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"questions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"choice"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"reaction"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"instructions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Choose an appropriate reaction for the avatar based on the user's latest message."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"choices"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"neutral"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"happy"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"sad"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"concerned"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Swap "reaction" for &lt;code&gt;escalate&lt;/code&gt;, &lt;code&gt;needs_human&lt;/code&gt; or &lt;code&gt;intent&lt;/code&gt;, and you have the side channel every voice agent needs: a continuous, cheap read on the conversation that never interrupts it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it does not fit
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Open answers.&lt;/strong&gt; If the answer is a sentence, a plan or code, use a generation model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decisions that need tools.&lt;/strong&gt; Decisions answers from the context you send. If it needs to look something up first, do the lookup, then decide.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unlabelled guesses.&lt;/strong&gt; A confidence number is only meaningful once you have checked it against real outcomes. Log every decision with its confidence and the eventual truth, and recalibrate monthly.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The bigger lesson
&lt;/h2&gt;

&lt;p&gt;Most agent cost is not in the clever steps. It is in the dozens of small, boring classifications around them. Pulling those out into a dedicated, structured, thresholded layer is good architecture whichever vendor you use: it makes agents faster, cheaper and easier to debug, because every routing choice now has a name, a confidence and a log line.&lt;/p&gt;

&lt;p&gt;We design agent architectures like this at &lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;Axionry&lt;/a&gt;, an AI product development studio. If you are planning an agent and want to know what it will cost to run, try our &lt;a href="https://axionry.com/tools/ai-product-cost-estimator" rel="noopener noreferrer"&gt;AI product cost estimator&lt;/a&gt;, or see how we build on our &lt;a href="https://axionry.com/services/ai-product-development" rel="noopener noreferrer"&gt;AI product development&lt;/a&gt; page.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>openai</category>
      <category>agents</category>
      <category>python</category>
    </item>
    <item>
      <title>Your Voice Agent Isn't Slow. It's Silent. Non-Blocking Tools, Explained.</title>
      <dc:creator>Neeraj Sharma</dc:creator>
      <pubDate>Sun, 25 Oct 2026 01:58:30 +0000</pubDate>
      <link>https://dev.to/neeraj_sharma_757fbb49aa7/your-voice-agent-isnt-slow-its-silent-non-blocking-tools-explained-973</link>
      <guid>https://dev.to/neeraj_sharma_757fbb49aa7/your-voice-agent-isnt-slow-its-silent-non-blocking-tools-explained-973</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs09m59b7sgz721o23pfu.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs09m59b7sgz721o23pfu.jpg" alt="A voice agent that keeps talking while background tools run" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In a chat UI, a 13 second database query gets a spinner and nobody complains. In a phone call, the same 13 seconds is dead air, and dead air sounds like a dropped call. People hang up.&lt;/p&gt;

&lt;p&gt;Google's ADK team published a guide this week that puts a number on it. In their sales report benchmark on &lt;code&gt;gemini-3.8-live&lt;/code&gt;, switching one tool from blocking to streaming cut the time to the agent's first words from &lt;strong&gt;14.4 seconds to 1.2 seconds&lt;/strong&gt; (&lt;a href="https://developers.googleblog.com/step-by-step-guide-build-a-voice-agent-that-never-goes-silent-using-adk/" rel="noopener noreferrer"&gt;Google Developers Blog, Oct 19 2026&lt;/a&gt;). The query still took 13 seconds. The caller just stopped noticing.&lt;/p&gt;

&lt;p&gt;This post walks through why voice agents go silent, the two patterns that fix it, and the production details that bite you once real callers show up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why prompting doesn't fix it
&lt;/h2&gt;

&lt;p&gt;The instinct is to add "tell the user you are working on it" to the system prompt. It doesn't work, and the reason is mechanical. A normal tool is a coroutine that returns once:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;query_sales_database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Runs the quarterly sales report for a region.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;report&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;run_expensive_query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# ~13 seconds
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;success&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;report&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;report&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In a live bidirectional session, the model's turn is waiting on that function response. It cannot speak the "one moment" line, because it has not been given control back yet. The fix has to be architectural: the tool must hand control back before the work is done.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern 1: stream it when the wait has a story
&lt;/h2&gt;

&lt;p&gt;If the job has real stages (connect, scan, aggregate, format), make the tool an async generator. In ADK, changing &lt;code&gt;return&lt;/code&gt; to &lt;code&gt;yield&lt;/code&gt; is enough. The framework marks the declaration as non-blocking and immediately sends the model an acknowledgement, so the agent can talk while the work runs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AsyncGenerator&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;query_sales_database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;AsyncGenerator&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Runs the quarterly sales report. Call this ONCE; it pushes updates.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Connecting to the sales warehouse&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.5&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Scanning order records&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;3.5&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Aggregating revenue by product line&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;3.5&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Formatting the report&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.5&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PROGRESS: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;DONE: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; revenue is up 12.4% year over year.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each &lt;code&gt;yield&lt;/code&gt; arrives at the model as a partial function response, and the model paraphrases it ("Okay, I'm into the warehouse, pulling the order records now"). The caller hears progress instead of silence. If they ask an unrelated question mid-query, the agent answers it without cancelling the background task.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern 2: background it when the wait is just a wait
&lt;/h2&gt;

&lt;p&gt;Narrating every stage gets annoying when the user has already moved on. If there is nothing worth saying, yield exactly once, at the end:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;query_sales_database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;AsyncGenerator&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Runs the quarterly sales report. Call this ONCE; results arrive when ready.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;run_expensive_query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because it is still a generator, the model gets an immediate pending signal, says one line ("I'm pulling that together now"), and stays quiet until the result lands.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deciding when the answer is spoken
&lt;/h2&gt;

&lt;p&gt;Execution shape and delivery timing are separate knobs. ADK exposes the second one as &lt;code&gt;response_scheduling&lt;/code&gt; on a &lt;code&gt;FunctionTool&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th&gt;What happens when the result arrives&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;WHEN_IDLE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Waits for the current turn to finish, then delivers at the next natural pause&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;INTERRUPT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Cuts into current speech immediately (save this for urgent results)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SILENT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Added to context only; the agent uses it if the topic comes up&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google.adk.tools&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;FunctionTool&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google.genai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;types&lt;/span&gt;

&lt;span class="n"&gt;sales_tool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;FunctionTool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query_sales_database&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;sales_tool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;response_scheduling&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FunctionResponseScheduling&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WHEN_IDLE&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In Google's benchmark, a report arriving under &lt;code&gt;WHEN_IDLE&lt;/code&gt; waited politely while the model finished a separate 20 second answer, then slotted in. Under &lt;code&gt;INTERRUPT&lt;/code&gt; it cut the model off mid sentence. Streaming tools can even choose per update: keep intermediate progress &lt;code&gt;SILENT&lt;/code&gt;, deliver the final figure &lt;code&gt;WHEN_IDLE&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The rule of thumb from the guide is worth memorising: &lt;strong&gt;if the wait has a story, stream it. If the wait is just a wait, background it.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The production details that bite
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Duplicate calls.&lt;/strong&gt; Because a streaming tool spans several turns, a model that hears the user repeat themselves may call it again. A docstring that says "Call this ONCE" helps, but a latch is what actually protects you, especially when the tool does something with side effects like booking a slot:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;_inflight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Task&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;book_slot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;slot&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;AsyncGenerator&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;slot&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;_inflight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ALREADY_RUNNING: booking for this slot is in progress&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="n"&gt;task&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create_task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;_do_booking&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;slot&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;_inflight&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PROGRESS: checking the calendar&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;DONE: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;_inflight&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In a multi-instance deployment, move that key to Redis with a TTL, or pass it to the downstream API as an idempotency key.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Barge-in does not cancel tools.&lt;/strong&gt; When the caller talks over the agent, the audio stops and you get an &lt;code&gt;interrupted&lt;/code&gt; event, but running tools keep running and their results still arrive. If you want "actually, never mind" to stop the work, give the agent an explicit cancel tool. ADK has a built-in &lt;code&gt;stop_streaming(function_name)&lt;/code&gt; that cancels the generator's task and raises &lt;code&gt;CancelledError&lt;/code&gt; inside it, so put your connection cleanup in a &lt;code&gt;finally&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Narration lags.&lt;/strong&gt; The model paraphrases updates as they arrive, and narration can trail the tool by several seconds. Don't emit a progress line every 200 ms. Three or four meaningful stages beat twenty tiny ones.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Measure the right number.&lt;/strong&gt; Backend latency did not change in any of this. What changed is time to first audio and the length of the longest silence in the call. Log both per call. If you only watch p95 tool latency, you will never see the problem your callers hang up on.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same idea outside ADK
&lt;/h2&gt;

&lt;p&gt;None of this is ADK specific. On any realtime stack (OpenAI Realtime compatible endpoints, Pipecat, LiveKit Agents), the pattern is the same: return an immediate acknowledgement to the model, run the work as a task, push results back into the session as they arrive, and decide whether each one should interrupt, wait or stay silent.&lt;/p&gt;

&lt;p&gt;A web page that takes ten seconds feels slow. A voice agent that goes quiet for ten seconds feels broken. That gap is the whole reason this pattern exists.&lt;/p&gt;

&lt;p&gt;We build voice agents and AI products end to end at &lt;a href="https://axionry.com" rel="noopener noreferrer"&gt;Axionry&lt;/a&gt;, an AI product development studio. If you are planning one, our &lt;a href="https://axionry.com/services/voice-ai-development" rel="noopener noreferrer"&gt;voice AI development&lt;/a&gt; work starts with exactly this kind of latency budget, and the &lt;a href="https://axionry.com/tools/voice-ai-cost-calculator" rel="noopener noreferrer"&gt;voice AI cost calculator&lt;/a&gt; will tell you what a minute of it costs.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>agents</category>
      <category>architecture</category>
    </item>
  </channel>
</rss>
