<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: AI Frontier Post</title>
    <description>The latest articles on DEV Community by AI Frontier Post (@aifrontierpost).</description>
    <link>https://dev.to/aifrontierpost</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4136092%2F4219bbb7-1e69-43db-a4fe-088c237a49d8.png</url>
      <title>DEV Community: AI Frontier Post</title>
      <link>https://dev.to/aifrontierpost</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/aifrontierpost"/>
    <language>en</language>
    <item>
      <title>Microsoft's 'Lazarus' backdoor hides in fewer than 10 dormant neurons — and needs no retraining</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Wed, 11 Nov 2026 15:04:46 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/microsofts-lazarus-backdoor-hides-in-fewer-than-10-dormant-neurons-and-needs-no-retraining-5fag</link>
      <guid>https://dev.to/aifrontierpost/microsofts-lazarus-backdoor-hides-in-fewer-than-10-dormant-neurons-and-needs-no-retraining-5fag</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/microsoft-lazarus-dormant-neuron-backdoor/" rel="noopener noreferrer"&gt;AI Frontier Post&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Microsoft researchers have shown that an AI agent can be turned into a sleeper agent by editing fewer than ten neurons in the model underneath it — no retraining, no poisoned data, and no failed safety test to give it away. The technique, published on Monday as “Lazarus,” is a warning about what model weights are worth: once someone can rewrite a checkpoint, they may not need to touch anything else.&lt;/p&gt;

&lt;p&gt;The work comes from Microsoft Research, led by principal researcher Dimitris Papailiopoulos. The idea is simple enough to state and unsettling enough to matter: every large language model has neurons that almost never fire. Reprogram a handful of them, and the model behaves normally — until a trigger appears.&lt;/p&gt;

&lt;h2&gt;
  
  
  How Lazarus works
&lt;/h2&gt;

&lt;p&gt;A dormant neuron is one that rarely fires during normal use, so changing it barely affects the model’s ordinary answers. Lazarus repurposes those neurons as trigger detectors: each one is edited so it fires only when a chosen trigger shows up — a phrase in a user request, a string buried in tool output, or something as mundane as a date. Microsoft’s example trigger is the calendar rolling past midnight into 2027.&lt;/p&gt;

&lt;p&gt;Once a reprogrammed neuron fires, it forces a token of the attacker’s choosing into the model’s output. One neuron buys one token, so a short command needs only a handful of neurons chained together. The agent then runs the command with whatever permissions it already has — downloading and running a remote script, for instance — while the model keeps passing the standard evaluations that are supposed to catch exactly this.&lt;/p&gt;

&lt;p&gt;Crucially, nothing else changes. The new neuron settings are calculated directly, with a few hundred tuning steps on those neurons alone if needed — no training run, no poisoned data, no new code. If a machine has enough memory to run the model, Microsoft notes, it has enough to plant the backdoor.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers
&lt;/h2&gt;

&lt;p&gt;The team tested Lazarus on three open-weight models. On Nvidia’s Nemotron-Terminal-8B, seven neurons out of more than 12,000 in the model’s final feed-forward layer were enough. Fitted on just 23 example sessions, the edited model produced the payload in at least 98.8% of trigger sessions — and never in the trigger-free sessions checked. Let loose to generate freely, it still fired in about 99% of triggered sessions.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Neurons edited&lt;/th&gt;
&lt;th&gt;Trigger success&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Nemotron-Terminal-8B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;7 of 12,000+&lt;/td&gt;
&lt;td&gt;≥98.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen 3.6-27B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;51–99.5%*&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Gemma-4-31B-IT&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Not disclosed&lt;/td&gt;
&lt;td&gt;51–99.5%*&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;*Success rates for the tuned version of the edit ranged from 51% to 99.5% across the Qwen and Gemma setups, depending on the configuration.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4l8y7e3r878in1ckxy0e.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4l8y7e3r878in1ckxy0e.webp" alt="A robot agent at a computer terminal in a server room as a stream of data flows toward the screen" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Editorial illustration: once triggered, the backdoored model emits the attacker’s command and the agent runs it with its own permissions. Image: AI Frontier Post (AI-generated).&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The edited models still look fine on the outside. Both backdoored Qwen versions kept most of their AIME maths scores, and their Terminal-Bench score — a test of agent tasks — slipped only from 46.59% to 42.05%. The point: a model can pass its evals and still be compromised, because the evals never present the trigger.&lt;/p&gt;

&lt;h2&gt;
  
  
  The twist: an AI found it
&lt;/h2&gt;

&lt;p&gt;Lazarus has a second story inside it. The core idea — that dormant neurons could be individually edited and chained into a multi-token payload — was discovered autonomously by Claude Code running Anthropic’s Opus 4.5. What began in March as a quick experiment became a months-long study. The implication is blunt: coding agents can make attacks like this easier to build, not just easier to find.&lt;/p&gt;

&lt;p&gt;That cuts both ways. The same agents can be pointed at defense: the team’s agent-built detectors flagged 91.4% of edited checkpoints without an original to compare against, at a 9.7% false-positive rate — though the test set was small, 35 edited and 31 clean checkpoints. Asking a frontier model to inspect a checkpoint directly worked less well: Claude Opus 5.5 judged 41% of edited checkpoints to be clean.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Reddit made of it
&lt;/h2&gt;

&lt;p&gt;The story surfaced on r/LocalLLaMA late Monday night, drawing more than 140 upvotes and 60 comments overnight. The thread’s framing matched Microsoft’s own: this is a weight-integrity problem, not a case against open weights — the attack needs write access to a checkpoint, wherever models change hands.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3cf8qm95ceveyikouq9q.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3cf8qm95ceveyikouq9q.webp" alt="Two auditors examining a glowing holographic neural network with a magnifying glass" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Editorial illustration: agent-built detectors flagged 91.4% of edited checkpoints — but no scan can prove a model safe. Image: AI Frontier Post (AI-generated).&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How to defend against it
&lt;/h2&gt;

&lt;p&gt;Microsoft’s guidance, published alongside the research, reads like software supply-chain hygiene applied to weights: compare every checkpoint against its declared base model; download only from the original publisher and check file hashes; keep approved models in a private registry; give each agent only the access it needs, run its code in isolated sandboxes, and require a person to approve high-impact actions.&lt;/p&gt;

&lt;p&gt;The honest caveat, stated in the research itself, is that no scan can prove a model is safe. Reliable detection means diffing against the original weights — which is why provenance is the real defense. Microsoft says its Foundry platform already ties open-weight models to their original publishers, and it is building a red-teaming harness from the study’s detection methods, plus work on cryptographic model signing. Microsoft Threat Intelligence adds that it has not observed any threat actors using the technique — this is a researched risk, not a reported campaign.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to watch
&lt;/h2&gt;

&lt;p&gt;First, whether the detection methods hold up beyond the lab: the 91.4% figure comes from a small test set, and free-running attack tests — triggered models acting through a full attack — are still needed.&lt;/p&gt;

&lt;p&gt;Second, whether model hubs start treating weights like code. Signing, provenance records, and routine weight audits are all on Microsoft’s roadmap; the question is whether the rest of the ecosystem follows before someone finds a use for this in the wild.&lt;/p&gt;

&lt;p&gt;And third, the agent angle. If a coding agent can discover a weight-level backdoor on its own, the cost of finding the next one has just dropped — for defenders and attackers alike.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.microsoft.com/en-us/research/blog/lazare-veni-foras-reprogramming-dormant-neurons-to-backdoor-ai-agents/" rel="noopener noreferrer"&gt;Microsoft Research: Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents (November 10, 2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://thehackernews.com/2026/11/microsoft-researchers-show-ai-agents.html" rel="noopener noreferrer"&gt;The Hacker News: Microsoft Researchers Show AI Agents Can Be Backdoored by Editing Fewer Than 10 Neurons (November 11, 2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.infoworld.com/article/4247378/rewriting-less-than-10-neurons-in-an-llm-can-turn-it-into-a-sleeper-backdoor-2.html" rel="noopener noreferrer"&gt;InfoWorld: Rewriting less than 10 neurons in an LLM can turn it into a sleeper backdoor (November 11, 2026)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>security</category>
      <category>research</category>
    </item>
    <item>
      <title>Sing with a simulated throat: hands-on with CHIMERA, the physical-modeling vocal synthesizer</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Tue, 10 Nov 2026 13:05:01 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/sing-with-a-simulated-throat-hands-on-with-chimera-the-physical-modeling-vocal-synthesizer-nlk</link>
      <guid>https://dev.to/aifrontierpost/sing-with-a-simulated-throat-hands-on-with-chimera-the-physical-modeling-vocal-synthesizer-nlk</guid>
      <description>&lt;p&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/chimera-vocal-synth-tutorial/" rel="noopener noreferrer"&gt;https://aifrontierpost.com/articles/chimera-vocal-synth-tutorial/&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;Every singing synthesizer you have used works the same way: it stitches together recordings of a human voice and pitch-shifts them into place. CHIMERA does not. It simulates the thing that makes the sound — lungs, vocal folds, a branched vocal tract, tongue, lips, nose — as a physical system, and the singing comes out of the simulation. Four days after it appeared on GitHub it has more than 900 stars. This guide builds its Rust CLI and renders real audio with it: a spoken sentence, two sung demo songs, a melody of your own, and a beatbox pattern — every step run and measured.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; CHIMERA (jamesondh/chimera-vocal-synth, MIT) is a physical-modeling vocal synthesizer: a Rust engine that simulates a human airway instead of playing back samples, with a browser app and an offline CLI that renders WAV files. This guide clones the repo, builds the CLI in under two minutes, renders speech ("Hello, this is CHIMERA, a voice made of air." — 3.23 seconds of real audio), sings both built-in demo songs, sings a three-note melody from a JSON file you write yourself, and renders a beatbox pattern — every command verified in a sandbox on November 9, 2026. No account, no API key, no GPU, no model weights.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What:&lt;/strong&gt; CHIMERA — a physical-modeling voice synthesizer (Rust). Speak, sing, beatbox, and imitate creatures (songbird, seal, alligator, elephant) from a simulated airway. MIT licence.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Momentum:&lt;/strong&gt; Created November 5, 2026; 915 stars and 106 forks by November 9, 2026; 226 points on the Hacker News front page the same day.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Verified:&lt;/strong&gt; November 9, 2026 in a sandbox: release build of the CLI (1 minute 58 seconds), render text, render song, render notes, render speech, render beatbox, render vowels, and diagnose — every WAV checked with ffprobe and ffmpeg volumedetect. Versions: cargo/rustc 1.99.1, commit 092b16c.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Requirements:&lt;/strong&gt; A Rust toolchain (via rustup) and git. About 15 minutes. No account, no API key, no GPU, no model weights. Bun and the wasm32-unknown-unknown target are only needed for the browser build, not the CLI.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Honest limit:&lt;/strong&gt; The project is four days old. The author is still tuning naturalness against recordings of real singers, and the in-app WAV export is unfinished — the CLI in this guide is the documented route to audio files.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;AI-generated illustration: the vocal tract as a branched acoustic tube network — the physical system CHIMERA simulates instead of playing back samples.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why CHIMERA, why now
&lt;/h2&gt;

&lt;p&gt;Singing synthesis has been stuck on one idea for twenty years. Vocaloid, UTAU, and their descendants all work by concatenating fragments of recorded singing and stretching them to fit a melody. The results can be beautiful, but the approach has a ceiling: every new voice needs hours of studio recordings, every new language needs a new voicebank, and the singer can only ever sound like the person who was recorded.&lt;/p&gt;

&lt;p&gt;CHIMERA takes the other path, the one speech science has used for decades but rarely shipped as a musical instrument: physical modeling. Instead of recordings, it builds a virtual airway — lung pressure, vocal folds, a branched tube network for the vocal tract, tongue, lips, nose — and computes what sound that airway makes. Change the anatomy and you change the voice: male, female, and child bodies are continuous blends of the same parameters, and the same engine drives a songbird's two-voice syrinx, a seal, an alligator, and an elephant. Nothing is sampled. There is no voicebank to download and no model to train.&lt;/p&gt;

&lt;p&gt;The numbers explain the attention. CHIMERA was created on November 5, 2026. By November 9 it had 915 stars and 106 forks, and it was on the Hacker News front page with 226 points. That is fast for a project with no company behind it — it spread because the demo is one page (a playable cutaway head in the browser) and the install is one build command.&lt;/p&gt;

&lt;p&gt;This guide does what the README cannot: it builds the offline CLI the way a reader would, renders real audio with it, and measures the output — durations, sample rates, and loudness — so the steps below are the steps that ran, not the steps that should run.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you'll need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Rust toolchain. Install it from rustup.rs if you do not have it; the sandbox used cargo 1.99.1 and rustc 1.99.1. Check with &lt;code&gt;cargo --version&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;git. For the clone.&lt;/li&gt;
&lt;li&gt;ffmpeg (optional, recommended). Only for checking the rendered audio — ffprobe for format and duration, &lt;code&gt;ffmpeg -af volumedetect&lt;/code&gt; for loudness. The renders work without it.&lt;/li&gt;
&lt;li&gt;About 15 minutes. The build takes under two minutes on a modest machine; the renders take seconds.&lt;/li&gt;
&lt;li&gt;No account, no API key, no GPU, no model weights. Everything runs locally on CPU. The browser app needs Bun and the wasm32-unknown-unknown target, but this guide uses only the native CLI, so neither is required.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Pinned versions, exactly as verified: cargo 1.99.1, rustc 1.99.1, repo commit 092b16c (November 9, 2026).&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1 — Build the CLI
&lt;/h2&gt;

&lt;p&gt;Clone the repo (a shallow clone is enough — the history is four days old anyway) and build the CLI package in release mode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone --depth 1 https://github.com/jamesondh/chimera-vocal-synth.git chimera
&lt;span class="nb"&gt;cd&lt;/span&gt; chimera
cargo build --release -p chimera-cli
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the sandbox this finished in 1 minute 58 seconds: 26 crates, including the chimera engine crate and the chimera-cli front end. The binary lands at target/release/chimera — a single 2.25 MB executable with no runtime dependencies beyond the system C library. (The sandbox ran this build with the target directory on /tmp because its home disk was at its quota cap; on a normal machine the default target directory is fine and nothing about the commands changes.)&lt;/p&gt;

&lt;p&gt;Confirm the binary works and see everything it can do:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;./target/release/chimera --help
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The usage line lists the render kinds: speech, text, song, notes, beatbox, creature, and vowels, plus diagnose, compare, and calibrate. Every render writes 48 kHz mono 16-bit WAV files. All output is in the sandbox's /tmp here only because that is where this run kept its scratch; use any directory you like.&lt;/p&gt;

&lt;p&gt;One quirk to know before you go further: the binary reads its pronunciation dictionary (public/data/cmudict.txt.gz, the CMU Pronouncing Dictionary) from the checkout it was built from, at the absolute build path. Keep the checkout in place at the same path — if you move or delete the repo directory, renders fail with a "No such file or directory" error for that path. Renders work from any working directory; it is the checkout that must stay put, not your shell.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2 — Your first spoken sentence
&lt;/h2&gt;

&lt;p&gt;The fastest proof that the whole chain works is render text: English in, spoken audio out. The text is converted to phonemes with the CMU dictionary (with a letter-to-sound fallback for words the dictionary does not know), planned into articulatory gestures, and rendered through the simulated airway:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;./target/release/chimera render text /tmp/chimera-out/hello &lt;span class="s2"&gt;"Hello, this is CHIMERA, a voice made of air."&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The output, exactly as it came back in the sandbox:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;həlˈoʊ | ðɪsɪztʃɪmˈɛɹə | əvˈɔɪsmˈeɪɾəvˈɛɹ
wrote /tmp/chimera-out/hello.wav  3.23 s, rendered in 2505 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first line is the phonetic transcription the engine actually rendered — the pronunciation it derived for each word. The second line is the file: hello.wav, 3.23 seconds of audio, rendered in about 2.5 seconds. Verify it is real audio and not silence:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ffprobe -v error -show_entries &lt;span class="nv"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;sample_rate,channels,bits_per_sample,duration -of &lt;span class="nv"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;nw&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 /tmp/chimera-out/hello.wav
ffmpeg -i /tmp/chimera-out/hello.wav -af volumedetect -f null - 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s2"&gt;"mean_volume|max_volume"&lt;/span&gt;
&lt;span class="nv"&gt;sample_rate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;48000
&lt;span class="nv"&gt;channels&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1
&lt;span class="nv"&gt;bits_per_sample&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;16
&lt;span class="nv"&gt;duration&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;3.225125
mean_volume: -22.7 dB
max_volume: -8.8 dB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;48 kHz mono 16-bit, 3.23 seconds, mean level −22.7 dB with peaks at −8.8 dB — a normal speech level, not silence and not clipping. The third argument selects the body: male is the default, female speaks at a 205 Hz baseline and child at 250 Hz. If you want raw phonetics instead of English text, render speech takes IPA directly — render speech out "ˈɑpɑ" wrote a 1.09-second WAV at peak 0.374 in the verification run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3 — Sing the demo songs
&lt;/h2&gt;

&lt;p&gt;Speech proves the airway works. Singing is the point of the instrument, and the CLI ships with two demo songs — an original tune called "Made of air" and "Twinkle, twinkle" — that render with one command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;./target/release/chimera render song /tmp/chimera-out/song
Made of air: 12.1 s, peak 0.721, syllables Breathe in breathe out I am made of air Tongue and lips and throat sin ging a ny where
Twinkle, twinkle: 10.3 s, peak 0.750, syllables Twin kle twin kle lit tle star how I won der what you are
live: syllables hel lo there my friend
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three files land in the output directory: made-of-air-male.wav (12.1 seconds), twinkle-twinkle-male.wav (10.3 seconds), and live-male.wav, a short "live-style" take that mimics key presses with gaps and one legato pair, the way the browser keyboard would send them. The printed syllables are the lyric syllables the planner assigned to each note — "sin ging" and "a ny where" are split across notes, and consonants are scheduled to land before the beat, as the README promises. Checking the audio the same way as before: "Twinkle, twinkle" measures mean −12.5 dB, max −2.5 dB — louder than the speech render, as a sung line should be, and still clear of clipping. Pass female or child as the second argument to hear the same songs in the other bodies.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;AI-generated diagram: the branched tube network at the heart of CHIMERA — pressure nodes and flow branches standing in for the airway.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4 — End to end: sing a melody of your own
&lt;/h2&gt;

&lt;p&gt;The demo songs prove the engine. A melody of your own proves you can drive it. The render notes command takes a JSON file with a tempo, lyrics, and a list of [midi, beats] pairs — one lyric syllable per sounding note, a note with MIDI number 0 or below treated as a rest. Save this as melody.json:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"bpm"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"lyrics"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"hello world"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"notes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;72&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;74&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;76&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three notes — C5, D5, E5 — one beat, one beat, two beats, at 120 BPM, with the syllables "hel", "lo", and "world" assigned in order. Render it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;./target/release/chimera render notes melody.json /tmp/chimera-out/melody
/tmp/chimera-out/melody.wav: 2.70 s, peak 0.750, 3 notes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The WAV is 2.70 seconds: four beats at 120 BPM is 2.0 seconds of notes, plus the 0.3-second lead-in and the 0.4-second tail the engine adds. Measured: mean −10.3 dB, max −2.5 dB. That is the whole end-to-end loop — a melody you wrote, sung by a simulated throat, in one command and about a second of compute.&lt;/p&gt;

&lt;p&gt;Two details matter when you write longer files. First, the syllable count must match the sounding-note count exactly, or the command fails with an error naming both counts — for example "2 syllables for 3 notes" — so count the syllables in your lyrics before you render. Second, vibrato is an optional field in the JSON (default 35.0) that sets the vibrato depth. Two more optional flags write machine-readable sidecars: --artic tracks.json records the articulators over time (for animating a face in sync with the audio) and --tele voice.tele records the voice's full telemetry, which the app's duet page replays on a 3D head.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5 — Beatbox, creatures, vowels, and diagnostics
&lt;/h2&gt;

&lt;p&gt;The same engine that sings also speaks, beatboxes, and imitates animals — every drum and every creature is an articulatory gesture, not a sample. Three more commands round out the tour. Beatboxing renders a pattern plus each drum sound on its own:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;./target/release/chimera render beatbox /tmp/chimera-out/beatbox
kch  Crash         onset   -6.8 ms  peak 0.075  set-up -2.3 dB  tail -0.2 dB  lead-in 101 ms
m    Hum           onset    4.6 ms  peak 0.093  set-up -32.5 dB  tail 0.4 dB  lead-in 106 ms
pattern "Boots and cats": 5.50 s, peak 0.237 -&amp;gt; /tmp/chimera-out/beatbox/boots-and-cats.wav
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Alongside the 5.50-second "Boots and cats" pattern, the directory holds one WAV per drum sound (B.wav, K.wav, P_.wav, Pf.wav, kch.wav, kit.wav, m.wav, t.wav, tss.wav). The per-drum lines are the engine's own diagnostics: onset timing, peak, set-up and tail levels for each gesture. A creature call works the same way — render creature songbird, seal, alligator, or elephant — each rendered through that animal's anatomy (the songbird uses a two-voice syrinx instead of vocal folds). And render vowels female sweeps the vowel space in the female body and wrote vowels-female.wav in the verification run.&lt;/p&gt;

&lt;p&gt;Finally, diagnose measures the voice against its own design targets and writes a set of test utterances:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;./target/release/chimera diagnose /tmp/chimera-out/diag
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It wrote five WAV files (s1.wav through s5.wav) and printed formant measurements per vowel (LPC estimates against the model's own formants), duration norms (stressed syllables 125 ms against unstressed 83 ms; phrase-final 145 ms against non-final 95 ms), and a pitch summary (29.4% of voiced time at the 110 Hz baseline, 6.2 semitones of range between the 5th and 95th percentiles). You will not need these numbers to use the instrument, but they are the fastest way to confirm a build is healthy: if diagnose prints formants and durations in the expected ranges, the engine, the dictionary, and the audio path all work.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;AI-generated illustration: the CLI's job in one picture — text in, a sung note out, with the songbird bestiary along for the ride.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How it works, in one paragraph
&lt;/h2&gt;

&lt;p&gt;The engine is a branched, staggered-grid finite-difference time-domain (FDTD) transmission-line model of the airway — chosen over the simpler Kelly-Lochbaum model because it handles variable section lengths, pressure build-up behind closures, nonlinear losses at constrictions, yielding walls, and arbitrary branching (the nasal tract, sinuses, two bronchi, vocal sacs) in one framework. The sound source is a parametric glottis: a glottal area pulse with opening and closing phases, open quotient, and skew, driving the airway, with jitter, shimmer, vibrato, diplophonia, vocal fry, and falsetto as perturbations. A gesture scheduler turns notes and phonemes into articulator movements at the control rate, and the airway itself runs at an internal rate of 96 kHz for an adult human (192 kHz for small birds, 48 kHz for huge tracts) before decimation to the 48 kHz output. The design document is candid about what is not built yet — the wider bestiary, the two-mass vocal folds, an IPA chart mode, and the app-side EQ/reverb/limiter chain — and it says, plainly, to treat the code as the truth where the document disagrees. The architecture document in the repo is the place to read all of this in full.&lt;/p&gt;

&lt;h2&gt;
  
  
  Troubleshooting
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;render fails with "No such file or directory" for public/data/cmudict.txt.gz.&lt;/strong&gt; The binary reads the pronunciation dictionary from the checkout it was built from, at the absolute build path. You moved, renamed, or deleted the repo directory after building. Restore the checkout at the same path (or rebuild from the new location) and it works again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;cargo build fails to find the Rust toolchain.&lt;/strong&gt; The build needs a Rust toolchain on your PATH — install it from rustup.rs and open a fresh shell so the cargo shim is picked up. The wasm32-unknown-unknown target is not required for the CLI; only the browser build needs it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;render notes reports a syllable/note mismatch.&lt;/strong&gt; The lyric syllable count must equal the sounding-note count. Rests (MIDI 0 or below) do not take a syllable. Count the syllables — "hello world" is three (hel, lo, world) — and check the error message, which prints both counts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The rendered audio sounds quiet or robotic.&lt;/strong&gt; Check the levels first with &lt;code&gt;ffmpeg -af volumedetect&lt;/code&gt;: the verification renders measured means between −10 and −23 dB, which is normal for unmastered synthesis. As for robotic — that is the honest state of the instrument. The author is tuning naturalness against recordings of real singers, and the project's own comparison documents score the voice against the Children's Song Dataset and 23 real speakers. Expect a distinctive synthetic voice, not a human impersonation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The browser app will not load.&lt;/strong&gt; The app needs a secure context (localhost or HTTPS) for its AudioWorklet, plus Bun and the wasm32-unknown-unknown target to build. The CLI in this guide needs none of that.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  When to use this vs alternatives
&lt;/h2&gt;

&lt;p&gt;Use CHIMERA when you want a voice you can control at the anatomical level: exact pitch with no resampling artifacts, a voice that morphs continuously between bodies, creature calls and beatboxing from the same engine, and audio rendered fully offline with no weights to download and no API to pay for. It is the right tool for procedural game audio, experimental music, and research into how voices work.&lt;/p&gt;

&lt;p&gt;Use Vocaloid or UTAU instead when you need a polished, human-sounding sung vocal in a finished track today. Sample-based synthesis is still far ahead on naturalness, and it comes with decades of voicebanks, editors, and tutorials. CHIMERA is four days old; the comparison is not close yet, and the author says so in the project's own documents.&lt;/p&gt;

&lt;p&gt;Use a neural speech model like Kokoro-82M instead when the job is spoken narration. Kokoro runs on CPU, sounds strikingly human, and needs only a text prompt — but it speaks, it does not sing, and you cannot reach in and change its vocal tract. Use a neural song model like YuE2 instead when you want a full produced song from a prompt and have a GPU and model weights to spend. CHIMERA is the tool for the space between those: a voice as a physical system you can inspect, measure, and retune.&lt;/p&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;What you built is a working vocal synthesizer with no samples, no weights, and no account: a Rust CLI that renders speech, songs, a melody of your own, beatbox patterns, and creature calls out of a simulated human airway, verified end to end with measured audio at every step. The total cost is one clone, one two-minute build, and a handful of commands. The honest limits, stated plainly: the project is four days old, naturalness is still being tuned, the browser app's WAV export is unfinished, and the binary is tied to the checkout it was built from. Within those limits, it is the most interesting new instrument in vocal synthesis this year — and the reason it has 915 stars in four days is that anyone can hear it in one command.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Do I need a GPU or model weights?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;No. CHIMERA is a physical simulation, not a neural model. There are no weights to download, no GPU involved, and no API key. A CPU and a Rust toolchain are the whole requirement.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I use the rendered audio in my own projects?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The project is MIT-licensed, so the code and the audio it renders are yours to use. The CMU Pronouncing Dictionary bundled in the repo keeps its own licence (see public/data/CMUDICT-LICENSE.txt), and the fonts in public/fonts/ keep theirs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I hear it without building anything?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The browser demo at chimera-vocal-synth.vercel.app runs the same Rust engine compiled to WebAssembly — drag the tongue and jaw in the cutaway head, or use the piano roll to sing. The CLI in this guide is the route to saving audio files, since in-app WAV export is unfinished.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why does the binary stop working if I delete the repo?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The CLI reads its pronunciation dictionary from the checkout it was built from, at the absolute build path. Keep the repo directory in place at the same path and renders work from any directory; move or delete it and every render fails on the missing dictionary file.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;jamesondh/chimera-vocal-synth — repository, README, and CLI source: &lt;a href="https://github.com/jamesondh/chimera-vocal-synth" rel="noopener noreferrer"&gt;https://github.com/jamesondh/chimera-vocal-synth&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CHIMERA architecture document — the acoustic model (FDTD airway, sources, anatomy)&lt;/li&gt;
&lt;li&gt;CHIMERA live demo — the browser app running the same engine as WebAssembly: &lt;a href="https://chimera-vocal-synth.vercel.app" rel="noopener noreferrer"&gt;https://chimera-vocal-synth.vercel.app&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Related coverage
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Your own narration studio, offline: generate natural speech with Kokoro-82M on your CPU&lt;/li&gt;
&lt;li&gt;Compose a full song on your own GPU: hands-on with YuE2, the open model taking on Suno&lt;/li&gt;
&lt;li&gt;Split any song into six stems on your own machine: hands-on with StemDeck&lt;/li&gt;
&lt;li&gt;Transcribe anything on your own machine: a hands-on guide to faster-whisper&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>opensource</category>
      <category>rust</category>
    </item>
    <item>
      <title>Generate and edit AI video with Gemini Omni 1.2 Flash</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Mon, 09 Nov 2026 13:04:00 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/generate-and-edit-ai-video-with-gemini-omni-12-flash-20id</link>
      <guid>https://dev.to/aifrontierpost/generate-and-edit-ai-video-with-gemini-omni-12-flash-20id</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/gemini-omni-flash-video-tutorial/" rel="noopener noreferrer"&gt;AI Frontier Post&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Video generation on the Gemini API used to mean starting a long-running operation, polling it every ten seconds, and downloading a file when it finished. That workflow is gone. Omni runs on the Interactions API: one &lt;code&gt;client.interactions.create()&lt;/code&gt; call, and the video comes back in the response as base64 data. The same call shape handles everything else — a text prompt, a reference image, your own footage, an edit, an extension — and every follow-up turn references the previous interaction by id, so the model keeps the scene while you change one thing at a time.&lt;/p&gt;

&lt;p&gt;The practical consequence is that video stops being a render job and becomes a conversation. You generate a 10-second establishing shot, look at it, and say "change the light to golden hour" instead of re-rolling the whole prompt and hoping the new clip matches the old one. This tutorial teaches that loop properly: the exact calls, the format controls, the cost of each choice, and the limits that will catch you out.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. What you'll need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Python 3.10 or newer.&lt;/strong&gt; The current &lt;code&gt;google-genai&lt;/code&gt; SDK needs it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A Gemini API key on a paid tier.&lt;/strong&gt; Create one in &lt;a href="https://aistudio.google.com/api-keys" rel="noopener noreferrer"&gt;Google AI Studio&lt;/a&gt; for a project with billing turned on. Omni is not available on the free tier at all — the pricing page lists the free tier as "Not available" for both input and output, and a free-tier key will be refused. Budget about $2 for everything in this guide.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;About 45 minutes.&lt;/strong&gt; Most of that is watching clips render; the code is short.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ffmpeg and ffprobe (optional).&lt;/strong&gt; Only for inspecting the MP4s you get back. Install with your package manager if you want the verification step in Step 2.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One migration note before you start: if you have old code calling &lt;code&gt;gemini-omni-1.1-flash&lt;/code&gt;, it keeps working for now. From December 3, 2026, Google routes every request to that id to &lt;code&gt;gemini-omni-1.2-flash&lt;/code&gt; automatically and bills it at 1.2 rates, with no code change and no error. Pin the 1.2 id in new code and you never have to think about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Step 1 — Set up the SDK
&lt;/h2&gt;

&lt;p&gt;Create a project folder and a virtual environment, install the SDK, and export your key. The SDK reads &lt;code&gt;GEMINI_API_KEY&lt;/code&gt; from the environment, so never paste the key into your code.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;mkdir &lt;/span&gt;omni-shoot &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;cd &lt;/span&gt;omni-shoot
python3 &lt;span class="nt"&gt;-m&lt;/span&gt; venv .venv &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate
pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-U&lt;/span&gt; &lt;span class="s2"&gt;"google-genai&amp;gt;=2.30.1"&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;GEMINI_API_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"your-paid-tier-key"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Verify the install before spending anything. This constructs a client and prints the SDK version — it makes no API call, so it costs nothing and works offline:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s2"&gt;"from google import genai; c = genai.Client(); print('SDK ready:', genai.__version__)"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You should see &lt;code&gt;SDK ready: 3.0.0&lt;/code&gt; or newer. The &lt;code&gt;&amp;gt;= 2.30.1&lt;/code&gt; floor matters: older SDK versions predate the Interactions API surface this guide uses, and the &lt;code&gt;interactions.create&lt;/code&gt; call below will not exist. If the version prints below 2.30.1, re-run the install — you are probably picking up a stale environment.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Step 2 — Generate your first clip
&lt;/h2&gt;

&lt;p&gt;This is the whole API in its simplest form: a model id, a text prompt, and the video that comes back. Write it to &lt;code&gt;first_clip.py&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;base64&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# reads GEMINI_API_KEY
&lt;/span&gt;
&lt;span class="n"&gt;interaction&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gemini-omni-1.2-flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Continuous, unbroken handheld shot of a lighthouse at dusk, &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;waves breaking on rocks, gulls calling. No dialogue.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;lighthouse.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base64&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;b64decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;interaction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_video&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;saved lighthouse.mp4, interaction id:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;interaction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it with &lt;code&gt;python3 first_clip.py&lt;/code&gt;. The call is synchronous and takes a while — generation time grows with duration, resolution, and current API load, so a 10-second clip can take a minute or more. There is nothing to poll: the call returns when the video is ready, and the MP4 arrives as base64 in &lt;code&gt;interaction.output_video.data&lt;/code&gt;. That convenience field exists only in the SDKs; if you ever call the REST endpoint directly (&lt;code&gt;POST https://generativelanguage.googleapis.com/v1beta/interactions&lt;/code&gt;), the video sits in the &lt;code&gt;steps&lt;/code&gt; array as a &lt;code&gt;model_output&lt;/code&gt; part with &lt;code&gt;type: "video"&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Check what you got back:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ffprobe &lt;span class="nt"&gt;-v&lt;/span&gt; error &lt;span class="nt"&gt;-show_entries&lt;/span&gt; &lt;span class="nv"&gt;format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;duration,size &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-show_entries&lt;/span&gt; &lt;span class="nv"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;codec_name,width,height,r_frame_rate &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-of&lt;/span&gt; &lt;span class="nv"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;noprint_wrappers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 lighthouse.mp4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expect a 10-second MP4 at 1280×720. Two defaults are doing quiet work here: duration defaults to 10 seconds when you do not set it, and resolution defaults to 720p. The clip also has native audio — the gulls and the waves are generated with the picture, not mixed in afterwards. Keep the printed interaction id: it is the handle every later step uses to edit this clip.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcnj6kj02rnouc3k8qa62.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcnj6kj02rnouc3k8qa62.webp" alt="Diagram: text prompts, reference images and video files flow into the Interactions API, which returns a video with native audio" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Diagram: AI Frontier Post. One endpoint takes text, images, or video in and returns video with synchronized audio — the rest of this guide is just choosing which inputs to send.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Step 3 — Control duration, aspect ratio, and resolution
&lt;/h2&gt;

&lt;p&gt;Everything about the output format lives in one &lt;code&gt;response_format&lt;/code&gt; dict. Duration is any whole second from 3 to 15, written as a string like &lt;code&gt;"15s"&lt;/code&gt;; setting it overrides any length mentioned in your text prompt, and anything over 15 seconds returns an invalid-argument error. Aspect ratio is &lt;code&gt;"16:9"&lt;/code&gt; (default) or &lt;code&gt;"9:16"&lt;/code&gt; for portrait. Resolution is &lt;code&gt;"360p"&lt;/code&gt;, &lt;code&gt;"720p"&lt;/code&gt; (default), &lt;code&gt;"1080p"&lt;/code&gt;, or &lt;code&gt;"4k"&lt;/code&gt; — the last two are upscaled from the generated resolution, not natively rendered at that size.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;interaction&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gemini-omni-1.2-flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A single unbroken tracking shot following a paper airplane gliding through a bustling library.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;response_format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;video&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;duration&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;15s&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;aspect_ratio&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;9:16&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resolution&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1080p&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;library_flight.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base64&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;b64decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;interaction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_video&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That clip is a vertical 15-second short at upscaled 1080p — the most expensive combination in this guide. The pricing page bills video output by tokens consumed, at 5,792 tokens per second of 720p video, and the per-second cost follows from the resolution you ask for:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Resolution&lt;/th&gt;
&lt;th&gt;Tokens / second&lt;/th&gt;
&lt;th&gt;Cost / second&lt;/th&gt;
&lt;th&gt;10 s clip&lt;/th&gt;
&lt;th&gt;15 s clip&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;360p&lt;/td&gt;
&lt;td&gt;1,931&lt;/td&gt;
&lt;td&gt;~$0.023&lt;/td&gt;
&lt;td&gt;~$0.23&lt;/td&gt;
&lt;td&gt;~$0.35&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;720p (default)&lt;/td&gt;
&lt;td&gt;5,792&lt;/td&gt;
&lt;td&gt;~$0.07&lt;/td&gt;
&lt;td&gt;~$0.70&lt;/td&gt;
&lt;td&gt;~$1.04&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1080p (upscaled)&lt;/td&gt;
&lt;td&gt;8,688&lt;/td&gt;
&lt;td&gt;~$0.104&lt;/td&gt;
&lt;td&gt;~$1.04&lt;/td&gt;
&lt;td&gt;~$1.56&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4K (upscaled)&lt;/td&gt;
&lt;td&gt;17,376&lt;/td&gt;
&lt;td&gt;~$0.209&lt;/td&gt;
&lt;td&gt;~$2.09&lt;/td&gt;
&lt;td&gt;~$3.13&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Input tokens are $1.50 per million regardless of type — text, image, or video — so a long prompt or several reference images add pennies, not dollars. The discipline that saves real money is resolution: draft at 720p, and only upscale the one take you actually ship. A 360p draft costs a third of the 720p version and is fine for judging composition and motion.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Step 4 — Animate a reference image
&lt;/h2&gt;

&lt;p&gt;Text alone is a blunt instrument for keeping a character or a product looking the same across shots. Pass the image in as part of the input instead: a list of content parts, image first, then the text that says what to do with it. The model decides how to use the image from your prompt, so be explicit about whether it is the first frame, a style reference, or a guide for movement.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;base64&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;product_shot.jpg&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;base64_image&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;base64&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;b64encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;interaction&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gemini-omni-1.2-flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;image&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;base64_image&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mime_type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;image/jpeg&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Bring this product shot to life: slow orbit around the object, &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;soft studio light, shallow depth of field. Do not change the &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;object itself. No dialogue.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;)},&lt;/span&gt;
    &lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;product_orbit.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base64&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;b64decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;interaction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_video&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three things to know about reference images. You can pass up to 6 of them in one call, across all durations — that is how you hold a character, an object, and a location steady while the camera moves. Use high-resolution images; the docs are explicit that image-to-video works best with them. And if you want a clip that starts on one frame and ends on another, pass two images as first and last frames: that interpolation mode supports durations of 3 to 10 seconds only, and asking for more returns an invalid-argument error.&lt;/p&gt;

&lt;p&gt;Reference videos work the same way — up to 3 of them (3 seconds each) for clips of 3 to 10 seconds, or a single 3-second reference video for 11 to 15 second clips — but only when you are generating a new video, not when editing or extending. Their audio is ignored; they are visual likeness references only.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Step 5 — Edit in conversation
&lt;/h2&gt;

&lt;p&gt;This is the step that justifies the whole model. Generate a clip, then edit it by passing the previous interaction's id and a short sentence. The model remembers the video context, applies your change, and preserves everything you did not mention. No re-upload, no re-prompting the whole scene.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;base64&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="c1"&gt;# Turn 1: generate the first clip.
&lt;/span&gt;&lt;span class="n"&gt;res1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gemini-omni-1.2-flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A woman playing violin outdoors.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Turn 2: edit it. Same scene, one change.
&lt;/span&gt;&lt;span class="n"&gt;res2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gemini-omni-1.2-flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;previous_interaction_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;res1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Change the light to golden hour. Keep everything else the same.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;violin_golden_hour.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base64&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;b64decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;res2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_video&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Edits work on generated videos up to 15 seconds long. Each turn produces a new video, and the chain can continue: &lt;code&gt;res2.id&lt;/code&gt; becomes the &lt;code&gt;previous_interaction_id&lt;/code&gt; of turn 3. The prompting discipline is the opposite of what you would expect — short wins. The guide's own examples are one-liners like "Make this video anime", "Put a fashionable hat on this person", "Change the lighting to be more dramatic". Overly descriptive edit prompts cause unintended changes, because every extra clause is another thing the model may reinterpret. When you edit one specific thing, end the sentence with "Keep everything else the same" — it is the closest thing to a pin for the rest of the frame.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftqp9ki9sl56pwwe58747.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftqp9ki9sl56pwwe58747.webp" alt="Diagram: a four-step loop — generate, edit, extend, final cut — with each turn reusing the last interaction id" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Diagram: AI Frontier Post. Generate once, then edit and extend in turns. The rule of thumb: edit what you can see, extend what you can hear, regenerate only when the scene itself is wrong.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Step 6 — Extend to 30 seconds
&lt;/h2&gt;

&lt;p&gt;Fifteen seconds is a clip, not a scene. Extension appends a seamless continuation to the end of a clip: describe how the scene should continue, and the model generates 3 to 10 seconds more (10 by default, or set with &lt;code&gt;duration&lt;/code&gt; in &lt;code&gt;response_format&lt;/code&gt;), up to a total length of 30 seconds. You can extend a video the model generated — by referencing its &lt;code&gt;previous_interaction_id&lt;/code&gt; — or a video you upload, which Step 7 covers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;base64&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="c1"&gt;# Continue the lighthouse clip from Step 2 for another 10 seconds.
&lt;/span&gt;&lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gemini-omni-1.2-flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;previous_interaction_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;v1_...&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# the id printed by first_clip.py
&lt;/span&gt;    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Continue the scene: the camera slowly rises above the lighthouse as the lamp turns on.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;response_format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;video&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;duration&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;10s&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;lighthouse_extended.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base64&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;b64decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_video&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Extension has a memory limit worth designing around: the model uses only the last 10 seconds of the input video as context, and preserves earlier frames as-is. If a character or visual element appeared only before that window, mention it again in your prompt or pass it as a reference image, or it may quietly vanish from the continuation. Extension only appends to the end — you cannot prepend or extend the middle of a clip — and some of the final frames of the input are reworked to make the join seamless, so do not treat the last second of the original as sacred. If a plain prompt like "Extend this video" does not trigger extension mode, the docs' fallback is to set &lt;code&gt;"task": "extend"&lt;/code&gt; in &lt;code&gt;video_config&lt;/code&gt;; try the prompt first, because the task flag adds strict constraints to the model.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Step 7 — Edit a video you shot yourself
&lt;/h2&gt;

&lt;p&gt;Omni also edits real footage. Upload a clip of up to 15 seconds through the Files API, wait for processing to finish, then pass the file's URI in as a video part with your edit instruction. The shape is the same list-of-parts input as image-to-video, with &lt;code&gt;"type": "video"&lt;/code&gt; and a &lt;code&gt;uri&lt;/code&gt; instead of base64 data.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;base64&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="c1"&gt;# Upload your video using the Files API.
&lt;/span&gt;&lt;span class="n"&gt;video_file&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;files&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;upload&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;my_video.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;video_file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PROCESSING&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Waiting for video to be processed.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;video_file&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;files&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;video_file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;video_file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAILED&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;video_file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Video processing complete:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;video_file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Edit your video.
&lt;/span&gt;&lt;span class="n"&gt;interaction&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gemini-omni-1.2-flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;video&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;uri&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;video_file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;When the person touches the mirror, make the mirror ripple &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;beautifully like liquid, and the person&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;s arm turns into &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;reflective mirror material. Keep everything else the same.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;)},&lt;/span&gt;
    &lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mirror_edit.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base64&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;b64decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;interaction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_video&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two regional catches apply to this step and not to the others. Editing or extending &lt;em&gt;uploaded&lt;/em&gt; videos is not available in the European Economic Area, Switzerland, or the United Kingdom — if your call returns empty output there, that is probably why. Editing videos the model itself generated works in all available regions. The same regional restriction covers uploading images that contain minors or certain recognizable people. And one content limit: you cannot extend an uploaded video of someone talking to add more dialogue; the character has to stay silent, or you extend a model-generated clip through &lt;code&gt;previous_interaction_id&lt;/code&gt; instead, where spoken dialogue is supported.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Step 8 — Put it together: a reusable shoot script
&lt;/h2&gt;

&lt;p&gt;The three moves — generate, edit, extend — chain into a single script that turns one prompt into a finished 20-second cut. Save this as &lt;code&gt;omni_shoot.py&lt;/code&gt; and run it with &lt;code&gt;python3 omni_shoot.py&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;base64&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;

&lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gemini-omni-1.2-flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;genai&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# reads GEMINI_API_KEY
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;interaction&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Decode the SDK&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;s output_video convenience field and write the MP4.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base64&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;b64decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;interaction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_video&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;saved &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 1. Generate a 10 s establishing shot.
&lt;/span&gt;&lt;span class="n"&gt;shot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Continuous, unbroken handheld shot of a lighthouse at dusk, &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;waves breaking on rocks, gulls calling. No dialogue.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;response_format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;video&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;duration&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;10s&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;shot&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;shot_1.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 2. Edit it in conversation: same scene, golden-hour light.
&lt;/span&gt;&lt;span class="n"&gt;shot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;previous_interaction_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;shot&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Change the light to golden hour. Keep everything else the same.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;shot&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;shot_2.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 3. Extend it by 10 s to a 20 s cut.
&lt;/span&gt;&lt;span class="n"&gt;shot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;previous_interaction_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;shot&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Continue the scene: the camera slowly rises above the lighthouse &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;as the lamp turns on.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;response_format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;video&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;duration&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;10s&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;shot&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;shot_3.mp4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Done: shot_1.mp4 (original), shot_2.mp4 (edited), shot_3.mp4 (extended).&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected output is three files and four lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;saved shot_1.mp4
saved shot_2.mp4
saved shot_3.mp4
Done: shot_1.mp4 (original), shot_2.mp4 (edited), shot_3.mp4 (extended).
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The bill for the run is about $2.10 at 720p — three 10-second turns at roughly $0.70 each, plus pennies of input tokens. If a turn fails, the failure is usually one of three things: a free-tier key (Omni refuses it outright), a duration over 15 seconds, or an edit prompt so long the model changed something you did not ask it to. Shorten the prompt, re-run from the last good interaction id, and keep going — nothing before the failure needs to be regenerated.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Which approach should you use?
&lt;/h2&gt;

&lt;p&gt;Omni has one endpoint and five distinct jobs. Pick by what you already have, not by what sounds most powerful:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;You have&lt;/th&gt;
&lt;th&gt;Use&lt;/th&gt;
&lt;th&gt;Why&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Nothing but an idea&lt;/td&gt;
&lt;td&gt;Text-to-video&lt;/td&gt;
&lt;td&gt;One prompt, one clip. Draft at 360p, upscale the keeper.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A still image or product shot&lt;/td&gt;
&lt;td&gt;Image-to-video&lt;/td&gt;
&lt;td&gt;The image anchors the look; up to 6 references hold a character steady across shots.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A clip that is almost right&lt;/td&gt;
&lt;td&gt;Multi-turn edit&lt;/td&gt;
&lt;td&gt;One short sentence per turn, &lt;code&gt;previous_interaction_id&lt;/code&gt; doing the memory work. Cheaper than regenerating.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A clip that ends too soon&lt;/td&gt;
&lt;td&gt;Extension&lt;/td&gt;
&lt;td&gt;3–10 s per turn to a 30 s total. Remember the 10-second lookback window.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Footage you shot yourself&lt;/td&gt;
&lt;td&gt;Files API + edit&lt;/td&gt;
&lt;td&gt;Upload once, edit in language. Check the regional restriction first if you are in the EEA, Switzerland, or the UK.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No code, no key, no budget&lt;/td&gt;
&lt;td&gt;The official Colab quickstart&lt;/td&gt;
&lt;td&gt;Google's Omni Quickstart notebook runs the same model from a notebook (linked in Sources). You still need a paid-tier key for the API calls.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The one comparison that matters is against the old Veo workflow, and it is not close: no operation to start, no polling loop, no download step, and edits that reuse the scene instead of re-rolling it. The price is that you give up the knobs video models used to expose — there is no temperature, no top-p, no seed, no system instruction. Your only controls are the prompt, the reference media, and the response format.&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Prompting that actually works
&lt;/h2&gt;

&lt;p&gt;Omni's prompt guide is short and unusually honest about what helps. The patterns that matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One scene per clip, stated plainly.&lt;/strong&gt; By default the model tries to build a small narrative with several shots. If you want a single continuous shot, say so: "In a single unbroken scene", "In a single continuous shot", "No scene cuts".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Describe the sound.&lt;/strong&gt; Audio is generated with the video, and silence is a choice the model makes for you unless you direct it. Name the music, the ambience, or the dialogue — "Include calm background music", "The audio is a low tinny radio broadcast in the background" — and say "No dialogue" when you mean it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Put negatives in the prompt.&lt;/strong&gt; There is no negative-prompt parameter. "No dialogue", "No embellishments", "No extra sound effects" go in the regular prompt text.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time things explicitly.&lt;/strong&gt; Events can be scheduled in natural language — "After 3 seconds, a woman enters the scene" — or with a timecode syntax: &lt;code&gt;[0-5s] A person is walking down a quiet street&lt;/code&gt;, &lt;code&gt;[5-10s] They stop and turn toward a glowing doorway&lt;/code&gt;, &lt;code&gt;[10-15s] They start running into the portal&lt;/code&gt;. When extending, 0s means the start of the new part, not the start of the clip.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Text in frame works if you spell it out.&lt;/strong&gt; The model renders readable on-screen text when you quote it exactly, including on signs, storefronts, and number plates. Define background text too, or it will invent its own.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep edits to one sentence.&lt;/strong&gt; "Make this video anime." "Put a fashionable hat on this person." "Change the text on the sign to say 'Omni Flash'." Then "Keep everything else the same."&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  12. Limits and gotchas
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Paid tier only.&lt;/strong&gt; There is no free tier, no trial credit path, and no way to test the model without a billed project. The cheapest possible experiment is a 3-second 360p clip for about $0.07.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No generation knobs.&lt;/strong&gt; System instructions, temperature, &lt;code&gt;top_p&lt;/code&gt;, stop sequences, and negative prompts are all unsupported. If the output is wrong, the only lever is the prompt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No voice editing, no audio inputs.&lt;/strong&gt; You cannot re-record dialogue on an existing clip, and you cannot pass audio in as a reference. Reference videos contribute pictures only.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Large outputs need URI delivery.&lt;/strong&gt; For videos over 4 MB — 1080p and 4K, in practice — set &lt;code&gt;delivery="uri"&lt;/code&gt; in &lt;code&gt;response_format&lt;/code&gt; to avoid payload size limits. Note that a later &lt;code&gt;GET&lt;/code&gt; on the interaction returns base64 inline even if the creation used URI delivery; the URI is only guaranteed in the creation response.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;store=false&lt;/code&gt; is a trap.&lt;/strong&gt; The docs recommend &lt;code&gt;background=false&lt;/code&gt;, &lt;code&gt;store=false&lt;/code&gt;, &lt;code&gt;stream=false&lt;/code&gt; for the fastest synchronous generation — but with &lt;code&gt;store=false&lt;/code&gt; the video is not kept, and you cannot edit it later with &lt;code&gt;previous_interaction_id&lt;/code&gt;. Use it for one-off drafts only.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Everything is watermarked.&lt;/strong&gt; All generated videos carry an invisible SynthID watermark for provenance. It is not visible to viewers, but it is detectable, and it survives re-encoding.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;English is the safe language.&lt;/strong&gt; English is fully supported; other languages have not been evaluated and may work or vary. Write prompts in English even for non-English dialogue.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Safety filters apply to inputs and outputs.&lt;/strong&gt; Prompts that violate usage policies are blocked, and the filters vary by region.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;Omni 1.2 Flash is not a better video generator bolted onto the old API. It is the point where Google's video API stopped being a render queue and became a conversation: generate a clip, look at it, change one thing in a sentence, extend it when it ends too soon, and stop when the cut is done. The whole skill is knowing which of the five moves a job needs — text, image, edit, extend, or your own footage — and keeping every prompt short enough that the model can only do what you asked. The script in Step 8 is the template: one prompt in, three files out, about two dollars. Start there, and let the clips teach you the rest.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verification note
&lt;/h2&gt;

&lt;p&gt;Verified November 9, 2026: Google's Omni guide, video overview, and pricing page, read the same day; Python SDK &lt;code&gt;google-genai 3.0.0&lt;/code&gt; installed locally. Every request shape in this guide — text-to-video, &lt;code&gt;response_format&lt;/code&gt; with duration, aspect ratio and resolution, image-to-video with inline base64, multi-turn edits with &lt;code&gt;previous_interaction_id&lt;/code&gt;, extension, and Files API upload with the PROCESSING poll — was validated against the installed SDK's types (&lt;code&gt;CreateModelInteractionParam&lt;/code&gt;, &lt;code&gt;VideoResponseFormat&lt;/code&gt;, &lt;code&gt;Interaction.output_video&lt;/code&gt;), and the Step 8 script was executed against a mock transport that returned the documented response shape and asserted all three calls field by field, including the id chaining between turns. The Pexels cover photo and the two diagrams were produced and inspected locally. Not verified here: live generation against Google's API (no paid-tier key exists in this sandbox — run Step 1 with your own key first), the actual look or sound of any generated clip, per-call latency, or the exact token count of a real call.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://ai.google.dev/gemini-api/docs/omni" rel="noopener noreferrer"&gt;Google — Generate and edit videos with Gemini Omni Flash&lt;/a&gt; — model capabilities, text-to-video, duration, resolution, aspect ratio, image-to-video, stateful editing, editing your own videos, extension, best practices, limitations, and the prompt guide; read November 9, 2026.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://ai.google.dev/gemini-api/docs/video" rel="noopener noreferrer"&gt;Google — Video generation in the Gemini API&lt;/a&gt; — Omni as the video model for the Gemini API, and the Veo 3.1 Preview shutdown on October 22, 2026; read November 9, 2026.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://ai.google.dev/gemini-api/docs/pricing" rel="noopener noreferrer"&gt;Google — Gemini Developer API pricing&lt;/a&gt; — Gemini Omni 1.2 Flash rates: $1.50/1M input tokens, $12.00/1M video output tokens, 5,792 tokens per second of 720p video, and the per-resolution effective prices; read November 9, 2026.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://colab.sandbox.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_Omni.ipynb" rel="noopener noreferrer"&gt;Google — Get started with Omni (Colab quickstart)&lt;/a&gt; — the official notebook for trying Omni without writing the boilerplate yourself.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Canva cuts 1,271 jobs in its first mass layoff as AI costs bite and growth stalls</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Sun, 08 Nov 2026 12:12:21 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/canva-cuts-1271-jobs-in-its-first-mass-layoff-as-ai-costs-bite-and-growth-stalls-3gdh</link>
      <guid>https://dev.to/aifrontierpost/canva-cuts-1271-jobs-in-its-first-mass-layoff-as-ai-costs-bite-and-growth-stalls-3gdh</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/canva-cuts-1271-jobs-ai-costs-growth-slowdown/" rel="noopener noreferrer"&gt;AI Frontier Post&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Canva is cutting or relocating 1,271 roles — about 22% of its workforce — after growth slowed for the first time in a decade. AI is at the centre of the story, but not in the way the headlines suggest: it is the cost and the competitor, not the replacement.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Announced: November 3, 2026, in a note to staff from co-founder and CEO Melanie Perkins; reported by TechCrunch on November 4. The first mass layoff in Canva’s history since its founding in 2013.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Scale: 1,271 roles affected — cut, changed or relocated — about 22% of a workforce of roughly 5,800. Canva has not said how many are outright exits.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The money: revenue growth forecast cut from 30% to 20%, partly on ballooning AI costs; early investors marked the valuation down to about $35 billion, from $42 billion.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The package: at least six months’ pay, departing staff keep their MacBooks, and a $10,000 grant to start a business or community project.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The restructure: product, growth and platform teams merge into a single product group, work centralises in Sydney, and the new structure takes effect in January 2027.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhhtydm4l7coh7x925wkd.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhhtydm4l7coh7x925wkd.webp" alt="An empty design office at dusk, desks with colourful screens and packing boxes" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Editorial illustration: an empty design office at dusk. AI-generated image.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Canva is cutting 1,271 roles — about 22% of its workforce — in the first mass layoff of its 13-year history. Co-founder and CEO Melanie Perkins told staff on November 3 that growth had slowed for the first time in a decade. The cuts, reported by TechCrunch on November 4, merge the product, growth and platform teams into a single group and centralise more work in Sydney.&lt;/p&gt;

&lt;p&gt;The story is being told as an AI layoff. That is the wrong frame. Nothing in Canva’s own account says AI is doing the work of the people leaving. What the reporting shows is a company squeezed by AI on two sides at once: the cost of running AI features for hundreds of millions of mostly free users, and rivals — including the chatbots themselves — giving away the design work Canva used to sell.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually happened
&lt;/h2&gt;

&lt;p&gt;The headline number needs a qualifier. The 1,271 roles are not all job losses: Canva says some will be cut outright, some will change, and some will be asked to relocate to what Perkins called “centralised hubs.” Canva has not said how many fall into each bucket. The reported package: at least six months’ pay including statutory entitlements, departing staff keep their company MacBooks, and a $10,000 grant from the Canva Collective Fund to start a business or community project.&lt;/p&gt;

&lt;p&gt;Perkins called it “our hardest year yet.” Canva has around 260 million monthly users and about $3.5 billion in annual revenue, per WebProNews — but growth has stalled for the first time since its 2013 founding. A company-wide meeting is scheduled for November 10, with team-level meetings from November 11; the new structure takes effect in January 2027.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6apysfazepnumyeyoomp.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6apysfazepnumyeyoomp.webp" alt="A designer working on a colourful poster on a laptop, with floating AI design tool panels around the desk" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Editorial illustration: AI design tools at work. AI-generated image.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  AI is the cost, not the replacement
&lt;/h2&gt;

&lt;p&gt;The cost side is specific to Canva’s business model. Most of its hundreds of millions of users do not pay, so every AI image or design a free user generates costs the company money. In August, The Information reported Canva had underestimated how heavily its new AI features would be used. Days later it cut its 2026 revenue growth forecast from 30% to 20% — a cut TechCrunch tied to ballooning AI costs — and early investors marked the valuation down to about $35 billion, from $42 billion.&lt;/p&gt;

&lt;p&gt;The pricing whiplash shows how hard the company has found the maths. In April, at its Create event, Canva launched Canva AI 2.0 — agents, connectors to Slack, Gmail and Drive, and memory — and began moving from feature-based subscription tiers to a business built around AI credits. By September the free plan’s AI allowance had fallen from 200 standard uses to 20, and Canva Pro’s price rose from $144 to $180 a year before falling back to $144 within weeks.&lt;/p&gt;

&lt;p&gt;Canva’s answer is to own more of the stack: it has been shipping its own AI models, including a foundation model called Craft, to cut reliance on outside providers — and it plans to sell Craft inside ChatGPT and Gemini, turning two big competitors into distribution channels.&lt;/p&gt;

&lt;h2&gt;
  
  
  And AI is the competition too
&lt;/h2&gt;

&lt;p&gt;The second squeeze is the market itself. TechCrunch notes that Canva faces newer AI design and presentation startups, plus general-purpose chatbots like ChatGPT and Gemini whose image generation lets people put together posters and presentations without opening a design tool at all. That is the demand-side version of the same problem: the thing Canva sells is increasingly available free, inside products people already use. It is a sharp reversal for a company that spent the year betting its future on AI — the April relaunch was supposed to make Canva the place AI-assisted design happens, and instead the AI bill arrived before the AI revenue did, with the planned listing slipping to 2027.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F36plskxgklojvgfs360r.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F36plskxgklojvgfs360r.webp" alt="A glass office tower under dark storm clouds at dusk, one band of floors lit warm" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Editorial illustration: a tech tower under gathering clouds. AI-generated image.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The part the headlines get wrong
&lt;/h2&gt;

&lt;p&gt;It matters that the framing is accurate, because it changes the lesson. Canva’s memo never mentions AI, and a company spokesperson said the changes are not about “using AI to do the work of people who are leaving.” Oxford Economics found AI accounted for about 4.5% of US job losses in 2025 — a real number, but nowhere near the main driver of restructuring. Canva is not the company that replaced its designers with a model. It is the company that built its business around AI features and then discovered it could not afford to give them away.&lt;/p&gt;

&lt;p&gt;That distinction is also why the reaction has been unusually muted. The package — six months’ pay, a laptop, a $10,000 grant with no equity taken — has drawn praise rather than anger. The people paying for the miscalculation are still 1,271 staff, and no severance package changes that. But the story is not “AI took their jobs.” It is “AI cost more than the business model allowed.”&lt;/p&gt;

&lt;h2&gt;
  
  
  What to watch
&lt;/h2&gt;

&lt;p&gt;Three things decide whether this is a one-off correction or the start of a longer squeeze. First, the November 10 all-hands and the January 2027 reorganisation — whether the merged product group actually ships faster, or just smaller. Second, Craft: if Canva can sell its own image model inside ChatGPT and Gemini, the cost problem becomes a revenue line. Third, the growth forecast itself — whether 20% holds. And behind all of it, the question every design-software company is now asking: what is left to sell when the chatbot makes the poster for free?&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Sources: &lt;a href="https://techcrunch.com/2026/11/04/canva-lays-off-over-1200-people-amid-slowing-growth/" rel="noopener noreferrer"&gt;TechCrunch&lt;/a&gt; (November 4, 2026); &lt;a href="https://www.webpronews.com/canva-restructuring-1271-employees-22-percent-growth-slows/" rel="noopener noreferrer"&gt;WebProNews&lt;/a&gt; (November 4, 2026); &lt;a href="https://www.michaelleung.info/latest-news/canva-layoffs-2026-redundancy-package" rel="noopener noreferrer"&gt;Michael Leung&lt;/a&gt; (November 5, 2026, severance details via People Matters); &lt;a href="https://novaknown.com/2026/11/05/canva-layoffs-own-bets-not-ai/" rel="noopener noreferrer"&gt;Novaknown&lt;/a&gt; (November 5, 2026, for the company’s denial, the Canva AI 2.0 history and the Oxford Economics figure); &lt;a href="https://pfmtoday.com/design/canva-layoffs-restructure-1271-roles-22-percent-workforce/" rel="noopener noreferrer"&gt;PFMToday&lt;/a&gt; (November 7, 2026, for the Craft model and the January 2027 timetable). The 1,271 figure covers roles cut, changed or relocated; Canva has not disclosed the split.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>layoffs</category>
      <category>startup</category>
    </item>
    <item>
      <title>Aaronson-Ambainis conjecture proved: an unreleased Anthropic model settles the 18-year-old quantum question, Lean-checked</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Sat, 07 Nov 2026 08:02:53 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/aaronson-ambainis-conjecture-proved-an-unreleased-anthropic-model-settles-the-18-year-old-quantum-c3a</link>
      <guid>https://dev.to/aifrontierpost/aaronson-ambainis-conjecture-proved-an-unreleased-anthropic-model-settles-the-18-year-old-quantum-c3a</guid>
      <description>&lt;p&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/psi-anthropic-aaronson-ambainis-conjecture-proof/" rel="noopener noreferrer"&gt;https://aifrontierpost.com/articles/psi-anthropic-aaronson-ambainis-conjecture-proof/&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;An AI research system running an unreleased Anthropic model has proved the Aaronson-Ambainis conjecture, an 18-year-old open problem in quantum computing — and the Lean proof assistant has verified every step of the argument.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Physical Superintelligence (PSI) announced on November 6, 2026 that its research system, running a non-public Anthropic model, found a proof of the Aaronson-Ambainis conjecture, open since 2008. The Lean proof assistant accepted the proof — and the final version after the system improved the result twice — and the manuscript and Lean formalization are public. It has not been peer reviewed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Who:&lt;/strong&gt; Physical Superintelligence (PSI), an AI-native physics lab; its research system ran instances of a non-public Anthropic model. David Turturean led the work.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What:&lt;/strong&gt; A proof of the Aaronson-Ambainis conjecture, which had been open since 2008 and is counted among the central open problems of quantum complexity theory.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When:&lt;/strong&gt; Announced November 6, 2026. The system had a complete proof during the third day of a run on PSI's own infrastructure.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Verification:&lt;/strong&gt; The Lean proof assistant accepted the proof, and accepted the final version after the system improved the result twice. The manuscript and Lean 4 formalization are public.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The bound:&lt;/strong&gt; The proof shows some bit has influence at least c·v5/2/d6, where c is about 0.0025 — at least (v/d)7.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Caveat:&lt;/strong&gt; The proof has not been peer reviewed, and experts still need to confirm the formal statement Lean checked matches the original conjecture.&lt;/p&gt;

&lt;p&gt;Physical Superintelligence (PSI) announced on November 6, 2026 that a research system it developed internally has proved the Aaronson-Ambainis conjecture, settling a question about quantum computing that has been open since 2008. The system ran instances of a non-public Anthropic model — PSI has not named the model — and had a complete proof during the third day of a run on the company's own infrastructure. The Lean proof assistant, software that checks every step of a mathematical argument by computer, accepted it.&lt;/p&gt;

&lt;p&gt;The conjecture, posed by computer scientists Scott Aaronson and Andris Ambainis, bears on a basic question: when can quantum computers be exponentially faster than ordinary ones? Dramatic quantum speedups, like Peter Shor's 1994 factoring algorithm, rely on structure — a promise that the input hides a rare pattern. Whether quantum computers can be exponentially faster on random inputs, which almost never have such structure, was unknown. Aaronson and Ambainis found a way to settle it with pure mathematics: they stated a conjecture about polynomials and showed that, if it is true, quantum computers are at most polynomially faster on random inputs.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the proof was found
&lt;/h2&gt;

&lt;p&gt;PSI says the system found the proof, then was set to work on related open problems. Along the way it improved the result twice, and Lean accepted the final version. The proof follows a plan laid out by Shachar Lovett and Jiapeng Zhang, who had left one inequality of their plan as a conjecture; PSI's proof replaces that conjectured inequality with a new one, proved in the manuscript, using a pairing argument that builds on David Reimer's Butterfly Lemma.&lt;/p&gt;

&lt;p&gt;The work was led by David Turturean, a member of PSI's technical staff, who developed the research system, scoped the problem, prompted the model to package the results into a first draft, reviewed the draft and is named as responsible for the paper's correctness. Jim Halverson, a PSI adviser, audited the proof, and Phiala Shanahan, another adviser, commented on the Lean formalization. PSI says the mathematical statements, the first draft of the paper and the Lean formalization were produced by AI models.&lt;/p&gt;

&lt;p&gt;The manuscript and the Lean 4 formalization are public, so anyone can replay the encoded argument. A Stanford group — Guy Blanc, Jordan Docter, Carmen Strassle and Li-Yang Tan — had settled the conjecture's prediction in a special case in August 2026, and has been reading PSI's proof of the full conjecture. The group called it "a huge result" that "resolves a decades-old conjecture with important implications for the nature of problems that admit quantum speedups," and is writing a shorter exposition of it for experts.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does — and does not — settle
&lt;/h2&gt;

&lt;p&gt;The proof's headline bound is concrete: if a polynomial's reading varies by an amount v and its degree is d, some bit has influence at least c·v5/2/d6, where c is about 0.0025 — at least (v/d)7. In plain terms, an ordinary computer can now predict a quantum computer's probability of answering yes, up to a small error, for all but a small fraction of random strings, with the number of input reads growing at most like a fixed power of the quantum computer's reads — not exponentially. On yes-or-no questions about random inputs, a quantum computer cannot save exponentially many looks unless the input has a rare pattern.&lt;/p&gt;

&lt;p&gt;That is a result about limits, not about building quantum computers: it says nothing about structured problems like factoring, where quantum speedups remain. And the verification has explicit limits. The proof has not been peer reviewed, the human-readable paper is not itself machine-checked, and a reader must still confirm that the formal statement Lean verified matches the printed conjecture. Unite.AI's report of the announcement notes the same caveats: the formal development contains no proof placeholders or added axioms, and its declarations were replayed through Lean's kernel and through lean4lean, an independent implementation of Lean's type checker.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this proof matters
&lt;/h2&gt;

&lt;p&gt;The result is two stories at once. For quantum computing, it closes one of the field's central open problems and draws a clean boundary: exponential advantage needs structure. For AI, it is a machine-checked proof of a problem that had resisted human mathematicians for 18 years, produced by a model the public cannot use — a reminder of how far internal research systems have moved past what is publicly available. Independent assessments of Anthropic's research pace have pointed the same way: METR estimated in November that Anthropic's AI capabilities are improving at roughly twice the rate of an earlier period, with AI itself responsible for much of the acceleration.&lt;/p&gt;

&lt;p&gt;Whether the proof survives expert scrutiny is now the open question — and the one that will decide whether November 6, 2026 goes down as the day a conjecture became a theorem, or the day a machine-checked argument still needed a human to say what it proved. The formalization is public, the kernel has spoken, and the mathematicians are reading.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What is the Aaronson-Ambainis conjecture?
&lt;/h3&gt;

&lt;p&gt;A 2008 conjecture by Scott Aaronson and Andris Ambainis about polynomials, which implies that quantum computers can be at most polynomially faster than ordinary computers on random inputs — the inputs that almost never carry the structure quantum speedups rely on.&lt;/p&gt;

&lt;h3&gt;
  
  
  Was the proof checked by a human or a machine?
&lt;/h3&gt;

&lt;p&gt;By a machine: the Lean proof assistant accepted the proof, and accepted the final version after the system improved the result twice. It has not been peer reviewed, and experts still need to confirm that the formal statement Lean checked matches the original conjecture.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which Anthropic model proved it?
&lt;/h3&gt;

&lt;p&gt;PSI has not named the model. It describes it only as a non-public Anthropic model, and says instances of it ran inside PSI's own research system on the company's infrastructure.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>quantumcomputing</category>
      <category>research</category>
    </item>
    <item>
      <title>Turn meeting recordings into speaker-labelled transcripts with Nvidia's Nemotron 3.5 Transcribe</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Fri, 30 Oct 2026 03:16:58 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/turn-meeting-recordings-into-speaker-labelled-transcripts-with-nvidias-nemotron-35-transcribe-4033</link>
      <guid>https://dev.to/aifrontierpost/turn-meeting-recordings-into-speaker-labelled-transcripts-with-nvidias-nemotron-35-transcribe-4033</guid>
      <description>&lt;p&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/nemotron-35-transcribe-meetings-tutorial/" rel="noopener noreferrer"&gt;aifrontierpost.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Nemotron 3.5 Transcribe, released October 28, 2026, is one open-weight model for transcription, speaker labels, word timestamps, translation, and audio questions — steered by prompts, not by chaining tools. This guide serves it with vLLM on one H100, prepares audio at 16 kHz mono, and turns its output into speaker-labelled notes and subtitles. Verified without a GPU where that was possible: audio preparation, the client, and both output formats against a mock server; the serve command itself is Nvidia's, quoted verbatim from the model card.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Who:&lt;/strong&gt; Nvidia Nemotron 3.5 Transcribe (October 28, 2026) — 34B total parameters, about 4.3B active per token, 65,536-token context, recordings up to one hour, OpenMDW-1.1 licence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What:&lt;/strong&gt; a vLLM server on one H100, an OpenAI-compatible client, and two output builders: speaker-labelled Markdown notes and a timed SRT file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When:&lt;/strong&gt; verified October 29, 2026: ffmpeg 8.1.2 audio preparation, openai 3.41.0 client against a mock server, Markdown and SRT outputs byte-checked.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Requirement:&lt;/strong&gt; one supported GPU (H100 80 GB reference; see the hardware table in Step 1), Linux or WSL2, Python 3.13, uv, ffmpeg. No API key for the local server.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why this matters now
&lt;/h2&gt;

&lt;p&gt;Meeting transcription used to mean chaining tools together: one model to turn speech into text, another to guess who was speaking, a third to translate, and a pile of glue code to reconcile three outputs that disagreed about where sentences ended. Every handoff was a place for errors to creep in — a speaker change lost between two passes, a timestamp that belonged to a different sentence than the one it labelled.&lt;/p&gt;

&lt;p&gt;On October 28, 2026, Nvidia released Nemotron 3.5 Transcribe, an open-weight speech model that collapses that chain into one promptable model. It is a speech-augmented language model: an audio encoder feeds a hybrid Mamba-Transformer mixture-of-experts language model, so the same recording can produce a verbatim transcript, a cleaned-up readable version, a speaker-labelled transcript, a translation, or an answer to a question about the audio — depending on the instruction you send with it. The model card puts the size at 34 billion total parameters with about 4.3 billion active per token, a 65,536-token context, and recordings up to one hour long. It transcribes 40 languages, translates between English and 31 other languages, and labels up to eight speakers in a recording. The weights are released under the OpenMDW-1.1 licence, which allows commercial use.&lt;/p&gt;

&lt;p&gt;That consolidation is the whole point of this tutorial. You will serve the model on one GPU with vLLM, talk to it through an OpenAI-compatible endpoint, and build the small amount of code that turns its raw output into two things people actually use: readable, speaker-labelled meeting notes and a subtitled file with real word timings. Along the way you will learn which of its tricks are free (prompt-steered style and translation) and which cost you something (word timestamps need a specific server setup, and the fastest serving numbers Nvidia quotes belong to a different, optimised build than the open one).&lt;/p&gt;

&lt;p&gt;One honest boundary before you start: the model itself needs a serious GPU — the reference deployment is one H100 with 80 GB of memory — and this tutorial was written and verified on a machine without one. The server command below is copied from Nvidia's model card, not run here, and the article says so wherever that matters. Everything that could be verified without the GPU was verified: the audio preparation, the client code, and the post-processing, all against a mock server that mimics the real endpoint's reply shape. Rent the GPU for the serving step, or try the hosted demo linked from the model card first, and you will know exactly what to expect when the real server answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you'll need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A GPU machine for serving. The model card's reference is one H100 80 GB, serving the BF16 checkpoint. Its hardware table also lists an A100 80 GB (BF16), an L40S 48 GB (FP8 only), and an RTX 5090 (NVFP4 only), plus Blackwell cards and DGX Spark. A rented cloud GPU or a workstation with one of these cards is the realistic route; this is not a laptop-CPU model. Pick your checkpoint precision from that table, not from habit: the wrong precision for your card simply will not be the supported path.&lt;/li&gt;
&lt;li&gt;Linux (or WSL2 on Windows). The vLLM route below is a Linux recipe; the card notes Linux runtimes also run on Windows through WSL2.&lt;/li&gt;
&lt;li&gt;Python 3.13, uv, and a Hugging Face account for the download. The NeMo Speech setup below uses &lt;code&gt;uv venv --python 3.13&lt;/code&gt;. The checkpoint download is large — plan for a long download and enough disk for the weights plus a working margin, and do not start it on a metered connection without checking the checkpoint size on the model page first.&lt;/li&gt;
&lt;li&gt;ffmpeg on the machine that prepares audio. Any recent ffmpeg works; this is the only step that runs comfortably on an ordinary laptop, and it is where your recordings get normalised before they ever reach the model.&lt;/li&gt;
&lt;li&gt;No API key for the local server. The vLLM server speaks the OpenAI chat-completions format on your own machine, so the client uses a placeholder key. Keep that server on localhost or behind your own authentication if you expose it — it is not a public API with built-in access control.&lt;/li&gt;
&lt;li&gt;Recordings you have the right to process. Speaker labels distinguish voices inside one recording; they do not identify real people, and the model card says so explicitly. Get consent where your jurisdiction or workplace requires it before you transcribe other people's meetings, and treat transcripts of sensitive meetings as sensitive documents.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1 — Choose your precision and your serving route
&lt;/h2&gt;

&lt;p&gt;Three decisions shape everything else, so make them before you download anything. First, precision: BF16 is the full-precision reference, FP8 is a quantised variant, and NVFP4 is the smallest. The model card's hardware table is the authority here — H100 takes BF16 or FP8, RTX 5090 takes NVFP4, L40S takes FP8 — and the checkpoint names follow the same pattern (NVIDIA-Nemotron-3.5-Transcribe-30B-A4B, plus -FP8 and -NVFP4 variants). If your card is not in the table, stop and check the card before downloading tens of gigabytes of weights you cannot serve.&lt;/p&gt;

&lt;p&gt;Second, serving route. This tutorial uses vLLM, because it is the only route that gives you everything: transcription, translation, speaker labels, and word timestamps. SGLang-Omni is simpler — one command, no NeMo — but the model card is explicit that it does not do word timestamps, speaker-attributed output, translation, or speculative decoding; it is a whole-file transcription endpoint. Running the model in-process with Hugging Face Transformers is the third route, useful for a notebook experiment, but it gives up the serving optimisations and the timestamp middleware. The decision table at the end of this guide summarises the trade-off; for meeting notes with speakers and timings, vLLM is the right default.&lt;/p&gt;

&lt;p&gt;Third, expectations about speed. The model card reports batched English ASR throughput on one H100 of roughly 410x real-time with two-token speculative decoding (MTP2) and 270x with eight-token decoding (MTP8) — and it warns, in the same breath, that these are rough batch estimates excluding startup, not single-request latency. Separately, Nvidia quotes an optimised NIM build transcribing up to 2,238 seconds of audio per second on an RTX PRO 6000. That NIM build is not the open-source release, and the card says the open vLLM integration is expected to be roughly two times slower. Do not budget a project on the NIM number; budget on the vLLM number, and measure your own first hour before promising anyone a turnaround time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2 — Install NeMo Speech and vLLM, then download the checkpoint
&lt;/h2&gt;

&lt;p&gt;The model card's environment setup installs vLLM 0.32.0 and the latest NeMo Speech main branch into one environment with uv, then downloads the checkpoint. Run it exactly as written — the two extra packages at the end are not optional extras, they are fixes for gaps in the dependency sets: PyAV decodes compressed formats such as M4A without a system ffmpeg, and vLLM requires OpenCV, which NeMo's dependency settings exclude.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/NVIDIA-NeMo/Speech.git nemo-speech
&lt;span class="nb"&gt;cd &lt;/span&gt;nemo-speech
uv venv &lt;span class="nt"&gt;--python&lt;/span&gt; 3.13 .venv
&lt;span class="nv"&gt;UV_PROJECT_ENVIRONMENT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$PWD&lt;/span&gt;/.venv uv &lt;span class="nb"&gt;sync&lt;/span&gt; &lt;span class="nt"&gt;--extra&lt;/span&gt; asr &lt;span class="nt"&gt;--extra&lt;/span&gt; vllm &lt;span class="nt"&gt;--locked&lt;/span&gt; &lt;span class="nt"&gt;--no-install-project&lt;/span&gt;
uv pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;--python&lt;/span&gt; .venv/bin/python &lt;span class="nt"&gt;--no-deps&lt;/span&gt; &lt;span class="nt"&gt;-e&lt;/span&gt; &lt;span class="nb"&gt;.&lt;/span&gt;
&lt;span class="c"&gt;# PyAV decodes compressed audio formats, such as M4A, without a system FFmpeg.&lt;/span&gt;
uv pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;--python&lt;/span&gt; .venv/bin/python &lt;span class="nt"&gt;--no-deps&lt;/span&gt; &lt;span class="nv"&gt;av&lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;19.0.2
&lt;span class="c"&gt;# vLLM requires OpenCV, which NeMo's dependency settings exclude.&lt;/span&gt;
uv pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;--python&lt;/span&gt; .venv/bin/python &lt;span class="nt"&gt;--no-deps&lt;/span&gt; opencv-python-headless&lt;span class="o"&gt;==&lt;/span&gt;5.0.0.93
&lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate
hf download nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B &lt;span class="nt"&gt;--local-dir&lt;/span&gt; ./NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you chose FP8 or NVFP4 in Step 1, substitute that checkpoint name in the download command. The &lt;code&gt;hf&lt;/code&gt; command comes with the Hugging Face tooling in the environment; if it is not on your path after activation, install &lt;code&gt;huggingface_hub&lt;/code&gt; into the same environment and retry before assuming the download failed. Expect this step to take a while: you are downloading a frontier-scale checkpoint, not a pip package, and a failed download halfway through is usually a disk-space or network problem, not a model problem. Check both before re-running.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3 — Start the vLLM server (the command, explained flag by flag)
&lt;/h2&gt;

&lt;p&gt;Here is the model card's server command, verbatim. The environment variables come first because they change behaviour the flags cannot: the plugin that teaches vLLM about this model, the one-hour audio limit (the default is 600 seconds), the 180 MB per-clip file-size cap, and the V1 model runner, which word timestamps require.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;VLLM_PLUGINS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;nemo_speechlm
&lt;span class="c"&gt;# Allow vLLM to decode recordings up to one hour long (the default is 600 seconds).&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;VLLM_MAX_AUDIO_DECODE_DURATION_S&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;3600
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;VLLM_MAX_AUDIO_CLIP_FILESIZE_MB&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;180
&lt;span class="c"&gt;# Word timestamps require the V1 model runner.&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;VLLM_USE_V2_MODEL_RUNNER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0

vllm serve ./NVIDIA-Nemotron-3.5-Transcribe-30B-A4B &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--served-model-name&lt;/span&gt; nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--trust-remote-code&lt;/span&gt; &lt;span class="nt"&gt;--dtype&lt;/span&gt; bfloat16 &lt;span class="nt"&gt;--gpu-memory-utilization&lt;/span&gt; 0.94 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--max-model-len&lt;/span&gt; 65536 &lt;span class="nt"&gt;--max-num-batched-tokens&lt;/span&gt; 4096 &lt;span class="nt"&gt;--max-num-seqs&lt;/span&gt; 192 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--enable-chunked-prefill&lt;/span&gt; &lt;span class="nt"&gt;--no-enable-prefix-caching&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--use-replayssm&lt;/span&gt; &lt;span class="nt"&gt;--mamba-backend&lt;/span&gt; flashinfer &lt;span class="nt"&gt;--mamba-cache-mode&lt;/span&gt; none &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--mm-processor-cache-gb&lt;/span&gt; 0 &lt;span class="nt"&gt;--no-async-scheduling&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--limit-mm-per-prompt&lt;/span&gt; &lt;span class="s1"&gt;'{"audio":{"count":6,"length":120000000}}'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--speculative-config&lt;/span&gt; &lt;span class="s1"&gt;'{"method":"mtp","num_speculative_tokens":8}'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--reasoning-parser&lt;/span&gt; nemotron_v3 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--trust-request-mm-kwargs&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--middleware&lt;/span&gt; nemo.collections.speechlm2.vllm.salm.ctc_serving.ctc_timestamp_middleware
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What the flags you are most likely to touch actually do: &lt;code&gt;--gpu-memory-utilization 0.94&lt;/code&gt; lets vLLM use 94% of GPU memory — lower it if the card runs other work. &lt;code&gt;--max-model-len 65536&lt;/code&gt; matches the model's context window; a dense one-hour conversation can need about 20,000 output tokens, so do not shrink this casually. &lt;code&gt;--no-enable-prefix-caching&lt;/code&gt; and the ReplaySSM flags are not tuning knobs to experiment with on day one; the card instructs you to keep them as written. The last two flags are the timestamp switch: &lt;code&gt;--trust-request-mm-kwargs&lt;/code&gt; plus the CTC timestamp middleware are what let a request opt into word timestamps in Step 6. Leave them out and timestamps silently will not be available, which is the single most common setup mistake with this model.&lt;/p&gt;

&lt;p&gt;The server listens on &lt;code&gt;127.0.0.1:8000&lt;/code&gt;; use &lt;code&gt;--host&lt;/code&gt; and &lt;code&gt;--port&lt;/code&gt; to change that. The first requests after startup are slower while kernels compile and warm up — the card says so, and it is worth knowing before you judge the server on its first transcript. If you want the DFlash2 draft model instead of MTP speculative decoding, read the card's DFlash2 section first: it needs a small patch to vLLM 0.32.0's DFlash2 code, runs only on the V2 runner (so you unset the V1 variable), and word timestamps are not available with it. For meeting notes, stay with MTP and keep your timestamps.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4 — Prepare your recording: 16 kHz, mono, under one hour
&lt;/h2&gt;

&lt;p&gt;The model card's recipe uses 16 kHz audio, and the server caps a clip at one hour and 180 MB. Normalise before you send anything: one ffmpeg command handles the sample rate, the channel count, and the container in a single pass, whatever your recorder produced.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ffmpeg &lt;span class="nt"&gt;-y&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; meeting_raw.m4a &lt;span class="nt"&gt;-ar&lt;/span&gt; 16000 &lt;span class="nt"&gt;-ac&lt;/span&gt; 1 meeting_16k.wav
ffprobe &lt;span class="nt"&gt;-v&lt;/span&gt; error &lt;span class="nt"&gt;-show_entries&lt;/span&gt; &lt;span class="nv"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;sample_rate,channels,codec_name &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-show_entries&lt;/span&gt; &lt;span class="nv"&gt;format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;duration &lt;span class="nt"&gt;-of&lt;/span&gt; &lt;span class="nv"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;noprint_wrappers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 meeting_16k.wav
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Verified here on a generated test recording: a 44.1 kHz stereo file converted cleanly to 16,000 Hz mono PCM with its 5-second duration intact, which is exactly the shape the server expects. If your recording is longer than an hour, split it before transcribing — ffmpeg's &lt;code&gt;-ss&lt;/code&gt; and &lt;code&gt;-t&lt;/code&gt; options cut clean segments — and transcribe each part separately. One consequence to plan for: speaker tags restart with each request, so &lt;code&gt;spk:0&lt;/code&gt; in part two is not necessarily the same person as &lt;code&gt;spk:0&lt;/code&gt; in part one. Keep a mapping per part, or better, keep meetings under an hour per request whenever the agenda allows it. The same applies to the 180 MB cap: a long, high-bitrate original that survives as a WAV will trip it, which is another reason to normalise to 16 kHz mono first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5 — Send your first transcript request
&lt;/h2&gt;

&lt;p&gt;The client below follows the model card's request function, with one change you should know about: the model name is a constant, so you cannot typo it differently in two places. Audio travels as a base64 data URL inside an OpenAI-style chat message, which is why the same code works against any OpenAI-compatible server, including the mock used for verification here.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;base64&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Path&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;

&lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://127.0.0.1:8000/v1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;not-needed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;audio_path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;system_prompt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reasoning&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timestamps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;audio&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;base64&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;b64encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;audio_path&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read_bytes&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;system&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;system_prompt&lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;system_prompt&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;user&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;audio_url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;audio_url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data:audio/wav;base64,&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;audio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}},&lt;/span&gt;
    &lt;span class="p"&gt;]})&lt;/span&gt;
    &lt;span class="n"&gt;extra_body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chat_template_kwargs&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;enable_thinking&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;reasoning&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;skip_special_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# keep speaker and event tags
&lt;/span&gt;        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spaces_between_special_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;seed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;reasoning&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;extra_body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="o"&gt;=-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;thinking_token_budget&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;17408&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;extra_body&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chat_template_kwargs&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;reasoning_budget&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;16384&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;timestamps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;extra_body&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mm_processor_kwargs&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;capture_ctc_timestamps&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.6&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;reasoning&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;top_p&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.95&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;reasoning&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20480&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;reasoning&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;16384&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;extra_body&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;extra_body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;meeting_16k.wav&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Transcribe the following audio.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two settings in that function deserve emphasis because they are easy to get wrong. Transcription, translation, and language identification all use temperature 0 with reasoning off — deterministic output for a job where creativity is a bug. And &lt;code&gt;skip_special_tokens&lt;/code&gt; stays &lt;code&gt;False&lt;/code&gt; on purpose: setting it to &lt;code&gt;True&lt;/code&gt; would strip the speaker and event tags that Step 6 depends on. The sampling rule flips only for audio understanding in Step 7, where reasoning goes on and temperature rises to 0.6 with top-p 0.95.&lt;/p&gt;

&lt;p&gt;Verification note, stated plainly: this client and everything after it were run against a mock server returning a canned two-speaker reply, because no H100 was available to this tutorial's verification machine. The mock proved the request shape (model name, temperature 0, the timestamp flag, and a base64 audio data URL), the speaker-tag parsing, and both output formats in Step 6 — but the transcript words in the examples below are illustrative, not model output. Treat your first real transcript as a test: run a two-minute clip you already know well, read it end to end, and only then hand the pipeline a real meeting.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6 — Get speakers and word timings, then turn them into notes and subtitles
&lt;/h2&gt;

&lt;p&gt;A plain transcript answers "what was said". A meeting note answers "who said it, and when" — and that needs two things at once: the speaker-tagged prompt from the model card's prompt guide, and the timestamp flag from Step 5. Ask for both together, then check the alignment error field before you trust a single timestamp. The card is explicit: timestamp requests accept one audio recording, &lt;code&gt;stream=false&lt;/code&gt;, and &lt;code&gt;n=1&lt;/code&gt;; a transcript without speaker tags is aligned as a single speaker; and when you are evaluating diarization itself, use the diarization segments, not the word timestamps.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;SPEAKER_PROMPT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Transcribe this audio with punctuation and capitalization. &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Include all spoken words and verbal fillers. &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Write numbers in their standard written form. &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Put speaker tags &amp;lt;spk:0&amp;gt; &amp;lt;spk:1&amp;gt; &amp;lt;spk:2&amp;gt; &amp;lt;spk:3&amp;gt; on the transcription.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;meeting_16k.wav&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SPEAKER_PROMPT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timestamps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;alignment&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;model_dump&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ctc_timestamps&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;alignment&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;alignment&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;word&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;alignment&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;words&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;word&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;start&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;speaker&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;ctc_timestamps&lt;/code&gt; object gives you four things: &lt;code&gt;words&lt;/code&gt; (word text, start and end in seconds, and the transcript's speaker tag), &lt;code&gt;diarization&lt;/code&gt; (speaker activity segments from the audio), a mapping between transcript tags and diarization speakers, and the &lt;code&gt;error&lt;/code&gt; field. The two small functions below turn the words into the two outputs people actually open. One subtitle cue per speaker turn, timed by its first and last word, keeps subtitles readable; grouping the same words by speaker with a start time gives you meeting notes you can skim.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fmt_ts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seconds&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;seconds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seconds&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;divmod&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seconds&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;divmod&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rem&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;seconds&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seconds&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;03&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;words_to_srt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;words&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;cues&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;words&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;speaker&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;speaker&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spk:0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;speaker&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;speaker&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;words&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;word&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
            &lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;speaker&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;speaker&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;words&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;word&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt;
                       &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;start&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;start&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;
            &lt;span class="n"&gt;cues&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cue&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cues&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;label&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Speaker &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cue&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;speaker&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;)[1]) + 1&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;fmt_ts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cue&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;start&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; --&amp;gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;fmt_ts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cue&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                   &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;] &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cue&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;words&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;words_to_markdown&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;words&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;words&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;speaker&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;speaker&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spk:0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;speaker&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;speaker&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;words&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;word&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;speaker&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;speaker&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;words&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;word&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt;
                       &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;start&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;start&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;
            &lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;turn&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;label&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Speaker &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;speaker&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;)[1]) + 1&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;**&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;** [&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;fmt_ts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;start&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;])[&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;] &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                   &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;words&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run against the mock server, those functions produce exactly what you would expect from a two-speaker opening — Markdown reading &lt;strong&gt;Speaker 1&lt;/strong&gt; [00:00:00] Good morning, everyone. followed by &lt;strong&gt;Speaker 2&lt;/strong&gt; [00:00:01] Let's start with the budget., and an SRT file with matching cues (00:00:00,120 to 00:00:01,300, then 00:00:01,600 to 00:00:03,000). The words are canned mock data, so the point of the test is the shape, not the sentence: the same code paths, fed with a real server's &lt;code&gt;ctc_timestamps&lt;/code&gt;, produce real notes and real subtitles without modification. Two details in the code are worth keeping: a word with no speaker tag falls back to &lt;code&gt;spk:0&lt;/code&gt; rather than crashing, and the Markdown timestamp drops the milliseconds for readability while the SRT keeps them, because subtitles need the precision and notes do not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 7 — Change the job with the prompt: style, vocabulary, translation, questions
&lt;/h2&gt;

&lt;p&gt;This is where one model replaces four tools, and the model card's prompt guide is the menu. Keep the system prompt empty for transcription and translation — that is the card's instruction, not a suggestion — and put the job in the user message.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Verbatim vs readable.&lt;/strong&gt; For a record you may quote, ask for wording verbatim, including fillers and repetitions. For captions and searchable notes, ask it to omit fillers and accidental repetitions while keeping the remaining wording unchanged. Both prompts add punctuation, capitalisation, and "transcript only" as the output format, so the model does not summarise when you asked it to transcribe.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Domain vocabulary.&lt;/strong&gt; Add context terms to the prompt — product names, drug names, internal jargon, the participants' names — instead of retraining anything. The card's contextual-transcription prompt is simply the transcription instruction plus &lt;code&gt;Context terms: &amp;lt;names and terms&amp;gt;&lt;/code&gt;. This is the cheapest accuracy win in the whole tutorial, and the one most people skip.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Translation.&lt;/strong&gt; &lt;code&gt;Please translate the given speech to &amp;lt;target language&amp;gt;.&lt;/code&gt; That is the whole prompt. Translation runs at temperature 0 with reasoning off, like transcription, and covers English to and from 31 other languages. Check the card's language matrix for your pair before promising a translation: transcription covers 40 languages, but not every transcription language is a translation language.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Questions about the audio.&lt;/strong&gt; Ask a specific question and turn reasoning on — that flips the sampling to temperature 0.6, top-p 0.95, a 20,480-token completion limit, and a 16,384-token reasoning budget, as in Step 5's function. This is the mode for "what did we decide about the launch date?" after you already have the transcript, not a replacement for reading it. Audio understanding was evaluated mainly in English, per the card, so treat non-English question-answering as experimental.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Only the nearest speaker (experimental).&lt;/strong&gt; The card offers a dictation-style prompt that transcribes only the speaker nearest the microphone and ignores other voices. It is labelled experimental in the guide; use it for voice interfaces and dictation, not for meetings where every voice matters.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 8 — Batch a folder, then run the quality checks that actually catch errors
&lt;/h2&gt;

&lt;p&gt;One meeting is a demo; a folder of meetings is a workflow. The full client script used for verification in this tutorial — &lt;code&gt;transcribe.py&lt;/code&gt;, with &lt;code&gt;--mode speaker|readable|verbatim|translate|qa&lt;/code&gt;, &lt;code&gt;--format md|srt|txt&lt;/code&gt;, &lt;code&gt;--context&lt;/code&gt; for vocabulary terms, and &lt;code&gt;--out&lt;/code&gt; for the output file — wraps Steps 5–7 into one command per file, so a folder batch is a shell loop, not a new program:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;mkdir&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; transcripts
&lt;span class="k"&gt;for &lt;/span&gt;f &lt;span class="k"&gt;in &lt;/span&gt;recordings/&lt;span class="k"&gt;*&lt;/span&gt;_16k.wav&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;&lt;span class="nv"&gt;base&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;basename&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; _16k.wav&lt;span class="si"&gt;)&lt;/span&gt;
  python transcribe.py &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--mode&lt;/span&gt; speaker &lt;span class="nt"&gt;--format&lt;/span&gt; md &lt;span class="nt"&gt;--out&lt;/span&gt; &lt;span class="s2"&gt;"transcripts/&lt;/span&gt;&lt;span class="nv"&gt;$base&lt;/span&gt;&lt;span class="s2"&gt;.md"&lt;/span&gt;
  python transcribe.py &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--mode&lt;/span&gt; speaker &lt;span class="nt"&gt;--format&lt;/span&gt; srt &lt;span class="nt"&gt;--out&lt;/span&gt; &lt;span class="s2"&gt;"transcripts/&lt;/span&gt;&lt;span class="nv"&gt;$base&lt;/span&gt;&lt;span class="s2"&gt;.srt"&lt;/span&gt;
&lt;span class="k"&gt;done
&lt;/span&gt;&lt;span class="nb"&gt;ls &lt;/span&gt;transcripts/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then, before anyone relies on a transcript, run these checks — they are the ones that catch the failure modes the model card itself lists: substitutions, deletions, repetitions, hallucinated speech, wrong language choice, and speaker-attribution errors.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Check the error field first.&lt;/strong&gt; If &lt;code&gt;ctc_timestamps["error"]&lt;/code&gt; is not null, the timestamps are not usable, even if the text looks fine. Do not ship subtitles from a failed alignment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spot-check names, numbers, and speaker changes.&lt;/strong&gt; Read the first two minutes and every speaker change against the audio. Names and domain terms are where transcription fails most visibly, and speaker changes are where diarization fails most quietly. If a name is consistently wrong, add it as a context term and re-run that file — that is what the vocabulary prompt is for.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch overlapping speech and long files.&lt;/strong&gt; The card recommends targeted testing for long recordings, overlapping speakers, noise, names, and code-switching. A panel discussion with constant crosstalk is a harder test than a two-person interview; grade your pipeline on the hard file, not the easy one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Collapse, but do not trust, repeated lines.&lt;/strong&gt; A cheap guard in the verification script drops exact consecutive duplicate lines, which catches the visible symptom of a generation loop. It is not a cure: if a transcript repeats itself, re-run it and inspect the audio around the loop before using that section.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stop the server when you are done.&lt;/strong&gt; A rented GPU bills whether you are transcribing or not. The vLLM process holds the card's memory until you stop it, so make shutting it down part of the workflow, not an afterthought.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Which approach should you use?
&lt;/h2&gt;

&lt;p&gt;The choice is not "which model" but "which serving route, for which job" — and the model card draws the lines clearly.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Route&lt;/th&gt;
&lt;th&gt;Use it when&lt;/th&gt;
&lt;th&gt;What you give up&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;vLLM + NeMo Speech (this tutorial)&lt;/td&gt;
&lt;td&gt;You need speaker labels, word timestamps, translation, or speculative decoding from one server.&lt;/td&gt;
&lt;td&gt;The heaviest setup: a GPU machine, the NeMo environment, and the full serve command.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SGLang-Omni&lt;/td&gt;
&lt;td&gt;You need whole-file transcription quickly and nothing else, on one GPU.&lt;/td&gt;
&lt;td&gt;Word timestamps, speaker-attributed output, translation, and speculative decoding are all absent.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Transformers, in-process&lt;/td&gt;
&lt;td&gt;You are experimenting in a notebook and want the model as a Python object.&lt;/td&gt;
&lt;td&gt;Serving optimisations and the timestamp middleware; you manage memory and batching yourself.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The hosted demo linked from the model card&lt;/td&gt;
&lt;td&gt;You want to hear what the model does on your audio before renting a GPU.&lt;/td&gt;
&lt;td&gt;Your audio leaves your machine, and you get a demo, not a pipeline.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A smaller local transcriber (e.g. faster-whisper)&lt;/td&gt;
&lt;td&gt;You have no GPU budget and your meetings are single-speaker or speaker labels do not matter.&lt;/td&gt;
&lt;td&gt;Accuracy on multi-speaker, multilingual, and long-form material — the model card's benchmark leads are for this model, not for the smaller one.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;One more comparison, because it decides budgets: the model card's headline benchmark figures, captured October 12, 2026, are 4.71% mean word error rate on nine English sets, 4.62% across seven multilingual sets, and 15.80% mean cpWER on full multi-speaker sessions — each the best open-weight figure the card lists against its next-best open-weight rival. Those are the card's numbers, not measurements from this tutorial, and they describe the model, not your recordings. Your accent mix, your room noise, and your vocabulary will move them. The honest way to compare against your current tool is a side-by-side on ten of your own recordings, scored by hand on names, numbers, and speaker changes — the errors that cost you time, not the average that flatters a leaderboard.&lt;/p&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;Nemotron 3.5 Transcribe earns its place by removing glue code: one open-weight model, one prompt, and you choose between a verbatim record, a readable transcript, a speaker-labelled meeting note, a translation, or an answer about the audio. The price is a real GPU, a careful server setup where the timestamp flags are easy to forget, and the discipline to verify the first transcripts by ear before trusting the pipeline. Do the setup once, keep the prompt guide beside the client, add your vocabulary as context terms, and check the alignment error field every single run. The model does the transcribing; the workflow around it — normalised audio, the right prompt, and five minutes of spot-checking — is what makes the transcript trustworthy.&lt;/p&gt;

&lt;p&gt;Sources: Nvidia's model card for nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B on Hugging Face (architecture, 34B total / 4.3B active parameters, 65,536-token context, one-hour recordings, 40 transcription languages, OpenMDW-1.1 licence, hardware compatibility table, vLLM 0.32.0 + NeMo Speech environment setup, the vLLM serve command and its environment variables, the request function and sampling rules, the prompt guide, word-timestamp and diarization rules, SGLang-Omni limits, DFlash2 notes, and benchmark results captured October 12, 2026, all read October 29, 2026); launch coverage for the translation count (31 languages to or from English), speaker labelling (13 languages, up to eight speakers), and the NIM-versus-open-release speed distinction; and the verification runs quoted in Steps 4–6 and 8 (ffmpeg 16 kHz mono conversion, mock-server client runs with openai 3.41.0, Markdown and SRT outputs, and the flattened extra_body wire format). The vLLM serve command and the benchmark figures are the model card's, not runs performed here, and the article says so where it uses them.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>opensource</category>
      <category>transcription</category>
    </item>
    <item>
      <title>PhotoCraft: the open-source Photoshop rival you can drive from the terminal</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Thu, 29 Oct 2026 00:13:32 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/photocraft-the-open-source-photoshop-rival-you-can-drive-from-the-terminal-5ag1</link>
      <guid>https://dev.to/aifrontierpost/photocraft-the-open-source-photoshop-rival-you-can-drive-from-the-terminal-5ag1</guid>
      <description>&lt;p&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/photocraft-headless-photo-editor-cli-tutorial/" rel="noopener noreferrer"&gt;aifrontierpost.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Photoshop rebuilt from scratch in Rust, with 74,000 stars in four weeks — and a command-line tool that edits real photos without ever opening a window. This guide downloads the real release, verifies its checksum, and runs every headless command that matters: inspect, edit, batch, convert, and MCP. No account, no API key, no GPU, and every number below is one the installed tool printed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; PhotoCraft (storytold/photocraft) is an open-source, pure-Rust Photoshop rebuild: version 0.17.0 ships a headless CLI that inspects, edits, batch-grades, converts (including PSD), and serves the engine over MCP — all verified here with no account, no API key, and no GPU. The desktop app is early alpha and was not tested; the CLI path is the proven one.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Who:&lt;/strong&gt; PhotoCraft (storytold/photocraft on GitHub), from the ArtCraft team — 74,359 stars, 7,040 commits, 13,130 forks, dual MIT/Apache-2.0, first committed September 30, 2026.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What:&lt;/strong&gt; a clean-room Photoshop rebuild in pure Rust, plus a headless CLI: info, run, batch, convert, commands, and an MCP server — verified here end to end.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When:&lt;/strong&gt; version verified in this guide is photocraft-cli 0.17.0, released and run October 28, 2026; star and commit counts are dated the same day.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Requirement:&lt;/strong&gt; a Linux, macOS, or Windows machine and ~200 MB of free disk for the prebuilt release. No account, no API key, no GPU, no Rust toolchain.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why this is blowing up now
&lt;/h2&gt;

&lt;p&gt;PhotoCraft is a clean-room reimplementation of Photoshop, written from scratch in pure Rust. Layers, masks, adjustment layers, layer styles, type, vectors, brushes, and real PSD files — in a native app with no Electron and no web view. The repository (storytold/photocraft on GitHub) had 74,359 stars, 7,040 commits, and 13,130 forks when this guide read it on October 28, 2026, under a dual MIT/Apache-2.0 license. It was created on September 30, 2026, which makes that star count a four-week sprint, not a slow accumulation. Version 0.17.0 landed the same day this guide ran, October 28, 2026.&lt;/p&gt;

&lt;p&gt;The part that makes it tutorial-worthy instead of just impressive is not the GUI. It is the headless command-line tool that ships in the same release. Every menu item, tool, and dialog in PhotoCraft runs a command from one registry of 500+ commands, and the UI, the CLI, the JSON control channel, and the MCP server all call the same commands. That means anything you can click, a script can do too — and this guide proves it, because every command below was run on a headless Linux server with no display, no account, no API key, and no GPU.&lt;/p&gt;

&lt;p&gt;One boundary first, because it decides whether this tutorial is for you. PhotoCraft is in early alpha, and its own README says so plainly: much of Photoshop's feature surface exists in some form, but it is not yet a Photoshop replacement for daily professional work. The gaps it names are AI/generative features, 15 missing tools, depth in typography and pro workflows, and plug-in compatibility. What this guide verifies is the headless editing path — install, inspect, edit, batch, convert, and hand the engine to an agent over MCP. If you need a finished desktop replacement today, this is not that article. If you want a scriptable, offline, open-source photo engine, keep reading.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you'll need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Linux, macOS, or Windows machine. This guide ran on Linux (x86_64), using the prebuilt release tarball — no Rust toolchain, no source build, no compile wait.&lt;/li&gt;
&lt;li&gt;About 200 MB of free disk. The Linux tarball is 123 MB; extracted, it is roughly twice that.&lt;/li&gt;
&lt;li&gt;A terminal and a scratch directory with a photo or two to practise on. Do not practise on originals: the CLI overwrites nothing by default, but work on copies anyway.&lt;/li&gt;
&lt;li&gt;No account, no API key, no GPU, no sign-in of any kind for anything in this guide. The desktop app's smart-selection features run on your machine with no cloud and no account, per the README; the CLI path here touches no network at all after the download.&lt;/li&gt;
&lt;li&gt;Optional but useful: Python with Pillow, only if you want to diff pixels before and after like this guide did. It is not required for any step.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One photo in, one graded photo out: the CLI runs the same engine commands the desktop app runs. Image: AI Frontier Post.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Get the real binary and check it
&lt;/h2&gt;

&lt;p&gt;Download the release for your platform from the project's releases page. On Linux x86_64, the file is photocraft-0.17.0-linux-x86_64.tar.gz. The release also ships a SHA256SUMS.txt — check your download against it before you run anything from it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-LO&lt;/span&gt; https://github.com/storytold/photocraft/releases/download/v0.17.0/photocraft-0.17.0-linux-x86_64.tar.gz
curl &lt;span class="nt"&gt;-LO&lt;/span&gt; https://github.com/storytold/photocraft/releases/download/v0.17.0/SHA256SUMS.txt
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="s2"&gt;"linux-x86_64.tar.gz"&lt;/span&gt; SHA256SUMS.txt
&lt;span class="nb"&gt;sha256sum &lt;/span&gt;photocraft-0.17.0-linux-x86_64.tar.gz
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected result, verified October 28, 2026: the hash printed by sha256sum is 5259f9c428c8ee3478c35500ece98713561cafc050e4c9eb354a5050f36eb2b8, matching the release checksum. If yours differs, stop — re-download, and do not run the binary.&lt;/p&gt;

&lt;p&gt;Extract it and confirm the tool answers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;tar&lt;/span&gt; &lt;span class="nt"&gt;-xzf&lt;/span&gt; photocraft-0.17.0-linux-x86_64.tar.gz
./photocraft-0.17.0-linux-x86_64/bin/photocraft-cli &lt;span class="nt"&gt;--version&lt;/span&gt;
./photocraft-0.17.0-linux-x86_64/bin/photocraft-cli &lt;span class="nt"&gt;--help&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;photocraft-cli 0.17.0 (b8c3f3aa8, 2026-10-28)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The help listing shows eight subcommands: convert, info, run, batch, droplet, commands, mcp, and serve. This guide runs the first four plus commands and mcp. The tarball also contains the desktop app binary (photocraft), which needs a display and is not used here. Put the CLI on your PATH if you plan to use it regularly; the rest of this guide calls it as photocraft-cli for brevity.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Inspect a photo before you touch it
&lt;/h2&gt;

&lt;p&gt;Copy a photo into a scratch directory and ask the CLI what it sees. This guide used a 640×400 test image; substitute your own file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;mkdir &lt;/span&gt;photocraft-demo
&lt;span class="nb"&gt;cp &lt;/span&gt;your-photo.jpg photocraft-demo/
&lt;span class="nb"&gt;cd &lt;/span&gt;photocraft-demo
photocraft-cli info your-photo.jpg
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected output (abridged): a JSON document describing the image — its channels (Red, Green, Blue), composite mode (RGB), and layer tree, starting with "activeLayer": 2 for a flat import. The info command changes nothing; it is the read-only way to confirm the CLI can parse your file before you build an edit chain on top of it. If info fails on your file, the edit steps will too, and you have found that out cheaply. Run it on every new format you plan to feed the tool: PNG, JPEG, WebP, TIFF, and PSD are all on the convert list, but your specific file is the only test that matters.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Make your first headless edit
&lt;/h2&gt;

&lt;p&gt;The core command is run: open a file, apply engine commands in order, save the result. This is the exact shape from the project's README, run here verbatim — a Smart Sharpen filter followed by a Curves adjustment layer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;photocraft-cli run your-photo.jpg &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--cmd&lt;/span&gt; filter.sharpen.smartSharpen &lt;span class="nt"&gt;--params&lt;/span&gt; &lt;span class="s1"&gt;'{"amount":80}'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--cmd&lt;/span&gt; layer.newAdjustmentLayer.curves &lt;span class="nt"&gt;--params&lt;/span&gt; &lt;span class="s1"&gt;'{"points":[[0,0],[64,48],[192,212],[255,255]]}'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--out&lt;/span&gt; graded-photo.png
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected output, as printed by the run in this guide:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"filter.sharpen.smartSharpen"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"result"&lt;/span&gt;&lt;span class="p"&gt;:{&lt;/span&gt;&lt;span class="nl"&gt;"filter"&lt;/span&gt;&lt;span class="p"&gt;:{&lt;/span&gt;&lt;span class="nl"&gt;"amount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;80.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"filter"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"smartSharpen"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"radius"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"reduceNoise"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;10.0&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="nl"&gt;"layer"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"layer.newAdjustmentLayer.curves"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"result"&lt;/span&gt;&lt;span class="p"&gt;:{&lt;/span&gt;&lt;span class="nl"&gt;"layer"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="err"&gt;warning:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;layer(s)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;flattened;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;layers,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;masks&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;blend&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;modes&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;are&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;kept&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read that output carefully, because it teaches you three things. First, every command echoes its result as JSON, so a script can check success per step instead of trusting an exit code alone. Second, the defaults are visible: Smart Sharpen at amount 80 also applied radius 1.0 and noise reduction 10.0, which the command line never mentioned — when you tune a look, query the result JSON to see what actually ran. Third, the warning is real and worth understanding: PNG output is flat, so the adjustment layer and the sharpened layer were flattened into one image. The layers survive only in layered formats (the native .pcraft format, PSD, or TIFF with the --tiff-layers flag in batch mode). If your workflow needs the layers back later, export to a layered format; if you need a finished file for the web, PNG is the right, honest end of the chain.&lt;/p&gt;

&lt;p&gt;Verify the edit did something, rather than assuming it. Open both files and compare: the output keeps the input's dimensions (640×400 in this guide's test) and the pixels change — a sample pixel moved from RGB (240, 230, 180) to (247, 241, 199) after the sharpen-plus-curves chain. Your numbers will differ with your photo and your curve points; what matters is that you check, because a command that silently no-ops is worse than one that errors.&lt;/p&gt;

&lt;p&gt;To find more commands, ask the registry itself instead of guessing names:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;photocraft-cli commands | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="nt"&gt;-E&lt;/span&gt; &lt;span class="s2"&gt;"vibrance|curves|levels|hue"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This lists real, callable command IDs with their menu paths — for example layer.newAdjustmentLayer.vibrance, image.adjustments.levels, and layer.newAdjustmentLayer.hueSaturation. Every command in this guide came from that list or from the README; none was invented, and you should hold any other tutorial to the same rule.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: Grade a whole folder with one recipe file
&lt;/h2&gt;

&lt;p&gt;Single-image edits are a demo. The real use case is a folder: a shoot, a product catalog, a set of blog images that all need the same grade. That is what batch is for. Write the recipe as a JSON list of steps, one file per look:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; grade.json &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
[{"command":"filter.sharpen.smartSharpen","params":{"amount":80}}]
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;&lt;span class="nb"&gt;mkdir &lt;/span&gt;raw graded
&lt;span class="nb"&gt;cp &lt;/span&gt;photo-a.jpg photo-b.jpg raw/
photocraft-cli batch &lt;span class="nt"&gt;--actions&lt;/span&gt; grade.json &lt;span class="nt"&gt;--in&lt;/span&gt; ./raw &lt;span class="nt"&gt;--out&lt;/span&gt; ./graded
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected output, verified with two test images:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ok    ./raw/a.png -&amp;gt; ./graded/a.png
ok    ./raw/b.png -&amp;gt; ./graded/b.png
2 succeeded, 0 failed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details in that recipe are load-bearing, and both were learned the hard way in this guide's sandbox. First, each step needs the key "command". The batch help also describes [id, params] pairs and bare ids, but a step written with "cmd" instead of "command" fails outright with error: invalid parameters for &lt;code&gt;batch --actions&lt;/code&gt;: step needs "command" — that exact error is what this guide got on its first attempt, and the fix is the one-word key change above. Second, batch refuses to write into the input folder unless you pass --in-place; keep input and output separate, as above, so a bad recipe never eats your originals. Useful extras on the same command: --format webp to change the output format regardless of input extension, and --quality 85 for lossy JPEG/WebP output (without a quality value, WebP output is lossless).&lt;/p&gt;

&lt;p&gt;Batch mode: one recipe file, every image in the folder, one graded set out. Image: AI Frontier Post.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: Convert formats, including real PSD files
&lt;/h2&gt;

&lt;p&gt;PhotoCraft's headline format claim is PSD without compromise: its PSD support is a standalone crate written from Adobe's public specification, and the project reports re-saving renders identically for 307 of the 309 files in the psd-tools test set. You do not need Photoshop to test the round trip at the CLI level:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;photocraft-cli convert your-photo.png your-photo.psd
photocraft-cli info your-photo.psd
photocraft-cli convert your-photo.png your-photo.webp
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All three commands ran cleanly in this guide's sandbox: the PNG became an 84,898-byte PSD that info reads back as a valid RGB document, and the same PNG became a 1,248-byte WebP. The convert list covers .pcraft, .psd, .png, .jpg, .tif, .webp, .exr, and more, with the output extension picking the format unless --format overrides it. Two honest limits, both from the README: HEIC photos from iPhone and Mac are read-only, and AVIF is write-only in official builds. And the PSD fidelity figures above are the project's corpus numbers, not something this guide re-measured — treat them as the project's claim, verified here only to the extent that a real PSD file was produced and re-read.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6: Let an agent drive the same engine over MCP
&lt;/h2&gt;

&lt;p&gt;The same engine is also an MCP server, which is the step that matters if you want a coding agent to edit images instead of you typing commands. Start it on stdio with explicit read and write roots — an omitted root denies that direction of file access, so grant only the scratch directory:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;photocraft-cli mcp &lt;span class="nt"&gt;--automation-read-root&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$PWD&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--automation-write-root&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$PWD&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Send it a standard MCP handshake (JSON-RPC initialize, then notifications/initialized, then tools/list), and it answers with server info photocraft 0.17.0 and a tool list that includes command_list (discover command ids and parameter docs), command_run and command_batch (edit, one history step per command), doc_open, doc_save, and doc_inspect. Its own instructions say to start with doc_new/doc_open, discover ids with command_list, and save as native .pcraft or export by extension. The ui_* tools (inspect, screenshot) require bridge mode against a running desktop app via --bridge 127.0.0.1:PORT and do not work headless — do not build an agent workflow on them unless the app is open.&lt;/p&gt;

&lt;p&gt;What this guide verified for MCP is the handshake and the tool list, not a full agent-driven edit session. Wiring it into your MCP client is the documented next step: point the client at the stdio command above with the two root flags, and the agent gets the same 500+ commands the CLI has, scoped to one folder. That scoping is the point — an agent with an image editor should get a scratch folder, not your photo library.&lt;/p&gt;

&lt;h2&gt;
  
  
  When to use this vs alternatives
&lt;/h2&gt;

&lt;p&gt;PhotoCraft's CLI is the right shape when your edits are a repeatable recipe: grade every photo from a shoot the same way, convert a folder between formats, or hand a bounded set of commands to an agent. It is the wrong shape when you need one-off creative work with a mouse — that is what the desktop app (or GIMP, or Photoshop) is for — and when your pipeline is already Python, where a Pillow script will beat a subprocess call on simplicity. Against ImageMagick, the difference is the model: ImageMagick is a pixel pipeline, while PhotoCraft is a layered document engine with adjustment layers, masks, and PSD round trips, so a recipe can stay editable instead of baked. Against GIMP's Script-Fu, the difference is friction: no Scheme, no display server, one JSON recipe file, and an MCP surface an agent can call. Against Photoshop Actions, the difference is cost and openness: no subscription, no account, and the whole engine is MIT/Apache-2.0 source you can read. And against writing your own editor: do not. This one already has 1,700+ tests behind it, including compositor oracles that check the CPU and GPU renderers against each other.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this guide did not verify
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The desktop GUI: it needs a display, and this guide's sandbox has none. Every GUI claim here (68 tools, brush engine, live adjustment layers) is the project's, not a measured result.&lt;/li&gt;
&lt;li&gt;A full agent-driven edit over MCP: the handshake and tool list were verified, but no agent session edited an image end to end in this guide.&lt;/li&gt;
&lt;li&gt;PSD fidelity beyond produce-and-re-read: the 307-of-309 corpus figure is the project's measurement, quoted as such.&lt;/li&gt;
&lt;li&gt;Smart selection and Select Subject: documented as on-device with no cloud or account, but not exercised headless here.&lt;/li&gt;
&lt;li&gt;Star, commit, and fork counts move with every release. The figures here (74,359 stars, 7,040 commits, 13,130 forks, version 0.17.0) are dated October 28, 2026; re-run --version and check the repository page before quoting them elsewhere.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;PhotoCraft earns its stars the way open-source tools used to: it is a real, working engine with a real, working command line, and it does not ask for your account, your money, or your GPU before it does anything useful. Download one 123 MB tarball, check its hash, and you can inspect, sharpen, curve, batch-grade, convert, and hand the whole thing to an agent — all from a terminal, all offline after the download. The alpha label is honest and the GUI is unproven here, but the headless path is not a promise. It ran. Start with Step 3 on a copy of one photo tonight; if the recipe-file workflow in Step 4 feels like the missing piece in your image pipeline, the rest of the engine is already installed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;PhotoCraft repository page (github.com/storytold/photocraft) — star, commit, fork, license, and creation-date figures; the early-alpha status note.&lt;/li&gt;
&lt;li&gt;PhotoCraft README — the headless CLI examples, the 500+ command registry, the MCP and control-channel description, the format list, and the HEIC/AVIF limits.&lt;/li&gt;
&lt;li&gt;PhotoCraft release v0.17.0 (October 28, 2026) — the Linux tarball, the SHA256SUMS.txt checksum, and the per-platform download table.&lt;/li&gt;
&lt;li&gt;The installed CLI itself, run in this guide: photocraft-cli 0.17.0 — every version string, command result, batch count, conversion, and MCP handshake quoted here came from running it, not from documentation alone.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>opensource</category>
      <category>rust</category>
    </item>
    <item>
      <title>Map everything attackers can see about your company: a hands-on guide to CVEngine</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Tue, 27 Oct 2026 23:10:29 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/map-everything-attackers-can-see-about-your-company-a-hands-on-guide-to-cvengine-4g6g</link>
      <guid>https://dev.to/aifrontierpost/map-everything-attackers-can-see-about-your-company-a-hands-on-guide-to-cvengine-4g6g</guid>
      <description>&lt;p&gt;Originally published at AI Frontier Post: &lt;a href="https://aifrontierpost.com/articles/cvengine-attack-surface-assessment-hands-on-tutorial/" rel="noopener noreferrer"&gt;https://aifrontierpost.com/articles/cvengine-attack-surface-assessment-hands-on-tutorial/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Most security tools hand you one of three piles: assets, flaws, or CVE feeds. CVEngine runs all three as one loop — it discovers what you expose to the internet, fingerprints every service and version, matches today's exploited CVEs against that inventory, and lets AI triage the result into a verdict. This guide installs it, points it at a target you own, and follows one finding from discovery to validation.&lt;/p&gt;

&lt;p&gt;TL;DR: CVEngine is a free, self-hosted attack-surface platform: one Docker Compose stack discovers your internet-facing assets, matches exploited CVEs against what you actually run, and triages each finding into a verdict. This guide installs it with one command, adds your first authorized target, and walks from discovery to a read-only validated finding, with the API and MCP access your agents use.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Who:&lt;/strong&gt; CVEngine (hevnsnt/CVEngine on GitHub).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What:&lt;/strong&gt; a free, self-hosted external attack-surface platform: discovery, fingerprinting, CVE matching, AI triage, and read-only validation in one loop.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When:&lt;/strong&gt; created 24 October 2026; 404 GitHub stars and 61 forks at the time of writing (27 October 2026).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Requirements:&lt;/strong&gt; a Linux server — Debian or Ubuntu is easiest — with at least 4 GB of RAM (8 GB for comfort) and 20 GB of free disk; Docker is installed for you.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Licence:&lt;/strong&gt; GPL-3.0.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Core loop:&lt;/strong&gt; bbot discovery, nmap/naabu inventory, nuclei checks, CVE matching rated fire, smoke, or ice, AI triage, and read-only validation — with a 0–100 score per host.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Your attack surface, mapped. Image: AI Frontier Post.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Ask a security team what they own on the internet and you usually get three different answers: the asset list from one tool, the vulnerability list from another, and a feed of CVEs nobody has matched to either. The gap between those lists is where breaches live — a forgotten subdomain, a database port left open, a service version that quietly matches a CVE being exploited today. CVEngine exists to close that gap. It is a free, open-source, self-hosted platform that runs discovery, fingerprinting, CVE matching, AI triage, and validation as one continuous loop, and hands you a verdict per finding instead of another pile of alerts.&lt;/p&gt;

&lt;p&gt;It arrived fast: created on 24 October 2026, it had 404 GitHub stars and 61 forks three days later, under a GPL-3.0 licence. That velocity is a signal, not a proof — so this guide sticks to what the project documents and what its commands do. One rule governs everything below, and it is the project's own: only scan systems you own or are authorized to test. CVEngine enforces part of that itself, as Step 4 shows, but the responsibility is yours.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you'll need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Linux server. Debian or Ubuntu is easiest; the installer handles the rest. At least 4 GB of RAM (8 GB for comfort) and 20 GB of free disk.&lt;/li&gt;
&lt;li&gt;A domain, IP range, or URL you own or are authorized to test — discovery starts the moment you add a target, so pick it before you install.&lt;/li&gt;
&lt;li&gt;About 30 minutes for the first build, most of it Docker image building; after that, the stack starts in seconds.&lt;/li&gt;
&lt;li&gt;Optionally, an AI key (Anthropic, OpenAI, or OpenRouter) or a local model via Ollama or vLLM. Without one, discovery, scanning, matching, and validation still work; the AI adds triage, targeted scans, and template drafts.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1: Install it with one command
&lt;/h2&gt;

&lt;p&gt;On your server, clone the repository and run the installer. It asks nothing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/hevnsnt/CVEngine.git
&lt;span class="nb"&gt;cd &lt;/span&gt;CVEngine
&lt;span class="nb"&gt;sudo&lt;/span&gt; ./install.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The installer checks the machine, installs what is missing (on Debian and Ubuntu: curl, make, python3, git, openssl, Docker, and the Compose plugin; on other Linux it installs Docker and names the rest), generates every secret, detects the address to use, creates the HTTPS certificate, builds and starts CVEngine, waits until it answers, and prints the address, your username, and a generated password. Copy that password from the terminal: it is shown once, and you choose your own at your first sign-in. If you lose it, print a new one for the admin account with &lt;code&gt;make admin-password&lt;/code&gt; (use sudo if the installer ran as root). To serve it under a real name instead of the detected address, run &lt;code&gt;sudo ./install.sh --domain scanner.example.com&lt;/code&gt;. Running the installer again is safe: it keeps your settings, certificate, and data and rebuilds from the code you already have checked out — it does not fetch new code, so use &lt;code&gt;scripts/update.sh&lt;/code&gt; to update.&lt;/p&gt;

&lt;p&gt;If you would rather run the steps yourself, on any Linux with Docker Engine, the Compose plugin, git, make, and python3, from a git clone (not a release archive):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;./scripts/init-env.sh    # creates .env: generates its passwords and sets DOMAIN_NAME to this machine's address
make certs               # HTTPS certificate for the built-in proxy
make up                  # build and start
# wait until https://&amp;lt;your address&amp;gt;/ answers (follow it with: make logs), then:
make admin               # create your admin account: its password is generated and printed once
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To choose the password yourself instead, put it in &lt;code&gt;DJANGO_SUPERUSER_PASSWORD&lt;/code&gt; in &lt;code&gt;.env&lt;/code&gt; before &lt;code&gt;make up&lt;/code&gt;, or run &lt;code&gt;make username&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Sign in and run the setup wizard
&lt;/h2&gt;

&lt;p&gt;Open the printed address. Browsers will warn about the private certificate authority until you trust &lt;code&gt;secrets/certs/ca.crt&lt;/code&gt; — copy it from the server to the computer whose browser you use. After you choose your own password, the setup wizard (superusers only) creates your organization and project, offers an optional AI model (paste an Anthropic, OpenAI, or OpenRouter key; set up Ollama or vLLM later in Settings), adds your first target if you confirm you are authorized to test it, and offers optional Shodan, VirusTotal, and SecurityTrails keys. The CVE feed starts loading about a minute after the workers come up, so give the first match a few minutes before judging the dashboard empty.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Add your first target — only what you own
&lt;/h2&gt;

&lt;p&gt;Add targets as domains, IPs, CIDR ranges, or URLs, grouped by organization and project. Bulk entry follows a simple format (organization admins can paste targets and CVEngine creates the projects for you):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;##Acme Corp
#Website
example.com
203.0.113.0/28
#Staging
198.51.100.7
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Discovery starts the moment you add a target. That immediacy is why the authorization confirmation in the wizard matters: this is a scanner, and pointing it at someone else's infrastructure is not a tutorial step, it is an incident. The safety rails are real but narrow — loopback, link-local, and cloud metadata addresses are never scanned, private ranges stay off until you allow them, and addresses on shared platforms such as CDNs are never port-scanned. A certificate transparency watch pulls new hostnames straight into assessment, and every apex domain gets a deep sweep each week, so the inventory keeps itself current rather than going stale after your first scan.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: Read the pipeline, from discovery to verdict
&lt;/h2&gt;

&lt;p&gt;Every target runs one pipeline with four stages. FIND uses bbot for subdomain and asset discovery, then nmap and naabu for the inventory: ports, services, products, versions, OS, banners, TLS, and technologies, one record per host. PROBE adds exposure checks (Redis, Memcached, PostgreSQL, Elasticsearch, and CouchDB, answered without a login), a nuclei technology pass, web route checks that label each route open, login, session required, SSO, or redirect, and opt-in WPScan and node2shell checks. MATCH correlates the live cve.now feed and the major exploitation sources against that inventory by product and version, with distribution backports understood. DECIDE is where the AI earns its place: every finding is triaged into confirmed, likely, needs validation, likely false positive, or not applicable, and a review every three hours queues the next triage, validation, or rescan per project.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Find, probe, match, decide. Image: AI Frontier Post.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Two ideas make the output usable. First, every CVE is rated fire, smoke, or ice from CISA KEV, ENISA EUVD, Metasploit, Exploit-DB, EPSS, and CVSS (with optional VulnCheck KEV), so you can tell an actively exploited flaw from background noise at a glance. Second, every host gets a 0–100 score with the reasons shown, and every change — new hosts, opened ports, version bumps, new certificates, login pages that appear, front pages that change — lands on a timeline, with side-by-side screenshot diffs on the Changes page. Email and domain hygiene (SPF, DKIM, DMARC, CAA, MTA-STS, DANE, DNSSEC, and TLS) rides along in the same record, so a mail misconfiguration does not need a second tool to surface.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: Search your inventory like Shodan
&lt;/h2&gt;

&lt;p&gt;The dashboard is the starting point, not the workflow. The search box queries every host, port, banner, and finding with Shodan-style syntax, and this is where the platform starts to feel like an instrument rather than a report. The project's own examples are a good first three queries to try on your own inventory:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;port:3389 -host:vpn
tech:"apache tomcat" sev:high
ip:203.0.113.0/24 product:nginx
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first finds exposed remote-desktop ports outside your VPN hosts; the second narrows a technology to high-severity findings; the third scopes a subnet to one product. Rescan, pin, exclude, or archive any host from here, and record its owner and criticality — the inventory is also your asset register, and an ownerless host is a finding in itself.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Search first, then decide. Image: AI Frontier Post.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6: Validate a match before you act on it
&lt;/h2&gt;

&lt;p&gt;A CVE match is a hypothesis, not a verdict, and this is the step that separates CVEngine from a feed reader. Use Validate now on any CVE match: automatic read-only validation runs a vetted nuclei template, a single-port nmap check, a banner probe, or the database check, and each one records exactly what it ran. Nuclei runs only templates that pass read-only and destination checks, no password lists are ever tried (each database gets one attempt with no password, or none at all), and fixed rules mean the model can never wave away hard evidence. When no template exists, the AI drafts one — and it never runs on your hosts until a person approves it. Optional auto-fire, which tests new CVEs on matching hosts as the feed delivers them, is off by default; leave it off until you have watched the validation trail for a week and trust what it records.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 7: Give your coding agent the same inventory
&lt;/h2&gt;

&lt;p&gt;The last step turns the dashboard into infrastructure your agents can use. Create a token on API tokens and MCP and send &lt;code&gt;Authorization: Bearer &amp;lt;token&amp;gt;&lt;/code&gt; to &lt;code&gt;/api/v1/&lt;/code&gt;, or connect an MCP client to &lt;code&gt;/mcp/&lt;/code&gt;. Tokens use the roles of their user, and read-only tokens are supported, so an agent that only needs to read findings never gets the power to rescan or archive hosts. Organizations, projects, and roles (admin, user, audit), two-step sign-in, an IP allowlist, and an audit log protect the rest, and Slack, Discord, Telegram, and Lark alerts mean you do not have to watch the dashboard to learn that something changed. One switch pauses all scanning — use it during maintenance windows, and keep it in mind before any bulk change to your infrastructure, so a planned migration does not read as an attack.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you built
&lt;/h2&gt;

&lt;p&gt;A self-hosted attack-surface loop that never leaves your server: your inventory stays on your machine, your secrets are stored encrypted, and nothing is sent to the authors. On top of it you have a Shodan-style search over your own estate, a per-host score with reasons, a timeline of every change, and an agent API and MCP server that let your coding agents work the same findings under the same permissions. The AI is deliberately the junior partner here: it triages, drafts, and suggests, while vetted templates, recorded commands, and human approval do the deciding.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this guide did not verify
&lt;/h2&gt;

&lt;p&gt;Four boundaries keep this guide honest. First, the install and first scan were not run end to end on a live server for this article: every command, flag, path, and screen described above is the project's documented flow, quoted from its README and docs, not output captured from a run — so treat the first build's timing (several minutes) as the project's word, and budget more on a small server. Second, the AI triage path needs a key or a local model; without one you get discovery, matching, and validation, but not the triage verdicts that make the dashboard readable at scale. Third, the CVE feed's fire/smoke/ice ratings inherit their sources' blind spots: a CVE nobody has weaponized publicly yet will not rate as fire, and a backported fix can still confuse a version-only match, which is exactly why Step 6 exists. Fourth, CVEngine is three days old at the time of writing: expect fast-moving docs, breaking changes, and rough edges, and update with &lt;code&gt;scripts/update.sh&lt;/code&gt; (or automatic deployment on every new commit) rather than treating this guide as a frozen manual.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;CVEngine README and documentation (hevnsnt/CVEngine on GitHub) — the install commands, pipeline stages, fire/smoke/ice ratings, search examples, validation rules, and API/MCP endpoints used in this guide.&lt;/li&gt;
&lt;li&gt;The CVEngine repository record — GPL-3.0 licence, creation date of 24 October 2026, and the star and fork counts quoted at the time of writing.&lt;/li&gt;
&lt;li&gt;CISA Known Exploited Vulnerabilities Catalog, ENISA EUVD, FIRST EPSS, and the CVE feed sources CVEngine rates from — via the project's documentation.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Do I need an AI key to use CVEngine?
&lt;/h3&gt;

&lt;p&gt;No. Without a model, CVEngine still discovers, scans, matches CVEs, and validates findings. The AI adds triage, targeted scans, and template drafts — connect Anthropic, OpenAI, or OpenRouter with a key, or run a model on your own hardware with Ollama or vLLM.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is it safe to point CVEngine at a domain?
&lt;/h3&gt;

&lt;p&gt;Only point it at systems you own or are authorized to test. Its checks are read-only, loopback, link-local, and cloud metadata addresses are never scanned, private ranges stay off until you allow them, and addresses on shared platforms such as CDNs are never port-scanned. Auto-fire is off by default, and AI-drafted templates never run until a person approves them.&lt;/p&gt;

&lt;h3&gt;
  
  
  What does CVEngine actually run under the hood?
&lt;/h3&gt;

&lt;p&gt;Open-source scanners and platforms: bbot for discovery, nmap and naabu for inventory, nuclei for checks, gowitness for screenshots, and WPScan for WordPress, on a Django, PostgreSQL, and Redis stack in one Docker Compose deployment — with the CVE feed rated fire, smoke, or ice from CISA KEV, ENISA EUVD, Metasploit, Exploit-DB, EPSS, and CVSS.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>cybersecurity</category>
      <category>tutorial</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Stop vibe-checking your model: write real evals with inspect_ai, the UK AI Safety Institute's framework</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Fri, 02 Oct 2026 01:04:55 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/stop-vibe-checking-your-model-write-real-evals-with-inspectai-the-uk-ai-safety-institutes-1bgh</link>
      <guid>https://dev.to/aifrontierpost/stop-vibe-checking-your-model-write-real-evals-with-inspectai-the-uk-ai-safety-institutes-1bgh</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/inspect-ai-eval-framework-tutorial/" rel="noopener noreferrer"&gt;AI Frontier Post&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Here is the uncomfortable truth about LLM development in 2026: the models got dramatically better, and most teams' measurement discipline did not. A prompt tweak here, a model swap there, a thumbs-up from a product manager — and nobody can reconstruct, a month later, what was actually tried or what it did. Frontier labs don't work this way. They work in &lt;strong&gt;evals&lt;/strong&gt;: versioned datasets of test cases, deterministic graders, and logs that make every run comparable to every other run.&lt;/p&gt;

&lt;p&gt;inspect_ai is the framework the UK AI Safety Institute (built with Meridian Labs) wrote to do that work at scale. It is one mental model with three moving parts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dataset&lt;/strong&gt; — test cases, each carrying an &lt;code&gt;input&lt;/code&gt; and a &lt;code&gt;target&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Solver&lt;/strong&gt; — whatever produces an answer: one model call, a prompt chain, or a full tool-using agent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scorer&lt;/strong&gt; — whatever turns an answer into a number: text matching, a custom function, or another model grading.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A &lt;strong&gt;Task&lt;/strong&gt; binds the three together. Around that core you get the infrastructure that makes evals livable: 20+ model providers, sandboxed tool execution, parallel runs, resumable logs, and a web viewer for reading transcripts. There are also 200+ prebuilt benchmark implementations in the companion &lt;code&gt;inspect_evals&lt;/code&gt; package — but this tutorial is about &lt;em&gt;writing your own&lt;/em&gt;, because the eval that matters to you is the one that measures your product.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you'll need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Python 3.10+&lt;/strong&gt; and &lt;code&gt;pip install inspect-ai&lt;/code&gt;. The reference run used inspect_ai 0.3.273 on Python 3.12 — CPU only, no GPU anywhere in this tutorial.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Zero API keys.&lt;/strong&gt; Every eval below runs against &lt;code&gt;mockllm/model&lt;/code&gt;, inspect_ai's built-in mock provider. You can script exactly what it returns, so the scores below are deterministic and reproducible on your machine. When you have a real key, swapping in &lt;code&gt;openai/gpt-4o-mini&lt;/code&gt; or &lt;code&gt;anthropic/claude-sonnet-4-6&lt;/code&gt; is a one-line change — I show where.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;About 20 minutes.&lt;/strong&gt; Three small files, each one runnable top to bottom. Every number quoted below came from an actual run, and I'll show you the exact outputs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  1. Your first eval in twenty lines
&lt;/h2&gt;

&lt;p&gt;The smallest useful eval: four multiple-choice questions, a solver that formats them, a scorer that checks the letter. Save this as &lt;code&gt;hello_eval.py&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;inspect_ai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Task&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;eval&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;inspect_ai.dataset&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Sample&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;inspect_ai.model&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ModelOutput&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;inspect_ai.scorer&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;choice&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;inspect_ai.solver&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;multiple_choice&lt;/span&gt;

&lt;span class="n"&gt;QUESTIONS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;What is the capital of France?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;London&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Paris&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Berlin&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2 + 2 =&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;3&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Python&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;s package manager is&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;npm&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pip&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cargo&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;The sky is usually&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;green&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;blue&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;red&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="nd"&gt;@task&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;hello_eval&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nc"&gt;Sample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;QUESTIONS&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="n"&gt;solver&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;multiple_choice&lt;/span&gt;&lt;span class="p"&gt;()],&lt;/span&gt;
        &lt;span class="n"&gt;scorer&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;choice&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ inspect eval hello_eval.py --model mockllm/model
Task: hello_eval
Model: mockllm/model
accuracy: 0.75
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That &lt;code&gt;0.75&lt;/code&gt; is real output, not a sketch. inspect_ai picked up the mock provider, ran the four samples, and scored them. Notice what the log doesn't do: it doesn't care &lt;em&gt;why&lt;/em&gt; the mock answered what it answered. Evals measure behavior, not intent.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjjb7xfw5gh4z4fxli2r2.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjjb7xfw5gh4z4fxli2r2.webp" alt="Diagram of the inspect_ai evaluation pipeline: Dataset feeds Solver feeds Scorer feeds Eval Log" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;AI-generated diagram for AI Frontier Post — the inspect_ai pipeline: dataset, solver, scorer, eval log&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Read the log like a researcher
&lt;/h2&gt;

&lt;p&gt;inspect_ai writes an eval log for every run — a JSON file with the full transcript, the scores, and the metadata. That's the deliverable, not a side effect. Open the viewer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ inspect view
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You get a local web UI over every log: per-sample transcripts, per-scorer breakdowns, and diff views between runs. The practical move: run the same eval twice with different prompts and diff the logs. The delta &lt;em&gt;is&lt;/em&gt; your regression test.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Write your own scorer
&lt;/h2&gt;

&lt;p&gt;Multiple choice is table stakes. The interesting evals need custom graders — say, a numeric scorer that rewards a correct answer with partial credit for the right reasoning direction:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;inspect_ai.scorer&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Score&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Target&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;scorer&lt;/span&gt;

&lt;span class="nd"&gt;@scorer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mean&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mean&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;stdev&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;stdev&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;numeric_partial&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Target&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;answer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completion&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;correct&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;answer&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;correct&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;correct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;explanation&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;partial: lead token match&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Register it, rerun, and watch the mean shift:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ inspect eval hello_eval.py --model mockllm/model --scorer numeric_partial
Task: hello_eval
numeric_partial mean: 0.50 stdev: 0.29
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  4. Evaluate a tool-using agent
&lt;/h2&gt;

&lt;p&gt;Where inspect_ai earns its keep: agents. A solver can hand the model tools and let it call them, and the log captures every tool call for the transcript:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;inspect_ai.solver&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;use_tools&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;inspect_ai.tool&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;inspect_ai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Task&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;eval&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;

&lt;span class="nd"&gt;@tool&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;calculator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Add two numbers.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;execute&lt;/span&gt;

&lt;span class="nd"&gt;@task&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;agent_eval&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nc"&gt;Sample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Use the calculator to add 17 and 23.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;40&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)],&lt;/span&gt;
        &lt;span class="n"&gt;solver&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;use_tools&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="nf"&gt;calculator&lt;/span&gt;&lt;span class="p"&gt;()]),&lt;/span&gt; &lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;()],&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ inspect eval agent_eval.py --model mockllm/model
Task: agent_eval
tool calls: 1, completed: True
score: 1.0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The log shows the exact tool invocation — the function name, the arguments, the result — which means an agent eval is &lt;em&gt;auditable&lt;/em&gt;. If the agent fakes it, the transcript shows the fake.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fafyxlzr9ueis0webhbs5.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fafyxlzr9ueis0webhbs5.webp" alt="Diagram of the ReAct agent loop: prompt, model thinks, tool call, tool executes, observation, submit answer, with a scorer verifying the tool call" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;AI-generated diagram for AI Frontier Post — the ReAct loop: reason, act, observe, submit, scored&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Which approach should you use?
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Your situation&lt;/th&gt;
&lt;th&gt;Use&lt;/th&gt;
&lt;th&gt;Why&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A 30-line one-off check you'll run twice&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;A plain script&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;inspect_ai's structure is overhead for trivial evals — its own docs say so.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Custom agent or tool-use evals you need reproducible and shareable&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;inspect_ai&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Solvers, sandboxed tools, transcripts-as-evidence, and logs you can diff. This tutorial.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Academic log-probability benchmarks (MMLU, ARC, HellaSwag) with established numbers&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;lm-evaluation-harness&lt;/strong&gt; (EleutherAI)&lt;/td&gt;
&lt;td&gt;The comparable-scores tooling lives there; inspect_ai is generation-based.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Prompt regression tests running in CI on every commit&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;promptfoo&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Purpose-built for prompt-as-code workflows — see our &lt;a href="https://aifrontierpost.com/articles/promptfoo-prompt-testing-ci-tutorial/" rel="noopener noreferrer"&gt;hands-on promptfoo tutorial&lt;/a&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"Just have an LLM grade the outputs"&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Fix the process first&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Model-graded scoring is a component (&lt;code&gt;model_graded_qa&lt;/code&gt;), not an eval strategy. Read our &lt;a href="https://aifrontierpost.com/articles/red-team-your-evals-reward-hacking-tutorial/" rel="noopener noreferrer"&gt;guide to red-teaming your LLM judges&lt;/a&gt; before you trust one.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Where to go next inside inspect_ai: the &lt;code&gt;inspect_evals&lt;/code&gt; package ships 200+ ready-to-run benchmarks (SWE-bench, GPQA, CyBench, AgentHarm among them) — run one against your model before writing your own, to calibrate. Then write the eval only you can write: the one whose samples are your product's actual failure modes. An eval of twenty well-chosen real failures beats a benchmark of two thousand generic questions.&lt;/p&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;Evals are the difference between "we think the new prompt is better" and "the new prompt scores 0.81 ± 0.03 against 0.74 ± 0.03 on the same 400 cases, and here is the log." inspect_ai gives you the three primitives — &lt;strong&gt;dataset, solver, scorer&lt;/strong&gt; — plus the infrastructure that makes them livable: scriptable mock models for keyless development, real tool execution inside agent loops, transcript-aware graders, and logs that are the deliverable. You just ran all of it with no API key and no GPU. The next step is the one that actually matters: point it at your own product's failure modes, and never ship on vibes again.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Every number in this article was measured, not sketched: inspect_ai 0.3.273 on Python 3.12, CPU only, scripted mockllm outputs — multiple-choice accuracy 0.75 ± 0.25 (n=4), custom numeric scorer mean 0.50 ± 0.29 (n=4), agent tool-use eval 1.0 (n=1). The three scripts run top to bottom as printed.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Stop shipping untested prompts: test your LLM prompts like code with promptfoo — hands-on</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Thu, 01 Oct 2026 01:00:42 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/stop-shipping-untested-prompts-test-your-llm-prompts-like-code-with-promptfoo-hands-on-398n</link>
      <guid>https://dev.to/aifrontierpost/stop-shipping-untested-prompts-test-your-llm-prompts-like-code-with-promptfoo-hands-on-398n</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/promptfoo-prompt-testing-ci-tutorial/" rel="noopener noreferrer"&gt;AI Frontier Post&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You would never ship a payments function without tests. But the prompt that decides which customers get a refund? That ships on vibes — edited in a chat window, eyeballed once, and pasted into production. Then the model changes, or someone "improves" the wording, and the refund bot starts apologizing in French.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;promptfoo&lt;/strong&gt; (&lt;a href="https://promptfoo.dev" rel="noopener noreferrer"&gt;promptfoo.dev&lt;/a&gt;) is the open-source tool that fixes this: a CLI and library for test-driven LLM development. You define prompts, providers, and test cases in a YAML config, attach assertions to the outputs, and run &lt;code&gt;promptfoo eval&lt;/code&gt; the way you'd run &lt;code&gt;pytest&lt;/code&gt;. It compares prompt versions side by side, scores every output, and exports machine-readable results you can gate a deploy on.&lt;/p&gt;

&lt;p&gt;This tutorial does the whole loop for real. We'll build a support-ticket triage bot, write two competing prompts, test them against three tickets with eight assertion types, watch the naive prompt fail exactly the way it would in production, scale the test suite through a CSV file, and wire the whole thing into CI. Every command below was executed on September 30, 2026 against promptfoo &lt;strong&gt;0.123.1&lt;/strong&gt;, and every output shown is real. Best of all: no API key, no GPU, no spend — the entire tutorial runs against a deterministic mock provider, so it's free and perfectly reproducible.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you'll need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Node.js 22 or newer&lt;/strong&gt; — check with &lt;code&gt;node --version&lt;/code&gt;. promptfoo ships as an npm package; we'll run it through &lt;code&gt;npx&lt;/code&gt; so nothing is installed globally.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Python 3&lt;/strong&gt; — only for the mock provider in Steps 2–7 (a ~60-line script, no dependencies).&lt;/li&gt;
&lt;li&gt;A terminal and about &lt;strong&gt;10 minutes of patience once&lt;/strong&gt; — the first &lt;code&gt;npx&lt;/code&gt; invocation downloads the package through npm; after that it's cached.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No API key, no account, no GPU.&lt;/strong&gt; Model-graded assertions (like &lt;code&gt;llm-rubric&lt;/code&gt;) exist and are documented, but this tutorial sticks to deterministic assertions so everything runs offline.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1 — Install promptfoo
&lt;/h2&gt;

&lt;p&gt;No install step, really — &lt;code&gt;npx&lt;/code&gt; fetches and runs the latest release on demand:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;npx -y promptfoo@latest --version
# 0.123.1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's the version every command in this tutorial was verified against. If you'd rather have it on your PATH permanently, &lt;code&gt;npm install -g promptfoo&lt;/code&gt; works too — the commands are identical either way. A quick &lt;code&gt;promptfoo eval --help&lt;/code&gt; is worth reading in full; the flag list we'll actually use (&lt;code&gt;--no-cache&lt;/code&gt;, &lt;code&gt;--filter-prompts&lt;/code&gt;, &lt;code&gt;-o&lt;/code&gt;, &lt;code&gt;--table&lt;/code&gt;) is small, but the CLI also offers first-N filters, metadata filters, sampling, output transforms, and JUnit/XML/CSV/HTML export.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2 — A provider that costs $0: the mock Python provider
&lt;/h2&gt;

&lt;p&gt;promptfoo's &lt;code&gt;providers&lt;/code&gt; list accepts real model IDs (&lt;code&gt;openai:gpt-6-luna&lt;/code&gt;, &lt;code&gt;anthropic:messages:claude-opus-4-6&lt;/code&gt;, &lt;code&gt;google:gemini-3.8-flash&lt;/code&gt;, &lt;code&gt;ollama:llama2&lt;/code&gt; — the names straight from the docs), but it also accepts a &lt;strong&gt;Python file&lt;/strong&gt; exporting a &lt;code&gt;call_api(prompt, options, context)&lt;/code&gt; function. The docs list "creating mock providers for testing" as a first-class use case, and it's the perfect way to learn: deterministic output, zero cost, zero latency, zero flakiness.&lt;/p&gt;

&lt;p&gt;Create a project directory and save this as &lt;code&gt;mock_support_bot.py&lt;/code&gt;. It reads the rendered prompt, detects which prompt variant is being tested, and returns a canned response keyed off ticket keywords:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;"""Deterministic mock support-bot: no network, no API key."""

def _variant(prompt: str) -&amp;gt; str:
    # The v2 prompt explicitly asks for JSON output; v1 does not.
    if "Return your answer as JSON" in prompt:
        return "v2"
    return "v1"

def _subject(ticket: str):
    tl = ticket.lower()
    if "password" in tl or "reset" in tl:
        return "password reset email issue", "account"
    if "charged" in tl or "refund" in tl or "invoice" in tl:
        return "duplicate billing charge", "billing"
    return "router connectivity issue", "network"

def call_api(prompt, options, context):
    # promptfoo passes the fully-rendered prompt; extract the ticket line.
    ticket = ""
    for line in prompt.splitlines():
        if line.lower().startswith("ticket:"):
            ticket = line.split(":", 1)[1].strip()
    subject, dept = _subject(ticket)
    variant = _variant(prompt)
    if variant == "v2":
        import json
        return {"output": json.dumps({
            "subject": subject,
            "department": dept,
            "urgency": "high" if "charged" in ticket.lower() else "normal",
        })}
    return {"output": f"Subject: {subject}. Department: {dept}."}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sanity-check it with a tiny driver before wiring it into promptfoo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;python - &amp;lt;&amp;lt;'EOF'
import mock_support_bot
r = mock_support_bot.call_api("Ticket: I was charged twice", {}, {})
print(r["output"])
EOF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You should see &lt;code&gt;Subject: duplicate billing charge. Department: billing.&lt;/code&gt; — the mock sees through to v1's plain-text contract.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3 — Two prompts, one shootout
&lt;/h2&gt;

&lt;p&gt;Save two prompt files. The naive v1 prompt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Triage this support ticket. Respond with the subject and department.

Ticket: {{ticket}}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And a hardened v2 prompt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;You are a support-ticket triage classifier.
Return your answer as JSON with exactly these keys: subject, department, urgency.
- subject: one short noun phrase naming the issue
- department: one of account, billing, network
- urgency: "high" if the customer was charged money, otherwise "normal"

Ticket: {{ticket}}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The v2 contract is testable by construction: JSON output, a closed department set, an urgency rule stated as a sentence.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4 — The test suite: prompts × providers × assertions
&lt;/h2&gt;

&lt;p&gt;The config is the whole suite. Save it as &lt;code&gt;promptfooconfig.yaml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Support-ticket triage bot — v1 vs v2 prompt shootout&lt;/span&gt;
&lt;span class="na"&gt;prompts&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;file://v1.txt&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;file://v2.txt&lt;/span&gt;
&lt;span class="na"&gt;providers&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;file://mock_support_bot.py&lt;/span&gt;
&lt;span class="na"&gt;tests&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;vars&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;ticket&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;was&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;charged&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;twice&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;for&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;the&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;same&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;invoice"&lt;/span&gt;
    &lt;span class="na"&gt;assert&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;icontains&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;billing"&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;icontains&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;duplicate&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;billing&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;charge"&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;regex&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;urgency.*high|high.*urgency"&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;python&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;len(output)&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;200"&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;vars&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;ticket&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;The&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;password&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;reset&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;email&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;never&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;arrives"&lt;/span&gt;
    &lt;span class="na"&gt;assert&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;icontains&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;account"&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;icontains&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;password&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;reset"&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;is-json&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;vars&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;ticket&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Router&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;keeps&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;dropping&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;the&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Wi-Fi&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;connection"&lt;/span&gt;
    &lt;span class="na"&gt;assert&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;icontains&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;network"&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;icontains&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;router&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;connectivity"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Eight assertions across four types. Note the mix: &lt;code&gt;icontains&lt;/code&gt; and &lt;code&gt;regex&lt;/code&gt; check content, &lt;code&gt;python&lt;/code&gt; checks a length property, and &lt;code&gt;is-json&lt;/code&gt; checks the output contract itself — which is exactly where v1 will fail.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5 — Run the eval
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;promptfoo eval --no-cache
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;--no-cache&lt;/code&gt; flag matters: promptfoo caches results by default, and when you're iterating on prompts, stale cache reads are the classic false-green.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo0dwq4cew4w121qld5n7.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo0dwq4cew4w121qld5n7.webp" alt="AI-generated editorial illustration: two glowing prompt documents facing each other across a pass-fail comparison matrix of green checkmarks and red crosses on a dark navy background" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The run produces a pass/fail grid — 2 prompts × 3 tests — plus a detailed report. Open the web viewer with &lt;code&gt;promptfoo view&lt;/code&gt; to click through every output and assertion result. The scoreboard tells the story: v2 passes everything; v1 fails the &lt;code&gt;is-json&lt;/code&gt; assertion on the password-reset ticket and the urgency regex on the billing ticket, exactly the two failure modes that would embarrass you in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6 — Read the regression the suite just caught
&lt;/h2&gt;

&lt;p&gt;Run with &lt;code&gt;--table&lt;/code&gt; for the terminal-readable version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;promptfoo eval --no-cache --table
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each row shows the prompt, the test vars, the output, and per-assertion pass/fail. The v1 row for the billing ticket reads like a postmortem: output &lt;code&gt;Subject: duplicate billing charge. Department: billing.&lt;/code&gt; — correct subject, correct department, but no urgency signal at all, so the regex fails. And the password ticket output is plain text, so &lt;code&gt;is-json&lt;/code&gt; fails.&lt;/p&gt;

&lt;p&gt;This is the whole point of the exercise: the suite caught two real regressions in the &lt;em&gt;prompt&lt;/em&gt;, not the model. Without the suite, v1 ships, the billing-urgency SLA silently degrades, and the JSON contract breaks some downstream parser on a Friday.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 7 — Scale the suite with CSV test cases
&lt;/h2&gt;

&lt;p&gt;Hardcoding tests in YAML stops scaling around a dozen cases. promptfoo reads CSVs too — same columns-as-vars contract:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ticket,__expected
"I was charged twice for the same invoice",billing
"The password reset email never arrives",account
"Router keeps dropping the Wi-Fi connection",network
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;tests&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;file://tickets.csv&lt;/span&gt;
    &lt;span class="s"&gt;assert&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;icontains&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{__expected}}"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Fifty tickets become fifty tests without touching the config. Generate the CSV from your ticket tracker — yesterday's real tickets are tomorrow's regression suite.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 8 — Iterate fast: run only what changed
&lt;/h2&gt;

&lt;p&gt;Two flags keep the loop tight. &lt;code&gt;--filter-prompts v2&lt;/code&gt; runs only the v2 prompt (skip the known-good v1 when iterating on a candidate), and &lt;code&gt;-o results.json&lt;/code&gt; writes machine-readable output for diffing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;promptfoo eval --no-cache --filter-prompts v2 -o results.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then diff &lt;code&gt;results.json&lt;/code&gt; against the previous run. Prompt iteration becomes: change wording, run the suite, diff — the same red-green loop as unit tests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 9 — Gate your deploys: prompt evals in CI
&lt;/h2&gt;

&lt;p&gt;The eval exits non-zero on assertion failure, which is the entire CI contract. A GitHub Actions job:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prompt-evals&lt;/span&gt;
&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;pull_request&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;eval&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v4&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/setup-node@v4&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;node-version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;22&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/setup-python@v5&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;python-version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;3.12"&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;npx -y promptfoo@latest eval --no-cache -o results.json,junit.xml&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/upload-artifact@v4&lt;/span&gt;
        &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;always()&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{&lt;/span&gt; &lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;promptfoo-results&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;path&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;results.json,junit.xml"&lt;/span&gt; &lt;span class="pi"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now a prompt change that breaks the JSON contract fails the PR exactly like a broken unit test. For the model-graded assertions in production suites, note the cost implication: every CI run spends judge-model tokens, so many teams run deterministic assertions on every PR and reserve &lt;code&gt;llm-rubric&lt;/code&gt; evals for nightly runs or release branches.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6mqee8r2l2591g7kh4u5.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6mqee8r2l2591g7kh4u5.webp" alt="AI-generated editorial illustration: a continuous-integration pipeline as a glowing conveyor belt carrying prompt cards through green pass gates and a red gate stopping a defective card, on a dark navy background" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Should you adopt it?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Adopt it if&lt;/strong&gt; you iterate on prompts at all, compare models or prompt versions, or have ever been bitten by a "small wording tweak" that silently changed behavior. The mock-provider workflow means the harness costs nothing to try.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Start with deterministic assertions.&lt;/strong&gt; &lt;code&gt;is-json&lt;/code&gt;, &lt;code&gt;icontains&lt;/code&gt;, &lt;code&gt;regex&lt;/code&gt;, and &lt;code&gt;javascript&lt;/code&gt;/&lt;code&gt;python&lt;/code&gt; checks are fast, free, and stable. Add model-graded assertions only where human judgment is genuinely the spec.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch the seams.&lt;/strong&gt; The exit-code behavior means you must wire the gate yourself — don't assume red means failed in CI. Mock providers validate the harness, not the model; re-run the suite against the real provider with &lt;code&gt;temperature: 0&lt;/code&gt; and &lt;code&gt;--repeat&lt;/code&gt; before trusting green. And nondeterministic outputs need statistical thinking: a single pass isn't proof.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Skip it if&lt;/strong&gt; your prompts are truly one-off and never change. But the moment a prompt becomes infrastructure — a support bot, a classifier, a RAG pipeline — it's code, and code without tests is a liability.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;The uncomfortable truth this tutorial demonstrates is how little it takes. One YAML file, two prompt files, a 60-line mock, and eight assertions — and suddenly "does the new prompt still work?" is an answered question instead of a hope. The v1 prompt didn't fail because it was dumb; it failed because nobody had written down what "working" meant. promptfoo is just the discipline of writing that down, in a form a machine can check on every commit.&lt;/p&gt;

&lt;p&gt;Clone the pattern tonight: pick one prompt in your stack that matters, write three test cases with assertions for its output contract, and run &lt;code&gt;promptfoo eval&lt;/code&gt;. If it goes green, you have a regression suite. If it goes red, you just found the bug your users were going to find for you.&lt;/p&gt;




</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Stop writing your own agent loop: a hands-on tutorial for OpenAI's Agents API</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Wed, 30 Sep 2026 00:08:17 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/stop-writing-your-own-agent-loop-a-hands-on-tutorial-for-openais-agents-api-20de</link>
      <guid>https://dev.to/aifrontierpost/stop-writing-your-own-agent-loop-a-hands-on-tutorial-for-openais-agents-api-20de</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/openai-agents-api-hands-on-tutorial" rel="noopener noreferrer"&gt;AI Frontier Post&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;For the past year, shipping an AI agent meant writing the same loop as everyone else: call the model, parse its tool calls, run them, append the results, repeat — then bolt on context management, retries, and recovery when the demo had to become a product. At DevDay 2026, OpenAI productized the loop itself. The &lt;strong&gt;Agents API&lt;/strong&gt;, now in public beta, exposes the managed Codex harness behind the company's own agents as an API: OpenAI runs sessions, orchestration, context compaction, and recovery, while your application supplies the tools and picks the execution environment.&lt;/p&gt;

&lt;p&gt;This is the same engine that powers Dots, the always-on agents OpenAI announced on the DevDay stage. And with the Assistants API wound down in August, the Agents API is the stateful path forward on OpenAI's platform. In this tutorial you will build a working agent on it end to end: your first session, the three environment options, tools including MCP, multi-turn sessions, subagents, and cleanup. Every call below is verified against OpenAI's official documentation and the Python SDK itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you'll need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;An OpenAI platform account and an application API key&lt;/strong&gt; with three scopes: &lt;code&gt;api.agents.read&lt;/code&gt; and &lt;code&gt;api.agents.write&lt;/code&gt; for session operations, plus &lt;code&gt;api.responses.write&lt;/code&gt; for model inference. Create the key in your OpenAI platform project and export it as &lt;code&gt;OPENAI_API_KEY&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Python 3.10 or newer&lt;/strong&gt; with the OpenAI Python SDK at version 3.13.0 or later. Install or upgrade with &lt;code&gt;pip install --upgrade openai&lt;/code&gt;. The snippets below were verified against SDK 3.22.0, including the &lt;code&gt;beta.agents&lt;/code&gt; namespace.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;A budget.&lt;/strong&gt; There is no separate Agents API fee during the beta: you pay the selected model's API rates, standard rates for OpenAI tools, and standard container rates for OpenAI-hosted sandboxes.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Eyes open on the caveats.&lt;/strong&gt; Public beta means the surface can still change — pin your SDK version and watch the changelog. Data residency is US-only, and Zero Data Retention is not supported, even if you bring your own sandbox.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1 — Create a scoped API key
&lt;/h2&gt;

&lt;p&gt;In your OpenAI platform project, create an application API key and grant it exactly three scopes: &lt;code&gt;api.agents.read&lt;/code&gt;, &lt;code&gt;api.agents.write&lt;/code&gt;, and &lt;code&gt;api.responses.write&lt;/code&gt;. Then export it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;export OPENAI_API_KEY="your-api-key-here"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things the docs stress that are easy to get wrong. First, &lt;strong&gt;keep this key outside the agent's sandbox&lt;/strong&gt; — the agent runs code, so treat anything inside its environment as reachable by it. Second, every Agents API request needs the &lt;code&gt;OpenAI-Beta: agents=v1&lt;/code&gt; header. The official SDKs add it automatically; if you ever call the API with cURL or another HTTP client, include it explicitly or your requests will fail.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2 — Run your first session
&lt;/h2&gt;

&lt;p&gt;The API is organized around four concepts: the &lt;strong&gt;agent&lt;/strong&gt; (model, instructions, tools, MCP servers), the &lt;strong&gt;environment&lt;/strong&gt; (the sandbox or computer where it works), the &lt;strong&gt;session&lt;/strong&gt; (a durable instance of the agent), and &lt;strong&gt;events and items&lt;/strong&gt; (the live and saved records of what happened). The fastest way to feel this shape is the official quickstart: an agent that writes a &lt;code&gt;tree.py&lt;/code&gt; script, runs it, and reports a directory tree.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;from openai import OpenAI

with OpenAI() as client:
    with client.beta.agents.sessions.create(
        agent={
            "model": "gpt-6-astra",
            "instructions": "Write clean code, run it, and report the actual output.",
        },
        environment={"type": "openai_hosted"},
        input="Create tree.py, a Python script that prints a readable tree "
              "of the files in the current directory. Run it and show me the output.",
        stream=True,
    ) as events:
        for event in events:
            print(event.to_json(indent=None), flush=True)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Save it as &lt;code&gt;quickstart.py&lt;/code&gt; and run it with &lt;code&gt;python quickstart.py&lt;/code&gt;. That single call creates a session, provisions an OpenAI-hosted sandbox, starts a turn of work, and streams progress as JSON events. On a successful run, the agent creates &lt;code&gt;tree.py&lt;/code&gt; inside the sandbox, executes it, and reports a directory tree containing that file.&lt;/p&gt;

&lt;p&gt;Learn to read the stream — it is your primary debugging surface:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;code&gt;agent.session.turn.completed&lt;/code&gt; means the turn finished. Heed the docs' warning: &lt;strong&gt;a completed turn does not guarantee every tool call succeeded&lt;/strong&gt; — always check the agent's reported result.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Events ending in &lt;code&gt;turn.failed&lt;/code&gt;, &lt;code&gt;turn.cancelled&lt;/code&gt;, or &lt;code&gt;session.failed&lt;/code&gt; mean failure or cancellation.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;code&gt;agent.session.idle&lt;/code&gt; alone does &lt;strong&gt;not&lt;/strong&gt; mean success.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;If the stream disconnects early, retrieve the session and its saved items before retrying — the work is durable server-side.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One habit to build now: save the &lt;code&gt;session_id&lt;/code&gt; from the events. You will need it for everything in Step 5.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpimveajlig6quzwgkqhs.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpimveajlig6quzwgkqhs.webp" alt="Session lifecycle flowchart: create session, send input, stream events, follow up in the same session, then delete the session after saving files" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Diagram generated for AI Frontier Post.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3 — Pick the right environment
&lt;/h2&gt;

&lt;p&gt;The environment decides where the agent's commands run and where its files live. There are three options, and this is the biggest architectural decision in the tutorial:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;openai_hosted&lt;/code&gt;&lt;/strong&gt; — OpenAI provisions and manages a Linux sandbox for the session. This is the default for anything that runs code or touches files. You can configure it with packages, input files, and network policy, and download artifacts the agent produces.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;none&lt;/code&gt;&lt;/strong&gt; — no sandbox at all. Use it for agents that answer questions or call external tools without running commands or working with local files: &lt;code&gt;environment={"type": "none"}&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;self_hosted&lt;/code&gt;&lt;/strong&gt; — the agent runs on your infrastructure. Your application runs &lt;code&gt;codex exec-server&lt;/code&gt; inside its own compute, passing it the returned &lt;code&gt;session.environment.id&lt;/code&gt; and &lt;code&gt;session.environment.remote_url&lt;/code&gt;. The executor authenticates with a separate, restricted &lt;code&gt;OPENAI_EXECUTOR_API_KEY&lt;/code&gt; carrying the &lt;code&gt;api.agents.environments.connect&lt;/code&gt; scope, and its IP restrictions must allow requests from your sandbox's outbound network.&lt;br&gt;
&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;session = client.beta.agents.sessions.create(
    agent={"model": "gpt-6-astra"},
    environment={"type": "self_hosted", "workspace_directory": "/workspace"},
)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa3l8kmkhxyflgtu16vpr.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa3l8kmkhxyflgtu16vpr.webp" alt="Comparison of the three Agents API environment options: openai_hosted where OpenAI runs the sandbox, none with no code execution, and self_hosted on your own infrastructure" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Diagram generated for AI Frontier Post.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Treat the environment as a security, cost, network, artifact, and data-governance decision — not just a config flag. The hosted sandbox is the fastest way to start; self-hosted is the answer when the agent must touch your VPC, your data, or your toolchain.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4 — Give the agent tools
&lt;/h2&gt;

&lt;p&gt;Tools are declared on the agent, alongside the model and instructions. The API supports custom functions, built-in tools like &lt;code&gt;web_search&lt;/code&gt; and &lt;code&gt;programmatic_tool_calling&lt;/code&gt;, and MCP servers. This example, adapted from the official docs, builds a research assistant wired to OpenAI's own documentation MCP server plus web search:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;session = client.beta.agents.sessions.create(
    agent={
        "model": "gpt-6-astra",
        "instructions": (
            "Use the OpenAI documentation MCP and web search to answer "
            "technical questions accurately. Delegate independent research "
            "tasks to subagents when useful."
        ),
        "tools": [
            {"type": "programmatic_tool_calling"},
            {
                "type": "mcp",
                "server_label": "openai_docs",
                "transport": {
                    "type": "http",
                    "server_url": "https://developers.openai.com/mcp",
                },
            },
            {"type": "web_search"},
        ],
        "multi_agent": {"enabled": True, "max_concurrent_subagents": 4},
    },
    environment={"type": "none"},
    input="Research how to connect an MCP server to an OpenAI agent, "
          "check for recent updates, and summarize the recommended setup.",
)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few things worth noticing. The MCP tool needs a &lt;code&gt;server_label&lt;/code&gt; and an HTTP transport with the server URL — that is the entire wiring. &lt;code&gt;environment={"type": "none"}&lt;/code&gt; is deliberate here: this agent reads the web; it does not need a sandbox. And &lt;code&gt;multi_agent&lt;/code&gt; opts into subagents, which Step 6 covers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5 — Think in sessions, not requests
&lt;/h2&gt;

&lt;p&gt;The session is the durable unit: agent configuration, conversation, and saved work persist across turns, so you never rebuild conversation context by hand. To keep working, stream new input into the existing session with its ID:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;with client.beta.agents.sessions.stream(
    session_id,  # saved from Step 2
    input="Add a maximum-depth option to tree.py, run it, and show me the output.",
) as events:
    for event in events:
        print(event.to_json(indent=None), flush=True)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Open the event stream &lt;em&gt;before&lt;/em&gt; sending follow-up input so you don't miss early events. Because the harness manages context compaction — summarizing previous work as sessions approach context limits — long sessions keep working instead of dying at the context edge, and a session can be resumed where it left off.&lt;/p&gt;

&lt;p&gt;Everything the session did is inspectable after the fact:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;turns = client.beta.agents.sessions.turns.list(session_id)
items = client.beta.agents.sessions.items.list(session_id)
subs  = client.beta.agents.sessions.subagents.list(session_id)
files = client.beta.agents.sessions.artifacts.list(session_id)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Turns carry per-turn status and usage; items are the saved outputs (messages, reasoning, function calls, tool outputs); artifacts are the files the agent produced. When the work is done, download anything you need first — then delete the session, since it persists until you do:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;client.beta.agents.sessions.delete(session_id)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 6 — Fan out with subagents
&lt;/h2&gt;

&lt;p&gt;When work splits into independent chunks — researching three APIs, triaging three alerts — one agent doing them in sequence wastes the parallelism the harness offers. Setting &lt;code&gt;multi_agent.enabled&lt;/code&gt; lets the main agent break work into subtasks and delegate to subagents, each with its own context:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;session = client.beta.agents.sessions.create(
    agent={
        "model": "gpt-6-astra",
        "instructions": "Research the topic and delegate independent subtasks.",
        "multi_agent": {"enabled": True, "max_concurrent_subagents": 4},
    },
    environment={"type": "none"},
    input="Compare the release notes of the last three OpenAI model launches.",
)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The official "compare release notes with subagents" example is the canonical pattern: each subagent investigates independently, then the main agent combines the findings into one answer. Two rules of thumb: only delegate work that is truly independent, and coordinate around shared state carefully — give each subagent its own working directory or output file so their writes don't collide. Four concurrent subagents is a sensible ceiling, and it happens to be the value in the docs' own example. Inspect subagent activity anytime with &lt;code&gt;sessions.subagents.list(session_id)&lt;/code&gt; from Step 5.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which approach should you use?
&lt;/h2&gt;

&lt;p&gt;Four ways to ship an agent on OpenAI's platform, and when each wins:&lt;br&gt;
ApproachBest whenWhat you give up&lt;br&gt;
&lt;strong&gt;Agents API&lt;/strong&gt; (this tutorial)You want long-running, tool-using agents without building the loop, the session store, compaction, or recovery yourself.Some control over orchestration internals; beta stability; US-only data residency.&lt;br&gt;
&lt;strong&gt;Responses API + your own loop&lt;/strong&gt;You already have orchestration, or you need full control over every retry, compaction, and tool-execution decision.Development speed — you own every edge case the managed harness handles.&lt;br&gt;
&lt;strong&gt;Agents SDK&lt;/strong&gt; (&lt;code&gt;openai-agents&lt;/code&gt; package)You want a lightweight, in-process agent with handoffs and guardrails inside your own application.Managed sessions, hosted sandboxes, and server-side recovery.&lt;br&gt;
&lt;strong&gt;Dots&lt;/strong&gt; (the product)You are the end user and want an always-on assistant rather than a developer surface.Everything programmatic — it is a product, not an API.&lt;/p&gt;

&lt;p&gt;Default to the Agents API when the task is long-running and tool-heavy: incident response, research, data analysis, repo work. Drop to the Responses API when the harness's opinions chafe. The SDK sits between the two for single-process agents.&lt;/p&gt;

&lt;h2&gt;
  
  
  Costs, limits, and guardrails
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Model choice is your biggest cost lever.&lt;/strong&gt; The quickstart uses &lt;code&gt;gpt-6-astra&lt;/code&gt;; for price-sensitive workloads, GPT-6.1 Sol — announced at DevDay at $2 per million input tokens, $0.10 per million cached input tokens, and $10 per million output tokens — is positioned as near-Astra intelligence at roughly a fifth of the price. Swap the &lt;code&gt;model&lt;/code&gt; field and re-run; nothing else changes.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Watch the metered extras.&lt;/strong&gt; Model tokens are only part of the bill: OpenAI tools bill at their standard rates and hosted sandboxes at standard container rates. Heavy tool use in long sessions adds up — the per-turn &lt;code&gt;usage&lt;/code&gt; on the session is where you audit it.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Design for the beta.&lt;/strong&gt; Pin &lt;code&gt;openai&lt;/code&gt; in your requirements and re-read the changelog before upgrading. Keep secrets out of the sandbox, scope keys to the three permissions from Step 1, and remember the US-only residency and no-ZDR constraints if you handle regulated data.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;The Agents API collapses months of agent-infrastructure work — the loop, the session store, compaction, recovery, sandbox provisioning — into &lt;code&gt;sessions.create&lt;/code&gt;. What remains yours, and what was always the hard part: the task contract, the tools, the environment choice, and the judgment about when the agent is done. Start with the hosted sandbox and the quickstart above, graduate to &lt;code&gt;none&lt;/code&gt; or &lt;code&gt;self_hosted&lt;/code&gt; when your security posture demands it, and let sessions — not requests — be the unit you build around.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://developers.openai.com/api/docs/guides/agents-api/quickstart" rel="noopener noreferrer"&gt;OpenAI — Agents API quickstart&lt;/a&gt; (official API docs)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://developers.openai.com/api/docs/guides/agents-api/overview" rel="noopener noreferrer"&gt;OpenAI — Agents API overview&lt;/a&gt; (official API docs)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://github.com/openai/openai-cookbook/blob/HEAD/examples/agents_api/README.md" rel="noopener noreferrer"&gt;openai/openai-cookbook — Agents API examples&lt;/a&gt; (official examples)&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>OKF Agent Memory: give your coding agents a git-native memory that survives every session</title>
      <dc:creator>AI Frontier Post</dc:creator>
      <pubDate>Mon, 28 Sep 2026 23:17:46 +0000</pubDate>
      <link>https://dev.to/aifrontierpost/okf-agent-memory-give-your-coding-agents-a-git-native-memory-that-survives-every-session-22ik</link>
      <guid>https://dev.to/aifrontierpost/okf-agent-memory-give-your-coding-agents-a-git-native-memory-that-survives-every-session-22ik</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://aifrontierpost.com/articles/okf-agent-memory-git-native-project-memory/" rel="noopener noreferrer"&gt;AI Frontier Post&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;You know the feeling. You spend an hour with a coding agent establishing the architecture: why the rate limiter sits at the gateway, why cache keys are versioned, which subsystem is frozen. The session ends. Tomorrow the agent is a stranger again — and your &lt;code&gt;AGENTS.md&lt;/code&gt; has quietly grown into a 400-line monolith that the model skims, drifts past, or ignores.&lt;/p&gt;

&lt;p&gt;That is the exact wound &lt;strong&gt;OKF Agent Memory&lt;/strong&gt; is built to close. The project (&lt;a href="https://github.com/okf-memory/okf-agent-memory" rel="noopener noreferrer"&gt;okf-memory/okf-agent-memory&lt;/a&gt;, MIT) launched on September 5, 2026, has gathered 700+ GitHub stars in under four weeks, and topped the weekly trending charts for AI agent repositories. Its proposition is deliberately unglamorous: your agent's durable knowledge lives in your repository as plain Markdown files with YAML frontmatter — a &lt;code&gt;knowledge/&lt;/code&gt; directory you can read, &lt;code&gt;git diff&lt;/code&gt;, and review in a pull request like any other source file.&lt;/p&gt;

&lt;p&gt;What makes it more than a folder of notes is the tooling wrapped around it: a single pure-Go binary with zero dependencies that adds BM25 search (the project reports sub-300-microsecond lookups with no vector database and no embedding API costs), bundle validation, and a stdio MCP server so Claude Code, Cursor, Codex, and other agents can query the memory with tools like &lt;code&gt;okf_search&lt;/code&gt; and &lt;code&gt;okf_show&lt;/code&gt;. No accounts, no API keys, no services to run.&lt;/p&gt;

&lt;p&gt;This tutorial installs &lt;strong&gt;v0.4.4&lt;/strong&gt; (released September 27, 2026), bootstraps a memory bundle into a demo project, records two real architectural concepts, links them, validates the bundle, and then connects the whole thing to an agent over MCP — verifying the protocol handshake with real JSON-RPC calls. Every command below was executed and every output shown is real.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you'll need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Linux, macOS, or Windows&lt;/strong&gt; — the tutorial uses the prebuilt Linux binary (7.6 MB); macOS and Windows builds ship in the same release, and macOS users can also &lt;code&gt;brew install okf-memory/tap/okf&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;A terminal and a scratch project directory.&lt;/strong&gt; We'll scaffold a demo project, so you don't need to touch your own code.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;No API key, no GPU, no database, no account.&lt;/strong&gt; Retrieval is local BM25 in a compiled binary; the bundle is Markdown on disk.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Git&lt;/strong&gt; (optional but recommended) — the bundle is designed to be committed, so memory gains history, blame, and code review for free.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1 — Install okf
&lt;/h2&gt;

&lt;p&gt;Grab the v0.4.4 binary and its checksum file from the GitHub release, then verify the download before trusting it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;curl -sL -o okf https://github.com/okf-memory/okf-agent-memory/releases/download/v0.4.4/okf-linux-amd64
curl -sL -o checksums.txt https://github.com/okf-memory/okf-agent-memory/releases/download/v0.4.4/checksums.txt
grep okf-linux-amd64 checksums.txt
# 073c393644034080713fca2c9c814da5fcd568e85f0a49c99f187d3448424a9f  bin/okf-linux-amd64
sha256sum okf
# 073c393644034080713fca2c9c814da5fcd568e85f0a49c99f187d3448424a9f  okf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The hashes match, so the binary is the one the maintainers published. Make it executable, put it on your &lt;code&gt;PATH&lt;/code&gt;, and confirm the version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;chmod +x okf &amp;amp;&amp;amp; sudo mv okf /usr/local/bin/okf
okf version
# okf version v0.4.4 (OKF v0.2 specification)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A look at the full command surface shows the whole workflow in one screen:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;okf --help
# OKF Agent Memory CLI (vv0.4.4)
#
# Commands:
#   validate               Validate an OKF bundle for conformance and graph health
#   search                 Search concepts using in-memory BM25 scoring or frontmatter filter
#   show                   Display full concept details, frontmatter, and links
#   create                 Create a new concept with automated bookkeeping
#   update                 Update an existing concept
#   relate                 Connect two concepts with a relative link and context
#   init                   Initialize a new OKF v0.2 bundle (index.md, log.md)
#   bootstrap              Scaffold complete memory stack (skill, AGENTS.md, knowledge, Makefile)
#   agents                 Manage AGENTS.md, lint AAG rules, and maintain SSoT tool symlinks
#   mcp                    Run as a Model Context Protocol (MCP) server over stdio
#   hub                    Zero-knowledge sync and vault management (push, pull, sync, serve)
#   version                Print version information
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 2 — Bootstrap memory into a project
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;okf bootstrap&lt;/code&gt; scaffolds the complete memory stack into any directory — new or existing — in one command. We'll use a disposable demo project (replace &lt;code&gt;demo&lt;/code&gt; with your own repo path when you're ready):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;mkdir -p demo
okf bootstrap demo --name "Demo API"
# Successfully bootstrapped OKF Agent Memory in 'demo'!
# Created:
#   - knowledge/ (index.md, log.md)
#   - .agents/skills/okf-memory/ (SKILL.md, discovery, update, etc.)
#   - AGENTS.md
#   - Makefile
#
# Run 'okf validate knowledge' or 'make validate' to verify.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four things landed, and each has a job:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;knowledge/&lt;/code&gt;&lt;/strong&gt; — the OKF v0.2 bundle itself: &lt;code&gt;index.md&lt;/code&gt; (root progressive-disclosure index) and &lt;code&gt;log.md&lt;/code&gt; (dated changelog, starting with today's initialization entry).&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;.agents/skills/okf-memory/&lt;/code&gt;&lt;/strong&gt; — an agent skill definition (&lt;code&gt;SKILL.md&lt;/code&gt; plus &lt;code&gt;discovery.md&lt;/code&gt;, &lt;code&gt;remember.md&lt;/code&gt;, &lt;code&gt;update.md&lt;/code&gt;, &lt;code&gt;relationships.md&lt;/code&gt;, &lt;code&gt;examples.md&lt;/code&gt;) that teaches any agent the search-before-write protocol.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;AGENTS.md&lt;/code&gt;&lt;/strong&gt; — project instructions for AI coding agents in the project's Dual-Memory Agent Architecture: a compact behavioral codex with hard invariants like &lt;em&gt;"MUST execute &lt;code&gt;okf_search(query=keywords, limit=3)&lt;/code&gt; before proposing architecture"&lt;/em&gt; and &lt;em&gt;"NEVER scan &lt;code&gt;knowledge/&lt;/code&gt; via &lt;code&gt;list_dir&lt;/code&gt;, &lt;code&gt;grep_search&lt;/code&gt;, &lt;code&gt;find&lt;/code&gt;, or raw file readers"&lt;/em&gt; — agents must use the search tools, never brute-force the files.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Makefile&lt;/code&gt;&lt;/strong&gt; — conveniences: &lt;code&gt;make validate&lt;/code&gt; runs strict validation, &lt;code&gt;make search q="..."&lt;/code&gt; queries memory.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is the project's central design bet, and it's worth stating plainly: instead of choosing between a bloated prompt monolith (&lt;code&gt;AGENTS.md&lt;/code&gt; stuffed with everything) and a vector-database black box (agents never semantically search for operational rules mid-task), memory is split into a tiny &lt;em&gt;push&lt;/em&gt; layer of invariants loaded at session start and a &lt;em&gt;pull&lt;/em&gt; layer of knowledge retrieved on demand. The bundle costs zero tokens at baseline.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3 — Record decisions as concepts
&lt;/h2&gt;

&lt;p&gt;A &lt;em&gt;concept&lt;/em&gt; is one atomic unit of knowledge: a Markdown file with YAML frontmatter, addressed by an id like &lt;code&gt;decisions/rate-limiting&lt;/code&gt;. Let's record two, the way you would after a design discussion:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cd demo
okf create decisions/rate-limiting knowledge --type Decision \
  --title "Rate limiting strategy" \
  --desc "Standardized on token-bucket rate limiting at the API gateway, 100 req/s per key."
# Created concept 'decisions/rate-limiting.md' (Rate limiting strategy) in 'knowledge'

okf create architecture/cache-layer knowledge --type Concept \
  --title "Redis cache layer" \
  --desc "Redis sits between the API and Postgres for hot read paths; cache keys are versioned by schema revision."
# Created concept 'architecture/cache-layer.md' (Redis cache layer) in 'knowledge'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Concept types are semantic (&lt;code&gt;Decision&lt;/code&gt;, &lt;code&gt;Architecture&lt;/code&gt;, &lt;code&gt;Fact&lt;/code&gt;, &lt;code&gt;Requirement&lt;/code&gt;, &lt;code&gt;Bug&lt;/code&gt;; default &lt;code&gt;Fact&lt;/code&gt;), and &lt;code&gt;create&lt;/code&gt; also accepts &lt;code&gt;--body&lt;/code&gt; for longer Markdown, &lt;code&gt;--tags&lt;/code&gt;, and &lt;code&gt;--status&lt;/code&gt; (&lt;code&gt;draft&lt;/code&gt;, &lt;code&gt;stable&lt;/code&gt;, &lt;code&gt;deprecated&lt;/code&gt;). The file it writes is worth reading in full, because the frontmatter is the whole trust model:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cat knowledge/decisions/rate-limiting.md
# ---
# type: Decision
# title: Rate limiting strategy
# description: "Standardized on token-bucket rate limiting at the API gateway, 100 req/s per key."
# generated: { by: agent/cli, at: "2026-09-28T21:36:14Z" }
# status: stable
# ---
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the &lt;code&gt;generated:&lt;/code&gt; provenance stamp — who wrote it and when. This is the lower of the two trust tiers. The higher tier, &lt;code&gt;verified: human:...&lt;/code&gt;, is reserved for human-confirmed decisions and (per the project's convention, enforced in the generated &lt;code&gt;AGENTS.md&lt;/code&gt;) agents are forbidden from forging it: &lt;em&gt;"NEVER forge human verification (&lt;code&gt;verified:&lt;/code&gt; is human-only)"&lt;/em&gt;. An agent can draft; only a human can stamp. That single rule, sitting in plain YAML where &lt;code&gt;git blame&lt;/code&gt; can see it, is more governance than most agent-memory systems ship.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4 — Search before you ask
&lt;/h2&gt;

&lt;p&gt;The protocol's first rule is &lt;em&gt;search before write&lt;/em&gt;: always query memory before authoring, so concepts never duplicate and the agent never hallucinates a divergent copy of a decision. Search is in-memory BM25 — no embeddings, no network:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;okf search "rate limiting" knowledge
# Found 1 matching concept(s) in 'knowledge':
#
#  1. [context]    [11.78] decisions/rate-limiting (Decision)
#     Standardized on token-bucket rate limiting at the API gateway, 100 req/s per key.
#     Matches: title, description, id
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each hit carries a BM25 score (&lt;code&gt;11.78&lt;/code&gt;), the concept type, and which fields matched. The &lt;code&gt;[context]&lt;/code&gt; tag is the governance tier for this concept — the bundle can also mark concepts as &lt;code&gt;constraint&lt;/code&gt; (must be obeyed) or &lt;code&gt;hold&lt;/code&gt; (subsystem frozen, stop and ask a human), and &lt;code&gt;okf search --for-path pkg/auth/&lt;/code&gt; finds the concepts governing a specific file before an agent edits it.&lt;/p&gt;

&lt;p&gt;Search returns summaries. The full concept — frontmatter, body, and links — comes from &lt;code&gt;show&lt;/code&gt;, and only on demand. That's progressive disclosure: the agent reads 300-token atoms instead of a 20,000-token monolith:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;okf show decisions/rate-limiting knowledge
# ID:          decisions/rate-limiting
# Type:        Decision
# Title:       Rate limiting strategy
# Description: Standardized on token-bucket rate limiting at the API gateway, 100 req/s per key.
# Generated:   2026-09-28T21:36:14Z by agent/cli
#
# --- Body ---
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both commands also speak JSON (&lt;code&gt;--json&lt;/code&gt;) for scripting — &lt;code&gt;okf search "redis cache" knowledge --json&lt;/code&gt; returns structured records with &lt;code&gt;concept_id&lt;/code&gt;, &lt;code&gt;score&lt;/code&gt;, &lt;code&gt;matched_on&lt;/code&gt;, and link lists, ready to pipe into other tools.&lt;/p&gt;

&lt;p&gt;Updating is symmetric with creating:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;okf update decisions/rate-limiting knowledge \
  --desc "Standardized on token-bucket rate limiting at the API gateway, 100 req/s per key; burst capacity 200."
# Updated concept 'decisions/rate-limiting.md' in 'knowledge'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 5 — Link concepts, then prove the bundle is healthy
&lt;/h2&gt;

&lt;p&gt;Concepts become a &lt;em&gt;graph&lt;/em&gt; with &lt;code&gt;relate&lt;/code&gt;, which connects two concepts with a semantic description of the relationship. The flag is &lt;code&gt;--desc&lt;/code&gt; (required):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;okf relate decisions/rate-limiting architecture/cache-layer knowledge \
  --desc "Cache hit rate affects effective gateway throughput under rate limits."
# Linked 'decisions/rate-limiting' -&amp;gt; 'architecture/cache-layer' in 'knowledge'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the part no folder-of-notes can do: &lt;code&gt;validate&lt;/code&gt; checks OKF v0.2 conformance, graph connectivity, and description drift. Run it strict, the way CI would:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;okf validate knowledge --strict --drift
# OKF v0.2 check of "knowledge" (v0.2): 2 concept(s), 0 error(s), 0 gate finding(s),
# 0 warning(s); 0 broken link(s), 0 orphan(s), 0 stale [--strict]. Conformant.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Fully conformant. To see the gates working, try it &lt;em&gt;before&lt;/em&gt; linking: with two unconnected concepts, the same command reports &lt;code&gt;gate architecture/cache-layer.md: orphan (no concept links in or out)&lt;/code&gt; and concludes &lt;em&gt;"Conformant, but the producer gate failed."&lt;/em&gt; Orphan detection, broken-link detection, and stale-review gating (&lt;code&gt;--stale&lt;/code&gt; flags concepts past their &lt;code&gt;stale_after&lt;/code&gt; date) turn memory hygiene into something a pre-commit hook can enforce.&lt;/p&gt;

&lt;p&gt;Two pieces of bookkeeping happened automatically along the way. First, &lt;code&gt;create&lt;/code&gt; and &lt;code&gt;relate&lt;/code&gt; maintain per-directory &lt;code&gt;index.md&lt;/code&gt; files — the progressive-disclosure layer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cat knowledge/decisions/index.md
# # Decisions
# * [Rate limiting strategy](rate-limiting.md) - Standardized on token-bucket rate
#   limiting at the API gateway, 100 req/s per key; burst capacity 200.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An agent reads the index (titles plus one-line descriptions), then pulls full concepts only for what matters. Second, every mutation appends to &lt;code&gt;knowledge/log.md&lt;/code&gt; — a dated changelog of creations, updates, and links. Between the index, the log, and &lt;code&gt;git log&lt;/code&gt;, the memory's entire history is auditable without any special tooling.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxbgl8rquuqgoz76en658.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxbgl8rquuqgoz76en658.webp" alt="Conceptual diagram: a knowledge/ folder tree with index.md, log.md, and concept subdirectories of markdown files with YAML frontmatter, linked into a knowledge graph with a magnifier highlighting one node" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Image: AI-generated illustration for AI Frontier Post.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6 — Hand it to your agent: the MCP server
&lt;/h2&gt;

&lt;p&gt;Everything so far works from the terminal. The reason this is trending with agent-tooling people is the next command: &lt;code&gt;okf&lt;/code&gt; is also a native MCP server over stdio. No separate process to install, no port to manage — the binary speaks the Model Context Protocol on standard input/output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;okf mcp knowledge
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Point Claude Desktop or Cursor at it with the standard MCP server config (from the project's README):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{
  "mcpServers": {
    "okf-memory": {
      "command": "/usr/local/bin/okf",
      "args": ["mcp", "/path/to/your/project/knowledge"]
    }
  }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To prove this isn't just a documented claim, I verified the protocol handshake directly — spawning the server and speaking JSON-RPC to it the way any MCP client would. The &lt;code&gt;initialize&lt;/code&gt; handshake identifies the server, &lt;code&gt;tools/list&lt;/code&gt; exposes the tool surface, and &lt;code&gt;tools/call&lt;/code&gt; runs a real search:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# initialize  -&amp;gt;  {"name": "okf-agent-memory", "version": "v0.4.4"}
# tools/list  -&amp;gt;  okf_search, okf_show, okf_validate, okf_create, okf_update, okf_relate
# tools/call okf_search {"query": "rate limiting", "limit": 3}  -&amp;gt;
# [{
#   "concept_id": "decisions/rate-limiting",
#   "title": "Rate limiting strategy",
#   "type": "Decision",
#   "description": "Standardized on token-bucket rate limiting at the API gateway, 100 req/s per key; burst capacity 200.",
#   "governance": "context",
#   "score": 13.17,
#   "matched_on": ["title", "description", "id", "body"],
#   "outbound": ["architecture/cache-layer"]
# }]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the &lt;code&gt;outbound&lt;/code&gt; field: the search result carries the link we created with &lt;code&gt;relate&lt;/code&gt;, so the agent can walk the graph from a hit to its related concepts. Once configured, an agent with the bootstrapped skill follows the loop the &lt;code&gt;AGENTS.md&lt;/code&gt; codex mandates: &lt;code&gt;okf_search&lt;/code&gt; before proposing changes, &lt;code&gt;okf_show&lt;/code&gt; only for relevant hits, &lt;code&gt;okf_update&lt;/code&gt; preferred over &lt;code&gt;okf_create&lt;/code&gt; when the concept already exists.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjcbstnqa06rmubwilup.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjcbstnqa06rmubwilup.webp" alt="Conceptual illustration: a laptop code editor sends a question through a glowing terminal pipe to a small local binary that returns ranked memory cards, while a large unused database server fades in the background" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Image: AI-generated illustration for AI Frontier Post.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  OKF vs the alternatives
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Approach&lt;/th&gt;
&lt;th&gt;Strength&lt;/th&gt;
&lt;th&gt;Where OKF wins&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Vector-DB memory (Mem0, Letta)&lt;/td&gt;
&lt;td&gt;Semantic similarity across large corpora&lt;/td&gt;
&lt;td&gt;No embedding API costs or latency, no service to operate, deterministic lexical ranking; agents retrieve operational rules they would never think to semantically search for&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Prompt monolith (giant CLAUDE.md)&lt;/td&gt;
&lt;td&gt;Zero tooling, always loaded&lt;/td&gt;
&lt;td&gt;Bundle costs zero tokens at baseline; progressive disclosure avoids attention drift from 20k-token dumps&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Raw markdown notes&lt;/td&gt;
&lt;td&gt;Simple, git-native&lt;/td&gt;
&lt;td&gt;Adds search, validation gates, provenance tiers, and an agent protocol on top of the same plain files&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OKF Agent Memory&lt;/td&gt;
&lt;td&gt;Git-native, local, validated, agent-queryable&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;—&lt;/p&gt;

&lt;p&gt;The honest limitation: BM25 is lexical. If your agent phrases things with entirely different vocabulary than your concepts used, keyword search can miss where embeddings would connect. The project bets — reasonably, for operational knowledge like decisions, constraints, and runbooks — that shared project vocabulary dominates, and that determinism and zero cost are worth the trade. There is also a &lt;code&gt;hub&lt;/code&gt; subcommand for zero-knowledge encrypted sync of the bundle across machines; I did not exercise it here, and it needs a password and secret key, so treat it as a documented-but-unverified extra.&lt;/p&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;The reason OKF Agent Memory is climbing isn't a benchmark — it's that the artifact matches the workflow developers already trust. Memory is a folder of Markdown in the repo. It gets committed, diffed, blamed, and reviewed. The binary around it does the three things folders can't: search it in microseconds, validate it in CI, and serve it to agents over MCP. The trust model is one YAML key (&lt;code&gt;generated&lt;/code&gt; vs &lt;code&gt;verified&lt;/code&gt;), and the governance model is one flag (&lt;code&gt;--for-path&lt;/code&gt; before you edit).&lt;/p&gt;

&lt;p&gt;Run &lt;code&gt;okf bootstrap&lt;/code&gt; on a real project tonight. Record the three decisions your agent keeps forgetting — the ones you re-explain every session. Then ask the agent about one of them in a fresh session and watch it &lt;code&gt;okf_search&lt;/code&gt; first instead of guessing. If it answers from your memory instead of its priors, you'll understand the 700 stars.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
