There is a common assumption that making animated content with AI means reaching for a text-to-video model. For a great deal of what actually gets published — a chart that counts up, a diagram that reveals step by step, a title that slides in — that assumption is both expensive and wrong.
Sora 2's API costs $0.10 a second at 720p and $0.30 to $0.70 a second for the Pro tier, and OpenAI has scheduled the whole API to sunset on September 24, 2026. Meanwhile the failure mode benchmarks keep documenting for these models is exactly the content above: legible text. Researchers call it glyph collapse — letters that melt, labels that almost spell the word. A generative video model is the wrong tool for a bar chart the way a diffusion model is the wrong tool for a spreadsheet.
The alternative is not "no AI." It is a different division of labour: the LLM authors a machine-checkable spec, and deterministic code renders the pixels. I want to lay out that architecture properly, because it is the thing that actually works, and almost nobody writing about AI video describes it.
When you actually need a generative model
Let me concede the real cases first. If you need a photoreal scene, a human being, camera motion through a physical space, or anything whose content has no data structure underneath it — a drone shot, an actor, weather — then you need a generative model, and nothing here applies. That is what these models are for, and when the subject is genuinely visual they are astonishing.
The argument is narrower: most of the video content businesses and developers actually ship is not that.
Most video content is structured — and structured content has a spec
Look at what actually gets published: product explainers, stat announcements, process walkthroughs, tutorial intros, quarterly-numbers posts, route maps, launch countdowns. Strip the styling away and every one of them is structured data — words, numbers, nodes and edges, coordinates, timestamps.
Content with structure can be described by a spec, and a spec can be rendered by code: deterministically, for free, at any resolution, with every glyph a real vector.
- Kinetic text — a string, a style, a timing curve.
- Charts and stats — rows of labels and values, a template, a duration.
- Diagrams and flowcharts — a typed graph of nodes and edges.
- Maps — stops, routes, a frame, a camera.
- UI walkthroughs — screens and highlights on a timeline.
The tooling for this is mature and boring, which is a compliment. Remotion renders React components to video; Motion Canvas and its pipeline-oriented fork Revideo drive animations from TypeScript generators; Manim is the engine behind every 3Blue1Brown-style math explainer. Our own approach is narrower still: everything is CSS keyframes or SMIL inside the SVG itself, which is why an exported file animates with no runtime library at all.
Different trade-offs, same property — the pixels are computed, not sampled from a distribution.
Every label and number in this chart is a real vector glyph — the exact thing text-to-video models are benchmarked failing at. Rendering it cost nothing and took milliseconds. (Shown here as a GIF for dev.to; the original is a 20 KB SVG.)
The architecture: agents author specs, code renders pixels
Here is the shape that works, end to end.
A storyboard agent goes first. It takes the brief and writes the story frame by frame — what the viewer sees, in what order, for how long. Storyboarding is a language task, which is why an LLM is good at it and why it must not be skipped: every downstream decision hangs off this document.
An animation reasoner then walks the storyboard and decides which parts need animation at all, and of what kind. Not everything moves. A held title over silence is a legitimate scene, and a model told it may choose stillness produces better pacing than one told to animate everything.
Then the part that carries the whole design.
Tool-call contracts are the interface between generation and determinism
Every animation category — kinetic text, chart, diagram, map, UI — is exposed to the model as a tool call, and the tool's schema is the taxonomy of that animation form.
// The chart contract. The model fills this; it never emits pixels,
// timelines, or code.
{
"name": "animate_chart",
"description": "Render an animated chart segment for one storyboard beat.",
"parameters": {
"type": "object",
"required": ["template", "rows", "durationSec"],
"properties": {
"template": {
"type": "string",
"enum": ["bar-chart", "bar-race", "count-up", "progress-ring", "pie-chart"]
},
"rows": {
"type": "array",
"minItems": 2,
"maxItems": 8,
"items": {
"type": "object",
"required": ["label", "value"],
"properties": {
"label": { "type": "string", "maxLength": 20 },
"value": { "type": "number", "minimum": 0 }
}
}
},
"durationSec": { "type": "number", "minimum": 1, "maximum": 12 }
}
}
}
Look at what that schema encodes: not just types, but the taxonomy of the form. Five templates, because those are the chart animations that exist in this system. maxLength: 20 on a label, because a longer string does not fit the bar. maxItems: 8, because nine rows will not lay out. The constraints a human designer would apply by eye are written down where the model must obey them.
The tool-calling layer forces the return into exactly the shape the renderer needs. Validation is not a review step bolted on afterwards — it is the interface.
Our flowchart studio runs precisely this loop in production. When the authoring agent's rows fail validation, the repair prompt receives the rejection reason and nothing else, and retries are bounded:
// Bounded repair. The model gets the reason, not the conversation.
async function authorSpec(brief, { maxRepairs = 2 } = {}) {
let attempt = await callModel(AUTHOR_SYSTEM, brief);
for (let i = 0; i <= maxRepairs; i++) {
const problems = validate(attempt); // pure, synchronous, no model
if (!problems.length) return attempt;
if (i === maxRepairs) throw new SpecError(problems);
attempt = await callModel(REPAIR_SYSTEM, {
spec: attempt,
problems, // ← the only new information
});
}
}
A spec that cannot be checked by a machine is a prompt, not a spec.
From there the pipeline is deliberately unintelligent. Deterministic renderers turn each filled contract into an animated segment — the same input producing the same bytes every time. A stitcher assembles the segments with ffmpeg. No model touches this stage, which is exactly why its output can be trusted.
The architecture. Generation on the left of each contract, determinism on the right — and a verifier that never has to watch a video.
The verifier samples frames, because nobody can watch a video
LLMs cannot watch video. Even with multimodal models, shipping a whole video through the context window is the wrong idea — expensive, lossy, and slow.
But the verifier does not need the video. Because rendering is deterministic, it can pick a handful of plausible timestamps — mid-reveal, at a label's settle point, the final frame — request stills, and inspect those.
// Sample where the animation is supposed to be interesting,
// not at uniform intervals.
const probes = [
{ t: 0.35 * dur, expect: "bars partially grown, labels legible" },
{ t: 0.70 * dur, expect: "ranking has changed at least once" },
{ t: dur, expect: "final values match the spec exactly" },
];
for (const probe of probes) {
const png = await renderStillAt(segment, probe.t); // deterministic seek
const verdict = await vision(png, probe.expect);
if (!verdict.ok) return repair(segment, verdict.reason);
}
This is a lesson we learned directly from our own capture pipeline: frames are produced by seeking the animation clock, never by racing wall-clock time, so "the frame at 2.4 seconds" is a stable, reproducible artifact. Verification of sampled stills is only trustworthy because the renderer is deterministic — one more reason the pixels must come from code. A failed check feeds the bounded repair loop back to the agent that owns the failing spec, with the reason attached.
The voiceover lane runs in parallel
Narration does not need to wait for pixels. A script agent writes the voiceover from the same storyboard the visuals came from, a TTS model speaks it, and a sync step aligns the audio to the stitched timeline.
Because both lanes derive from one storyboard with explicit durations, sync is arithmetic rather than guesswork. The storyboard is the single source of truth; everything else is a projection of it.
Tools that already work this way
Once you see the pattern, it is everywhere.
Google's NotebookLM Video Overviews began as narrated slides — an AI script over a deterministic slide renderer, the pattern in its purest form. The newer Cinematic tier has Gemini acting, in Google's own words, as a creative director orchestrating image and video models. The orchestration is still an agent making structural decisions; the generative models are tools it calls.
Easymotion turns chat into map animations and animated charts from uploaded spreadsheets — prompt in, templated deterministic motion out, from around $10 a month.
And it is how our own flowchart studio works. In Describe it mode, an agent authors rows for one of the same templates a human fills in by hand — it lands in the same editor, passes the same validation, renders through the same layout and timing code. In Upload one mode, a vision agent reads a photographed diagram into a typed graph. In both, the model authors structure and the templates own geometry and motion. The infographic studio draws the same line: AI may suggest the data; the animation is computed.
The class of output the Describe-it agent authors rows for. Layout and timing are computed by the template — the agent never touches either.
Everything moving in this post was made this way
Both animations above are our product's real output — authored as specs, rendered by code. On the original post they are self-contained SVGs that replay because the document itself describes the motion; the GIFs here came out of the same deterministic pipeline this article describes, which is also why they were free to produce.
No video model was involved, no render farm, no per-second bill.
That is the honest summary: when your content has structure, let a model author the spec and let code make the pixels — and save the generative model for the scenes that deserve it.
Top comments (0)