DEV Community

Cover image for Empty Folder to MP4 in 162 Seconds: What a Coding Agent Can't Do With Video
tia-forge
tia-forge

Posted on Originally published at hubwiz.com

Empty Folder to MP4 in 162 Seconds: What a Coding Agent Can't Do With Video

#ai #javascript #tutorial #opensource

The one-paragraph version. There are two unrelated technologies behind the phrase "AI video". One paints pixels from a prompt (Veo, Gemini Omni, Sora). The other renders a program into frames — a video is a React component tree or an HTML file, and the output is deterministic. Only the second one is something a coding agent can own end to end: it writes files, reads type errors, renders, looks at frames. We ran the whole loop on a clean machine and timed it. Total to first MP4: about three minutes. Then we hit the wall that every article about this topic skips — the agent has no way to know whether what it rendered is any good.

Disclosure. All commands and timings below were run by us on 2026-10-26 (Windows 11, 8-core CPU, Node 24.21.0, npm 11.19.0, Remotion 4.0.538). Everything else — license terms, skill counts, design rationale — is quoted from the vendor's own docs and is linked in Sources. We did not benchmark cloud rendering, and we did not run a paid generative model.

1. Two things are called "AI video"

Prompt → pixels Code → frames
Examples Veo 3.1, Gemini Omni Flash, Sora Remotion, HyperFrames
Unit of authoring A sentence A file in git
Output for the same input Different every time (sampling) Byte-identical
Editing a detail Re-prompt, hope Change one number, re-render
What the model must know Cinematography, physics, faces The API you gave it
Where an agent fits Calls an API Owns the whole repo

The distinction matters because the two have wildly different failure modes. A diffusion model fails by hallucinating a sixth finger; a render pipeline fails by throwing TypeError: undefined is not a function. Coding agents are good at the second kind of failure and useless at the first.

That is the whole reason this niche got hot in 2026: Remotion shipped Agent Skills in January, HeyGen open-sourced HyperFrames in April, and both are pure text — which is the only substrate a coding agent actually understands.

2. What "video as code" means mechanically

The mental model is one line: a frame is a pure function of time.

const frame = useCurrentFrame();          // 0, 1, 2, … 149
const p = spring({ frame, fps, config: { damping: 200 } });
return <div style={{ opacity: p }}>…</div>;
Enter fullscreen mode Exit fullscreen mode

There is no timeline, no keyframe UI, no video editor. You write a component that, given a frame number, returns what that frame looks like. The renderer then:

  1. bundles the project (Remotion) or loads index.html as-is (HyperFrames),
  2. opens headless Chrome and seeks each timestamp,
  3. screenshots the frame,
  4. pipes the frames into FFmpeg.

Both frameworks do this. HyperFrames' README states the property that makes it usable in CI: "The renderer seeks each frame in headless Chrome and encodes the result with FFmpeg, so the same input produces the same video." Remotion's pitch is the same idea with a React component tree as the authoring surface.

Why agents like it: the artifact is TypeScript or HTML, the feedback loop is a compiler, and the diff is reviewable. git diff on a video is a strange sentence that is now literally true.

3. The demo, with real numbers

Four commands, no editor, no API key.

npx create-video@latest --yes --blank demo   # scaffold
cd demo
npx skills add remotion-dev/skills -a claude-code -y   # 12 skills → .claude/skills
npx remotion render MyComp out/video.mp4      # render
Enter fullscreen mode Exit fullscreen mode

Measured, on the machine above:

Step Wall time Note
create-video --yes --blank 1 m 56 s almost entirely npm install; dominates the clock
skills add … 6.0 s 12 skill folders written
first remotion render (5 s @ 1080p) 39.9 s includes downloading headless Chrome
same render, warm 11.7 s 150 frames, 1080p30, --concurrency default = cores/2
60 s @ 1080p 68.7 s 1800 frames; ≈ 26 frames/s of output
60 s @ 1080p, --concurrency=2 84.4 s forcing 2 workers is 23% slower on 8 cores

Two things fall out of that table. First, render time is linear in duration: 150 frames took 11.7 s, 1800 frames took 68.7 s — the same ~26 frames per second of output either way (we only measured one composition, so treat "per frame" as roughly constant, not proven constant). Second, the "three minutes to your first video" claim is real, but it is scaffold time, not authoring time: 116 s of install + 6 s of skills + 40 s of cold render ≈ 162 s, and the npm install is two-thirds of it.

A note on the install command, because half the articles online get it wrong. Remotion's own docs now say npx remotion skills add. The generic installer also works: npx skills add remotion-dev/skills -a claude-code -y. They write to different directories — the first put 12 skills in .agents/skills/ and created a skills-lock.json; the second put 12 skills in .claude/skills/. If your agent "can't see the skills", check which directory got them before you debug anything else.

4. What the skills actually cost you in context

People assume installing agent skills means stuffing a manual into the prompt. It doesn't, and the numbers are worth knowing:

Measurement Value
Skills installed 12
Frontmatter (name + description) of all 12 1,031 chars ≈ 290 tokens — this is the always-visible part
Full SKILL.md bodies, all 12 43 KB ≈ 12k tokens, read on demand
Support files (docs, example .tsx, SVGs, map data) 285 files / 2.3 MB — opened only if the agent asks

The biggest single skill is remotion-best-practices, whose SKILL.md is a 4.2 KB router: it tells the agent which of the other eleven references to load for the task at hand. That's the whole design — pay ~300 tokens up front, buy 12k tokens of API knowledge. It also means the failure mode is silent: if the router doesn't fire, the agent writes Remotion from memory and you get plausible-looking code that renders a black screen.

5. The cost model nobody writes down

Two numbers matter: license and compute. Compute you can measure (§3). License is where the surprise is.

Remotion HyperFrames
License Source-available Remotion License Apache 2.0
Free tier Individuals and companies of up to 3 people; unlimited commercial use No per-render fees, no commercial-use threshold
Paid tier Company License: Creators $25/seat/mo, or Automators $0.01/render, $100/mo minimum —
Render engine Headless Chrome + FFmpeg Headless Chrome + FFmpeg
Authoring React components HTML + CSS + seekable animation (GSAP, Lottie, Three.js, WAAPI…)

The threshold is the thing to read twice. A three-person studio is free. Hire a fourth person and you are, by the license's own terms, in Company License territory — and if you're building a product that renders videos for other people, that's the Automators plan at $0.01/render with a $100/month floor. For a tool that renders 50,000 personalized clips a month, $500 is fine. For a side project that renders nine, it is not.

HyperFrames is the escape hatch: Apache 2.0, same rendering mechanism, different authoring bet (HTML instead of React). Its README is blunt about the trade-off — "Remotion's bet is React components; HyperFrames' bet is plain HTML that humans and agents can both write easily."

For the token side, one datapoint rather than a claim: NiharDaily's October 2026 hands-on reports $0.13 in model cost for a first prompt-to-MP4 run in Claude Code. Treat that as an order of magnitude, not a benchmark — it depends entirely on how many times the agent renders, reads a frame, and iterates.

6. Where it actually breaks

This is the section that decides whether you ship or abandon, and it's the part the tutorials skip.

The verification gap. "It compiles" is not "it looks good." An agent can run remotion render and get a green exit code from a composition where the text is white on white, the logo is cropped, or the animation finished at frame 12. Digital Applied calls this the hardest self-verification problem in the agentic toolchain, and they're right: there is no unit test for "does this look right", and the visual-regression tooling that exists measures pixel differences, not intent.

Practical workarounds, in order of usefulness:

  1. Give the agent a component library, not a blank canvas. Improvised motion is exactly where AI video looks cheap. If it composes <LowerThird/> and <StatCard/> instead of inventing springs, the floor on quality rises enormously.
  2. Make it render stills, not videos, while iterating. npx remotion still MyComp out/f.png --frame=45 took 6.7 s; a full 5-second render takes 11.7 s and a 60-second one takes 68.7 s. Bundling is a fixed ~5 s cost, so the longer your composition, the more the still saves — and it catches most layout errors.
  3. Have it read the frame back. Agents with vision can open out/frame.png. Without this, you are the verification loop.

Everything else that bites:

  • Your first render is not your render time. Headless Chrome download alone is most of the 40 s. Budget for it in CI warm-up, not in your demo.
  • Fonts and media load asynchronously. A frame screenshot that races an image decode is a flaky test you'll spend a day on. Local assets, not CDN URLs.
  • Long videos stay linear. 10 minutes at 1080p is ~11 minutes of render on this machine. That's fine locally, unacceptable in a request handler — this is what Remotion Lambda and HyperFrames' Lambda path exist for.
  • The license threshold is a headcount trigger, not a usage trigger. Track it before legal does.
  • Auto-generated is visible. If you can tell a video was made from a prompt, the tool didn't fail; the brief did.

7. When to use which

You need Use
Motion graphics, explainers, charts, lower-thirds, data-driven or parameterized clips Video as code (Remotion / HyperFrames)
Photoreal footage, people, b-roll, "a rainy street in Osaka" A generative model (Veo 3.1 / Gemini Omni Flash), then assemble in code
A talking head An avatar product; neither of the above
10,000 personalized variants of one template Video as code + Lambda; this is the killer use case
One 30-second promo, once Neither. Open an editor.

The honest framing: video as code is not a cheaper way to make the video you were going to make in Premiere. It's a way to make videos that didn't exist as a category before — one per customer, one per data update, one per pull request.

8. Checklist before you commit

  • [ ] Decide the authoring surface first (React vs HTML). Migrating later is a rewrite, not a refactor.
  • [ ] Read the license terms against your current headcount, not next year's.
  • [ ] Ship a component library on day one; never let the agent invent motion.
  • [ ] Use remotion still (or render --frames=0-59) for iteration; full render only on merge.
  • [ ] Keep media local; pin fonts.
  • [ ] Put a human on frame review — that loop is the product.
  • [ ] Measure render seconds per output second on your box; ours was 26 fps of output at 1080p.

Sources

Further reading (in Chinese)

Commands and timings are our own (2026-10-26). Product claims are attributed above. Chinese original: hubwiz.

Top comments (0)