How Agentic Video Editing Works: Four Designs and What Each One Costs You

In June 2026, Thariq Shihipar of Anthropic's Claude Code team had Claude Fable 5 edit the launch video for Fable 5 itself: 17 takes of raw 4K footage, about 25 GB. According to his deck, no video editor was opened.

What stands out to me is what the edit consisted of: a JSON file listing every cut with the reason for it, seven color lookup tables (LUTs), and a set of React components. The deck's summary is that "the edit is text", which let Claude read, compare and regenerate it. My argument is that this is the question to ask of any agentic video editing setup: what is the edit made of, and can anyone besides the agent open it?

Quick answer: Agentic video editing means an AI agent takes a goal, plans the steps, runs editing tools itself, checks the result and revises when you push back. Tools differ mainly in where the plan lives, either inside a vendor's app or in files you own. That choice sets what you can inspect, verify and change later.

What is agentic video editing?

An editor is agentic when it works from a goal, uses real editing tools in sequence, checks its own work and revises when you push back. That is Tella's definition. Invideo adds that the agent understands the footage and leaves its work on a timeline a person can change. A single feature such as auto-captions or silence removal does not qualify.

Agentic video editing vs agentic video generation

	Editing	Generation
Starts from	Footage you already have	An idea, script or novel
The agent's job	Decide what to keep, cut and add, then drive editing tools	Plan the story and shots, then call image and video models
Hard problem	Judgment about which moments to keep	Consistency of characters and scenes across shots
Example	The Fable 5 launch edit, video-use, Tella	ViMax

The generation column describes ViMax, an MIT-licensed framework whose configuration needs a chat model, an image generator and a video generator. A Reelful write-up draws the line from the editing side: a generator can always produce something that technically works, while an editor has to find the good moments inside imperfect footage.

A third group sits between the two: agents that author and revise a video as project files, such as CoAnimator. Its own page says it is not built for cutting footage you already shot, so it belongs with the tools for videos that do not exist yet.

How does agentic video editing work?

Most systems run the same loop and differ in how much a person approves along the way.

footage -> understand -> plan -> apply -> check -> render -> review
                           ^                |                   |
                           +-- revise ------+-------------------+

1. Understand: turn footage into text, such as transcripts with word timestamps.
2. Plan: write down the intended cuts. Many systems pause here for approval.
3. Apply: run editing tools, or write a file or code.
4. Check: validate the plan, re-transcribe the cut, or look at stills.
5. Render and review: produce the video. Your feedback restarts the loop.

I drew this loop from the examples below. No single source uses these names. Here it is on a real job, with every number taken from Thariq's deck:

Step	What the agent did	What it left behind
1. Listen	Transcribed all 17 takes with word timestamps	transcripts/*.json
2. Choose	One subagent per scene picked the best take; verifiers double-checked	final-edit.json with in and out points and a written reason for each pick
3. Cut	ffmpeg executed the JSON; Claude re-transcribed its own cut to confirm the filler words were gone	A 2:50 rough cut
4. Grade	Claude wrote candidate looks; Thariq chose one	7 hand-written .cube LUTs
5. Graphics	Rebuilt 11 static frames as React components, cued to word timestamps	Remotion components, six timing numbers
6. Design review	Exported to Figma through MCP, imported the designers' redesign, updated the video	A Figma file, two round trips
7. Render	4,334 frames at 3840x2160, 24 fps, with stills screenshotted for review before each full pass	The finished 3:00 file, 653 MB

It took four days (June 6 to 9), with about ten re-renders in one night.

What are the four ways to hold the edit?

Every setup keeps the edit somewhere. MCP, the Model Context Protocol, is the interface that lets an agent call an application's tools.

Design	Where the edit lives	Examples	What it costs you
1. Hosted timeline	Inside the vendor's editor	Invideo's agent works on the same multitrack timeline as manual editing; Goldcast's chat assistant	Whatever the product allows. Goldcast handles videos under five minutes, and a later agent run may restyle captions even though your manual edits are kept
2. Agent over MCP	Inside the vendor's app, reached through tools	Tella, Descript and invideo run hosted MCP servers, per Tella's comparison; Palmier Pro's is local	The agent can do only what the exposed tools allow, and access can depend on a plan or on the app being open
3. Data file	A JSON plan you own	The Fable 5 cuts; video-use, which waits for your approval before cutting; videopython, which validates plans before decoding frames	Only operations the format defines can appear. video-use needs an ElevenLabs API key, and videopython is pre-1.0
4. Code	Source code or project files you own	The Fable 5 graphics, where six timing numbers drive every animation; Reelful, which writes Remotion code; CoAnimator, where agents edit project files that sit in git	Taste has to become parameters, and someone has to be able to review the code

Which agentic video editing tools should you use in 2026?

Tool	Design	Best for	Access and limits
Tella	1 and 2	Screen recordings, demos, tutorials	In-editor agent included with paid plans and free trials, running on credits; hosted MCP
Descript	1 and 2	Talking-head video edited as text	Hosted MCP
Riverside	1 and 2	Podcasts and interviews	MCP on Grow plans and above
invideo	1 and 2	Multitrack timeline plus generation	Hosted MCP; credit-based plans
Goldcast	1	Event and webinar recordings	Videos under five minutes
video-use	3	Spoken footage, cut from a terminal	MIT license; ElevenLabs key; any agent with shell access
videopython	3	Scripted edits in Python	Apache-2.0; pre-1.0; local-first
Remotion	4	Motion graphics, code-based video	Source-available; free for individuals and companies of up to three people
CoAnimator	4	Demos, tutorials, launch videos and motion graphics that an agent authors as project files	Free tier: unlimited local renders at 720p with a watermark. A $299 one-time licence, listed as a limited-time price, adds 4K and no watermark. Works with Claude Code, Codex, Antigravity and OpenClaw.

Tella, Descript, Riverside and invideo rows come from Tella's comparison, a vendor page dated October 16, 2026, and Tella's docs. The rest come from each tool's own pages.

Where to start: for recorded talks or demos without code, a hosted editor. For spoken footage from a terminal, video-use. For control in Python, the agent built below. For demos, tutorials and launch videos that an agent authors as project files you keep in git and render locally, CoAnimator. For motion graphics in React, Remotion with a coding agent, as in its own five-minute tutorial:

Should an editing agent read the transcript or watch the pixels?

Invideo argues that a transcript cannot describe expressions, camera angles or continuity, so the agent should watch the footage. video-use takes the opposite default: its README says the model never watches the video. It reads a packed transcript (about 12 KB in its example) and asks for images only at decision points. The Fable 5 job also relied on transcripts and stills.

My read is that words and silence carry most of what a cut needs. Thariq's second job complicates that. In a July interview covered by Simon Willison, he said Fable noticed an auto-update popup in a slide recording, rebuilt the slides from his HTML deck, and cropped the stage video to follow him as he paced. He does not say how it perceived those things, but they are not transcript-level decisions. That is an opinion, and no source compares the approaches. The editing story starts at 41:50 in the full session:

How do you build an agentic video editing agent?

This agent cuts filler words and long pauses from a talking-head video. It needs Python 3.9 or later and ffmpeg. edit_agent.py uses only the standard library. Tested with Python 3.12 and ffmpeg 6.1.

1. Get word-level timestamps

Any speech recognition tool that returns a start and end time per word will do. Save this hand-made 8.5 second example, with two fillers and a long pause, as transcript.json:

{"words": [
  {"word": "Hello",    "start": 0.40, "end": 0.80},
  {"word": "everyone", "start": 0.85, "end": 1.40},
  {"word": "um",       "start": 1.60, "end": 1.90},
  {"word": "today",    "start": 2.40, "end": 2.80},
  {"word": "we",       "start": 2.85, "end": 3.00},
  {"word": "talk",     "start": 3.05, "end": 3.50},
  {"word": "editing",  "start": 3.55, "end": 4.10},
  {"word": "uh",       "start": 6.00, "end": 6.30},
  {"word": "it",       "start": 7.00, "end": 7.20},
  {"word": "works",    "start": 7.25, "end": 7.70}
]}

2. Write the agent

Save this as edit_agent.py. The model is a function you pass in.

import argparse
import json
import subprocess

MIN_SEGMENT, EPS = 0.5, 0.001  # seconds; EPS absorbs float noise


def probe(path, *args):
    out = subprocess.run(["ffprobe", "-v", "error", *args, path],
                         capture_output=True, text=True, check=True)
    return out.stdout.strip()


def duration(path):
    return float(probe(path, "-show_entries", "format=duration", "-of", "default=nw=1:nk=1"))


def is_num(x):
    return isinstance(x, (int, float)) and not isinstance(x, bool)


def validate(plan, words, total):
    """Return structured errors. An empty list means the plan is safe to render."""
    segs = plan.get("segments") if isinstance(plan, dict) else None
    if not segs or not isinstance(segs, list):
        return [{"error": "plan needs a non-empty 'segments' list"}]
    errors, prev_end = [], 0.0
    bad = lambda i, msg: errors.append({"segment": i, "error": msg})
    for i, seg in enumerate(segs):
        if not (isinstance(seg, dict) and is_num(seg.get("start")) and is_num(seg.get("end"))):
            bad(i, "needs numeric 'start' and 'end'")
            continue
        s, e = seg["start"], seg["end"]
        if s < 0 or e > total + EPS:
            bad(i, f"{s}-{e}s is outside the media (0-{total:.2f}s)")
        if e <= s:
            bad(i, "end must be after start")
            continue
        if e - s < MIN_SEGMENT:
            bad(i, f"shorter than {MIN_SEGMENT}s")
        if s < prev_end - EPS:
            bad(i, f"starts at {s}s, before the previous segment ends at {prev_end}s")
        for t in (s, e):
            for w in words:
                if w["start"] + EPS < t < w["end"] - EPS:
                    bad(i, f"cut at {t}s falls inside the word '{w['word']}' "
                           f"({w['start']}-{w['end']}s)")
        prev_end = max(prev_end, e)
    return errors


def run_agent(brief, words, total, propose, max_rounds=3):
    """Ask the model for a plan, validate it, and feed errors back until it passes."""
    feedback = []
    for n in range(1, max_rounds + 1):
        try:
            plan = propose(brief, words, feedback)
        except ValueError as exc:  # the reply was not usable JSON
            feedback = [{"error": f"reply was not valid JSON: {exc}"}]
            print(f"round {n}: unusable reply")
            continue
        errors = validate(plan, words, total)
        print(f"round {n}: {len(errors)} error(s)")
        for error in errors:
            print("  ", json.dumps(error))
        if not errors:
            return plan
        feedback = errors
    raise SystemExit(f"no valid plan after {max_rounds} rounds")


def render(plan, media, out_path):
    filters, labels = [], []
    for i, seg in enumerate(plan["segments"]):
        s, e = seg["start"], seg["end"]
        filters.append(f"[0:v]trim=start={s}:end={e},setpts=PTS-STARTPTS[v{i}]")
        filters.append(f"[0:a]atrim=start={s}:end={e},asetpts=PTS-STARTPTS[a{i}]")
        labels.append(f"[v{i}][a{i}]")
    filters.append(f"{''.join(labels)}concat=n={len(labels)}:v=1:a=1[v][a]")
    subprocess.run(["ffmpeg", "-y", "-i", media, "-filter_complex", ";".join(filters),
                    "-map", "[v]", "-map", "[a]", "-pix_fmt", "yuv420p", out_path],
                   check=True, capture_output=True)


def verify(plan, out_path, tolerance=0.15):
    """Check the rendered file against the plan. Returns a list of problems."""
    expected = sum(s["end"] - s["start"] for s in plan["segments"])
    actual = duration(out_path)
    problems = []
    if abs(actual - expected) > tolerance:
        problems.append(f"duration {actual:.2f}s, plan says {expected:.2f}s")
    if not probe(out_path, "-select_streams", "a", "-show_entries", "stream=codec_type",
                 "-of", "csv=p=0"):
        problems.append("output has no audio stream")
    return problems


def main(propose, argv=None):
    ap = argparse.ArgumentParser()
    ap.add_argument("media")
    ap.add_argument("transcript", help='JSON: {"words": [{"word", "start", "end"}]}')
    ap.add_argument("brief")
    ap.add_argument("--out", default="out.mp4")
    ap.add_argument("--yes", action="store_true", help="skip the approval prompt")
    args = ap.parse_args(argv)

    with open(args.transcript) as f:
        words = json.load(f)["words"]
    plan = run_agent(args.brief, words, duration(args.media), propose)
    with open("plan.json", "w") as f:
        json.dump(plan, f, indent=2)  # the edit is a file you can diff
    for i, seg in enumerate(plan["segments"]):  # show what each kept range says
        said = " ".join(w["word"] for w in words
                        if seg["start"] <= (w["start"] + w["end"]) / 2 <= seg["end"])
        print(f"{i}: {seg['start']:.2f}-{seg['end']:.2f}s  {said!r}  {seg.get('reason', '')}")
    if not args.yes and input("Render this plan? [y/N] ").strip().lower() != "y":
        raise SystemExit("not approved")
    try:
        render(plan, args.media, args.out)
    except subprocess.CalledProcessError as exc:
        raise SystemExit(exc.stderr.decode())
    problems = verify(plan, args.out)
    if problems:
        raise SystemExit("verification failed: " + "; ".join(problems))
    print(f"rendered {args.out} and verified it against the plan")

• validate rejects plans that are malformed, run outside the media, overlap, contain a segment under half a second, or cut inside a spoken word. The last rule follows video-use's README, which says its cuts come from speech boundaries and silence gaps. It returns every error at once.
• run_agent is the loop: ask for a plan, validate it, send errors back, repeat. A reply that is not valid JSON counts as an error.
• main saves plan.json, prints what each kept range says, waits for your approval, renders, and verifies the duration and audio.

3. Connect a model

Save this as propose_anthropic.py, run pip install anthropic, and set ANTHROPIC_API_KEY. It is the only provider-specific code, and any model that can follow a JSON format will work.

import json

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment
MODEL = "claude-sonnet-5-5"     # any model that can follow a JSON format works

FENCE = "`" * 3  # the code fence some models wrap JSON in

SYSTEM = (
    "You are a video editing planner. Reply with JSON only, in this shape: "
    '{"segments": [{"start": <seconds>, "end": <seconds>, "reason": "<why kept>"}]}. '
    "Segments are the ranges to KEEP, in order, with no overlap. "
    "Every start and end must fall in the silence between two words, never inside a word."
)


def propose(brief, words, feedback):
    lines = "\n".join(f"{w['start']:.2f}-{w['end']:.2f} {w['word']}" for w in words)
    content = f"Brief: {brief}\n\nWord timestamps:\n{lines}"
    if feedback:
        content += ("\n\nYour last plan failed validation. "
                    f"Fix every error:\n{json.dumps(feedback)}")
    message = client.messages.create(
        model=MODEL, max_tokens=2000, system=SYSTEM,
        messages=[{"role": "user", "content": content}],
    )
    text = message.content[0].text.strip()
    if text.startswith(FENCE):  # some replies wrap JSON in a code fence
        text = text.strip("`").removeprefix("json").strip()
    try:
        return json.loads(text)
    except json.JSONDecodeError as exc:
        raise ValueError(str(exc)) from exc


if __name__ == "__main__":
    from edit_agent import main
    main(propose)

It was tested against anthropic 2.3.0 with the HTTP layer mocked, and has not been run against the live API, so run it once with a real key before relying on it. If you mock the SDK yourself, note that 2.x expects httpx2 clients.

4. Run it

To see the loop without a key, use a scripted stand-in whose first reply has two deliberate mistakes. Save this as demo_scripted.py:

from edit_agent import main

# Two replies a model might give: the first has two mistakes, the second fixes them.
REPLIES = iter([
    {"segments": [{"start": 0.0, "end": 1.5, "reason": "greeting"},
                  {"start": 2.6, "end": 4.2, "reason": "main point"},
                  {"start": 6.9, "end": 9.0, "reason": "closing"}]},
    {"segments": [{"start": 0.0, "end": 1.5, "reason": "greeting, drops the 'um'"},
                  {"start": 2.3, "end": 4.2, "reason": "main point"},
                  {"start": 6.9, "end": 7.8, "reason": "closing, drops the 'uh'"}]},
])

main(lambda brief, words, feedback: next(REPLIES))

Create a matching clip and run it:

ffmpeg -f lavfi -i testsrc=duration=8.5:size=640x360:rate=30 \
       -f lavfi -i sine=frequency=440:duration=8.5 \
       -shortest -c:v libx264 -pix_fmt yuv420p -c:a aac talk.mp4

python3 demo_scripted.py talk.mp4 transcript.json \
  "Remove filler words and pauses longer than one second." --yes

round 1: 2 error(s)
   {"segment": 1, "error": "cut at 2.6s falls inside the word 'today' (2.4-2.8s)"}
   {"segment": 2, "error": "6.9-9.0s is outside the media (0-8.50s)"}
round 2: 0 error(s)
0: 0.00-1.50s  'Hello everyone'  greeting, drops the 'um'
1: 2.30-4.20s  'today we talk editing'  main point
2: 6.90-7.80s  'it works'  closing, drops the 'uh'
rendered out.mp4 and verified it against the plan

Round 1 fails twice, for a cut inside "today" and a range past the end of the media. Round 2 passes, and the output is 4.30 seconds, matching the plan. With a real model, run python3 propose_anthropic.py talk.mp4 transcript.json "..." and drop --yes to approve the plan first.

What to add before production

• Long videos: this sends every word in one prompt. Pack the transcript instead, as video-use does.
• Visual checks: it never looks at a frame. Re-transcribe the output or screenshot stills, as the Fable 5 job did.
• Audio fades: video-use adds 30 ms fades at every cut. This cuts hard.
• Retries and logs: video-use caps its fix passes at three, and this script caps planning rounds at three. Log every reply and error.
• Safety: render to a new file, keep the plan and a reason per cut under version control, and keep a person between plan and render.

Where does agentic video editing still fall short?

• Speed is uneven. In the deck, a watchable 2:50 cut arrived within minutes, and the finished video took four days.
• Transcripts are imperfect. The transcript misheard the presenter's name, and the cuts still worked because the word timestamps held.
• Taste stayed human. Claude produced candidate color looks, and Thariq chose one.
• The evidence is selected and not independent. The Reelful write-up cautions that the founder picked the examples shown, and both Fable 5 jobs are described by Thariq, who works at the model's maker. The memories.ai paper names latency and limited fine control as open problems.

Common questions

Do I need to code?

For hosted editors, no. For video-use and Remotion you prompt in plain language but work in a terminal with a coding agent, and you should be able to read what it produces. Remotion's docs note that most coding agents require a paid subscription.

Is it free?

It depends on the design. video-use is MIT-licensed and videopython is Apache-2.0, but the model that drives them may cost money. Remotion calls itself source-available, not open source: free for individuals and companies of up to three people, with a company license for four or more, and its license page says the terms change slightly in Remotion 5.0. Tella's docs say its agent is included with paid plans and free trials. CoAnimator's free tier has no expiry and unlimited local renders, and its one-time licence adds 4K and watermark-free export. Check current pricing first.

Will it replace video editors?

No. I think the role's not gonna disappear, it's definitely gonna change. Invideo says the editor moves from performing every mechanical operation to briefing, reviewing and directing, and in the Fable 5 job a person set the creative direction.

What is the difference between an editing API and an editing agent?

An API or MCP server exposes operations such as cut, zoom or export. An agent decides which to call, checks the result and revises.

What I would ask for before adopting one

I would ask for one thing first: show me the edit as a file. After a revision, I want to open it and see what changed. If the only answer is a rendered video, or a timeline reachable only through the vendor's interface, I would treat the result as a draft I cannot audit. That does not make a hosted editor a bad choice. For a single clip you will never revise, its timeline may be all you need. The file matters once the edit needs a second version.

Then run a small test. Take a 60-second recording, ask the agent for a plan before it renders, approve it, render once, and ask for one small change, such as rewording a single title. If only that piece changes, the design holds your edit in a form you can work with. If the whole cut shifts, you learned that before committing a real project.

In the Fable 5 case, the part worth copying was not the model. It was that the finished edit was a set of files a person could open and read.