Building an AI Video Editor: Reviewable Commands, State Tokens, and Browser Export

Disclosure: I work on Timeline Studio, the open-source editor discussed below. This article was prepared with AI assistance.

A generated video shot can be beautiful and still need another cut. The opening lingers, the narration arrives too late, or the sequence needs a different order.

We are building Timeline Studio around that next creative decision: a browser video editor where generated media and agent-assisted operations become part of an editable multitrack project.

Here are the architectural choices behind its command interface, with a small example you can follow in the repository.

Repository screenshot; interface details may differ in the current release.

1. Separate a source asset from a timeline clip

A source video can appear several times in one project, each time with a different trim, playback speed, transform, or effect. The asset identifies the media; the clip identifies one use of it.

When a user says “move the second shot to the beginning,” an agent first needs to resolve that description against the current sequence. Execution then targets a stable clip ID, rather than a filename or screen coordinate.

Timeline Studio exposes project, track, and clip inspection. An agent can read the overall structure and then request the relevant objects. Metadata inspection is distinct from understanding the picture or transcribing speech; those require their own processing routes.

2. Turn intent into an explicit plan

The local command interface accepts a versioned JSON plan. This example is illustrative: the revision, clip ID, and source range must match an inspected project.

{
  "schemaVersion": 1,
  "project": "/projects/input.timeline",
  "baseRevision": 12,
  "dryRun": false,
  "operations": [
    {
      "id": "trim-001",
      "type": "visual.trim",
      "clipId": "visual-123",
      "sourceIn": 2,
      "sourceOut": 6
    }
  ],
  "output": {
    "project": "/projects/output.timeline"
  }
}

The envelope states which project revision the plan assumes, identifies operations, and specifies a new output path. The operation states exactly which source interval to retain.

Planning interprets the request. Execution enforces the supported operations and project constraints. This boundary also lets non-conversational tools produce the same plans.

3. Validate meaning, not just JSON

Valid JSON can describe an impossible edit. The clip may no longer exist, the source range may be invalid, or the track may be locked.

We separate transport-shape validation from project-aware validation. A semantic dry run uses the shared command registry and state transformations to predict the actual result without writing an archive.

The local workflow is:

npm run agent -- project.inspect /projects/input.timeline
npm run agent -- project.diff /projects/plan.json
npm run agent -- project.run /projects/plan.json

The diff reports predicted changes to project fields, clips, timing, and duration. It gives review a concrete connection to execution. The local writer requires a new output path, preserving the input project.

4. Reject stale plans in a live editor

There is a concurrency problem even with a single human user:

1. The agent inspects the project.
2. The user splits or moves a clip.
3. The agent returns a plan based on the earlier state.

The browser interface uses a state token to bind preparation to the inspected state. A reviewed edit receives a preview ID. Applying it checks that the editor still matches the reviewed snapshot; otherwise, the agent must inspect again and replan.

Successful edits enter the editor's normal history mechanism, so the user can continue adjusting and undoing them.

5. Reuse editing rules across transports

The project provides a CLI, a bundled STDIO MCP adapter, and a browser WebMCP interface. Separate implementations of trimming, sorting, and link handling would invite inconsistencies.

The CLI and MCP adapter use the same application service. The browser adapter compiles reviewed operations into the shared command engine and commits through the editor.

Capability boundaries still matter. The browser has live media, inference jobs, and the full editor export workflow. The local renderer supports a defined portable subset and rejects unsupported compositions.

6. Keep generation and placement separate

Generation providers differ in authorization, job protocols, and output formats. Source-integrated adapters handle service-specific transport; a shared host receives downloaded media and commits it to My assets.

Completed generation does not automatically insert footage into the timeline. A creator may want to compare candidates first. This also prevents a long-running job from unexpectedly modifying an active edit.

Track submission, generation, download, and asset commit separately. A successful generation followed by a failed download needs a different recovery path from a failed generation. Stopping polling also does not by itself cancel a remote task.

7. Put preview and export on the same timeline clock

A source interval from 10 to 18 seconds, played at 2×, occupies four timeline seconds. For constant speed:

source time = source in + clip-local time × playback speed

Speed curves require accumulating the changing rate over time. Preview, splitting, thumbnails, and export need to agree on that mapping.

Timeline Studio uses native media paths for responsive playback and a separate deterministic offline rendering path for export, with WebCodecs-based composition and a compatibility fallback. Availability depends on the browser and device.

Resource scheduling matters too: thumbnail preparation is progressive, visible regions receive priority, and scrubbing pauses background refinement. Bounded processing queues keep throughput improvements from becoming unbounded memory growth.

What is available now?

ChatCut is usable, and the Anna edition includes video-generation integrations. Its setup and experience entry points are documented in the repository. Hosted generation is distinct from browser-local processing.

The command contract and browser adapter are useful starting points if you are building a similar system. The goal is to preserve an understandable project at every handoff: from request to plan, plan to reviewed operations, and editable state to rendered output.

Explore the Timeline Studio source and documentation on GitHub.

What would you put in an agent-facing editing contract first: timing, media identity, or transaction handling?