From Natural Language to an Editable Video: Inside Timeline Studio's Agent Skill

One instruction, one rendered video, and one editable .timeline project.

Repositories

Examples and reproducible workflows — start here

Timeline Studio Skills Handbook

The Handbook contains reference-video remakes, product promotions, multilingual localization, narrative marketing videos, and science explainers. Each case includes as much reproducible evidence as possible: the prompt, result video, editing decisions, and an editable .timeline project.

Main repository

MartinDelophy/ai-video-editor

The main repository contains the browser video editor, Agent Skill, timeline command protocol, local AI integrations, and build and deployment configuration.

Install the Skill with:

npx skills add MartinDelophy/ai-video-editor \
  --skill edit-timeline-studio

Why video production needs more than another generation API

Generative AI has made it easy to create text, images, speech, music, and captions independently. Producing one complete video is still a fragmented process.

A short video may require all of the following:

• inspecting and organizing source media;
• selecting useful shots;
• designing the narrative and pacing;
• generating or cleaning narration;
• creating and synchronizing captions;
• adding music, overlays, transitions, and effects;
• exporting and decoding the result;
• preserving a project that can still be edited later.

Most AI tools solve one item on this list. The creator still moves files between services and manually rebuilds timing relationships.

Timeline Studio's edit-timeline-studio Skill treats the entire process as an executable workflow. Its goal is not merely to produce an MP4. It also preserves an editable .timeline project that a human can reopen and refine.

1. Starting from a natural-language brief

The user describes the intended result rather than a sequence of UI operations:

Turn these product demo clips into a roughly 40-second vertical promo. Preserve the real product interaction, add English narration and captions, and deliver both the video and an editable project.

The Skill decomposes that request into a controlled sequence:

1. inspect the named assets;
2. analyze video, audio, speech, and text;
3. record editorial decisions;
4. build a declarative edit plan;
5. apply timeline operations;
6. run browser-local AI or rich editor features when required;
7. validate the project and rendered output.

Natural-language brief
        ↓
Media analysis → Editorial decisions → Edit plan
        ↓
Transactional timeline operations
        ↓
AI speech, captions, music, and effects
        ↓
Project validation → Video validation
        ↓
MP4/WebM + editable .timeline

The Skill is therefore more than a prompt template. It defines what the Agent must inspect, how it should execute the edit, and what evidence is required before the task can be called complete.

2. Analyze before cutting

A common automatic-editing failure is to start cutting before understanding the media.

Silence does not necessarily mean that a section is disposable. It may contain a product result, a useful reaction, intentional tension, or the preparation for an important action.

Depending on the task, Timeline Studio can inspect:

• duration, resolution, frame rate, and audio presence;
• representative frames;
• speech transcripts and OCR text;
• shot-change evidence;
• subject position, scale, and motion;
• global and subject-region optical flow;
• audio-energy changes;
• visual clarity.

The analysis is converted into a source-time decision record rather than immediately mutating the project:

{
  "assetId": "product-demo",
  "sourceStart": 12.8,
  "sourceEnd": 19.4,
  "decision": "shorten",
  "keepStart": 14.1,
  "keepEnd": 18.2,
  "reason": "Preserve the complete interaction and remove repeated setup",
  "confidence": 0.92,
  "protectSourceAudio": true
}

This makes keep, remove, shorten, and reorder decisions explainable and reviewable.

3. Turning the brief into declarative timeline commands

Once the plan is ready, the Agent uses stable project, track, asset, and clip identifiers wherever the command registry supports the requested operation.

npm run agent -- project.inspect /path/to/project.timeline
npm run agent -- project.diff /path/to/edit-plan.json
npm run agent -- project.run /path/to/edit-plan.json

• project.inspect reads the project, tracks, clips, media, and captions.
• project.diff validates the plan and calculates a non-writing semantic diff.
• project.run transactionally applies the validated plan.

A simplified edit plan looks like this:

{
  "version": 1,
  "projectRevision": 8,
  "operationId": "update-product-intro-v2",
  "operations": [
    {
      "type": "timed.move",
      "clipId": "voice-intro",
      "start": 2.5
    },
    {
      "type": "caption.update",
      "clipId": "caption-intro",
      "text": "Let AI handle the repetitive editing work.",
      "start": 2.5,
      "end": 5.8
    }
  ]
}

4. Making Agent edits safe

Video editing needs many of the same guarantees as other stateful systems. A plan may update visuals, captions, narration, and music together. A partial result is often unusable.

The command layer therefore uses several safeguards:

• Revision checks: reject a plan if the project changed after the plan was created.
• Preconditions: verify that clips, tracks, media, and expected state still exist.
• Transactions: apply one user-visible intent as a complete unit.
• Idempotency: use a stable operationId so retrying a request does not duplicate media or captions.
• Semantic diffs: show meaningful project changes before writing the new archive.

These mechanisms separate a reliable production Agent from a UI automation demo.

5. Two execution paths

Timeline Studio does not pretend that every editor feature is already a mature headless API. It uses two complementary paths.

Path	Typical work
Versioned command layer	Project inspection, media import, clip timing, captions, track operations, portable archive output
Browser editor	AI speech, automatic captions, advanced effects, digital humans, rich preview and full export

Stable operations run through the command registry. Features that have not yet moved into that registry remain available through the local or hosted browser editor.

After either path, the Skill reopens and verifies the project instead of assuming that a successful click or command means the edit is correct.

6. Why .timeline matters

Many AI video systems return only an MP4. That is a delivery file, not an editable source of truth.

Timeline Studio uses a portable .timeline archive:

project.timeline
├── project.json
└── media/
    ├── visual-001.mp4
    ├── visual-002.png
    ├── voice-001.wav
    └── music-001.wav

The project can preserve:

• the contiguous main Visuals track;
• timed picture-in-picture overlays;
• captions and stickers;
• voiceover, source audio, and music;
• transforms, masks, filters, animations, and keyframes;
• media identity and source-time mapping;
• canvas ratio and track state.

An Agent can produce a first edit, while a human can continue working in a familiar visual timeline. Changing one caption or replacing one shot does not require regenerating the entire video.

7. Captions must map to audible speech

Automatically generated videos often contain captions with missing, duplicated, or misaligned speech.

Timeline Studio applies a simple invariant:

If captions are enabled, every visible caption must map to exactly one audible speech clip for its full interval.

Transcribed captions bind to existing source dialogue. Agent-authored explanation or narration receives a generated or recorded voice clip instead of becoming silent text.

{
  "captionId": "caption-result",
  "audioClipId": "voice-result",
  "start": 18.4,
  "end": 22.8,
  "text": "One instruction can produce a project you can still edit."
}

Final verification checks caption boundaries, real audible output, duplicate narration, loudness differences, and unexpected stereo-channel offsets.

8. Browser-local AI

Timeline Studio follows a local-first architecture. Supported inference runs in the browser with technologies including WebGPU, ONNX Runtime Web, Web Workers, Cache Storage, and WebCodecs.

Current editor capabilities include:

• Whisper Small Q8 ONNX automatic captions;
• Piper/VITS Chinese voices and Kokoro English voices;
• multilingual browser TTS;
• Stable Audio 3 Small Q4 ONNX music generation;
• YOLOS Tiny subject detection;
• MODNet portrait matting;
• MI-GAN object removal;
• NanoVSR image and video enhancement;
• vocal and accompaniment separation;
• JoyVASA and LivePortrait digital-human workflows.

Models are loaded only when a feature needs them and are reused from browser caches. Owned Hugging Face and ModelScope mirrors can provide pinned model revisions while sharing a provider-independent cache identity where possible.

Local-first execution reduces remote inference cost and helps keep unpublished media on the creator's device.

9. Completion means more than “export succeeded”

A completed editing request normally produces both artifacts:

output/
├── result.mp4
└── result.timeline

Before delivery, the Skill verifies:

• Visuals continuity and clip ordering;
• transition and overlay timing;
• caption-to-speech relationships;
• the presence of real audible audio;
• output dimensions, duration, and container;
• full video decoding;
• the absence of repeated tails or stalled boundaries;
• successful reopening of the .timeline archive;
• archived media resolution and first-frame preview.

The render and editable project must both be usable. A single opaque output file is not considered sufficient for a complete editing task.

10. Current boundaries

The browser editor already provides multi-track editing, AI captions, speech, music, visual processing, and rich composition.

The command runner can inspect and modify .timeline projects and render its documented portable Visuals + Voiceover + Music subset. Rich captions, stickers, complex overlays, some effects, and AI generation may still require the browser editor.

The current architecture is best summarized as:

Stable command layer + browser compatibility layer

This boundary is intentional. The project does not describe UI automation as deterministic headless execution, and the command registry can expand without blocking users from accessing editor features today.

Conclusion

Timeline Studio's Agent Skill is not primarily about teaching an Agent to click through a video editor. It defines a production protocol for understanding media, recording editorial decisions, applying safe timeline mutations, and verifying the result.

Its key ideas are:

• multimodal media analysis;
• content-specific editing workflows;
• declarative timeline operations;
• revision checks, transactions, and idempotency;
• browser-local AI inference;
• portable, editable .timeline projects;
• validation of both the rendered video and the underlying project.

AI does not need to remove the creator from the process. It can remove the repetitive work while preserving human control over facts, taste, and final editorial decisions.

Explore the reproducible examples:

Timeline Studio Skills Handbook

Install, run, deploy, or contribute:

Timeline Studio main repository