I Built a Local-First AI Video Editor That Runs in the Browser

Modern video editors are powerful, but many AI-assisted workflows require uploading media to a remote server.

I wanted to explore a different direction:

How much of an AI video editing workflow can run directly inside the browser?

The result is Timeline Studio, an open-source, local-first AI video editor with a multi-track timeline, browser-side AI inference, and MP4/WebM export.

• GitHub: https://github.com/MartinDelophy/ai-video-editor
• Live demo: https://video-editor.ai-creator.top/
• Demo video: https://youtu.be/mUXduGpBmwE

What is Timeline Studio?

Timeline Studio is a desktop-style video editor built with React and browser media APIs.

The editing experience is inspired by tools such as CapCut. It provides a visual timeline where users can arrange media, create overlays, generate voiceovers, edit captions, apply effects, and export a finished video.

Its defining constraint is that project media and supported AI workflows remain in the browser.

There is no editing backend responsible for processing or storing the user's project media.

Editing features

The editor currently supports:

• A contiguous main Visuals track
• Timed picture-in-picture overlays
• Captions, stickers, voiceovers, source audio, and music
• Timeline snapping and full-height alignment guides
• Clip splitting, duplication, deletion, and reordering
• Direct canvas movement, resizing, and rotation
• Shape masks, filters, visual effects, and animations
• Property-scoped and grouped transform keyframes
• Video playback-speed controls
• Junction-based transitions
• Portable editable .timeline project files
• MP4 and WebM export

The preview canvas and export renderer share the same composition geometry. This helps keep captions, masks, overlays, transforms, and visual effects consistent between editing and export.

AI voiceovers in the browser

Timeline Studio supports multiple browser-side text-to-speech engines.

The voice catalog includes:

• Chinese voices using Piper/VITS ONNX
• English voices using Kokoro 82M ONNX
• Piper voices for German, Spanish, French, Italian, and Brazilian Portuguese

The heavier engines are loaded only when needed. WebGPU is preferred where the model path supports it, with an explicit WASM fallback.

Generated audio can be placed directly on the voiceover timeline and linked to its matching caption.

Instead of combining an entire script into one large narration clip, Timeline Studio can generate voiceovers caption by caption. Each voice clip starts at the corresponding caption time, making individual lines easier to regenerate and rearrange.

Automatic captions with Whisper

Automatic captions run through Whisper small q8 ONNX in a browser worker.

The caption pipeline includes:

• Browser-side speech recognition
• Waveform-aware timestamp adjustment
• Conservative Chinese homophone correction
• Editable timed caption clips
• Shared preview and export typography
• Caption placement and style controls

I initially experimented with smaller Whisper variants, but they produced unstable results on Chinese audio. The current path favors predictable output over the smallest possible download.

Smart framing and background removal

The Smart Frame workflow uses:

• YOLOS tiny for subject detection
• MODNet for portrait matting

For video, the editor does not analyze only the first frame and reuse that result for the entire clip.

Instead, it builds a timestamped temporal analysis track. Preview, smart crop, caption avoidance, background removal, and export resolve the analysis result for the current source time.

Inference is precomputed with adaptive sampling, so ONNX models do not have to run inside the real-time playback or export loop.

Talking-avatar generation

Timeline Studio also contains an experimental but functional browser-side digital-human pipeline.

It combines:

• JoyVASA for audio-driven motion
• LivePortrait for neural portrait rendering
• WebGPU inference
• A 256px preview path
• A 512px quality path
• Sparse neural keyframes with temporal interpolation

This is one of the heaviest workflows in the project. The interface reports its progress honestly because high-resolution neural rendering can still be slow on current browser GPUs.

The implementation avoids presenting simulated lip-sync or hand-authored visemes as a neural talking-avatar result.

A separate deterministic export path

Real-time preview and high-quality export have different requirements.

During editing, the browser should preserve smooth native video playback. Continuously seeking a video element to follow delayed React state creates visible stuttering, so the active video remains on its native playback path.

Export uses a separate deterministic renderer.

For every output frame, it resolves the exact timeline timestamp and composes:

• Main visual media
• Picture-in-picture overlays
• Captions
• Stickers
• Masks
• Keyframes
• Filters and effects
• Animations
• Transitions

Audio is prepared and mixed separately, including voiceovers, music, separated source audio, and enabled embedded video audio.

The primary export path uses WebCodecs to encode the final MP4 or WebM. MediaRecorder remains available as a compatibility fallback.

This separation makes the editor preview responsive without sacrificing deterministic export timing.

What’s new in v0.4.0?

Version 0.4.0 introduces several important workflow improvements.

A real overlay timeline

Visual assets now have explicit drop intentions:

• Dropping onto the main sequence adds or reorders main visual content
• Dropping over existing main content can create a picture-in-picture overlay
• Main clips can be dragged down and converted into overlays
• Overlay clips can be promoted back to the main Visuals track

The conversion preserves media identity and clip duration.

More reliable video audio export

A video clip can play its embedded audio before the user separates it.

After audio separation, the editor mutes the embedded element and uses the derived source-audio track to prevent doubled playback.

If no separated audio exists during export, Timeline Studio can extract and map enabled embedded video audio on demand.

Clearer drag-and-drop behavior

The editor now communicates whether a drop will:

• Append to the main sequence
• Insert at a specific seam
• Create an overlay
• Add media to an existing timed track

This removes hidden pointer thresholds and makes the resulting timeline operation easier to predict.

A Codex-compatible editing skill

The repository now includes an edit-timeline-studio skill for agent-assisted editing.

It provides a structured way for an agent to:

• Inspect media and preserve an editing brief
• Describe edits using stable clip IDs and timestamps
• Validate declarative edit plans
• Operate the browser editor
• Verify timeline placement and export results
• Keep the editable project as the source of truth

The current browser-driven workflow is usable today. A fully versioned headless command runner remains a future automation layer.

Technical stack

The main project uses:

• React 19
• Vite
• ONNX Runtime Web
• WebGPU and WASM inference
• WebCodecs
• OfflineAudioContext
• MediaRecorder as an export fallback
• Service workers for application and model caching

Timeline Studio is also installable as a PWA.

Large models are loaded lazily and cached so repeat sessions do not need to download them again.

What I learned

Building a browser video editor is less about drawing frames and more about maintaining one consistent interpretation of time.

The timeline, native media playback, waveforms, keyframes, captions, source-time mapping, transitions, and export renderer must all agree about where a clip is and which source frame belongs at a given timestamp.

AI introduces another layer of complexity. Model output often needs to become editable timeline state rather than a one-time generated artifact.

For example:

• Speech recognition becomes movable caption clips
• Voice generation becomes timed audio linked to captions
• Subject detection becomes a temporal analysis track
• Portrait animation becomes replaceable visual media
• Audio separation becomes independently editable vocal and instrumental clips

Keeping these results editable is more valuable than returning only a final rendered file.

Try it

Timeline Studio is open source and available now:

• GitHub: https://github.com/MartinDelophy/ai-video-editor
• Live editor: https://video-editor.ai-creator.top/
• Demo video: https://youtu.be/mUXduGpBmwE

You can run it locally with Node.js 20 or newer:

bash
git clone https://github.com/MartinDelophy/ai-video-editor.git
cd ai-video-editor
npm install
npm run dev