Modern video editors are powerful, but AI-assisted workflows often begin with the same trade-off: upload your media to someone else's server, wait for processing, and accept that the editable state lives somewhere else.
I wanted to explore a different constraint:
How much of a serious AI video editing workflow can run entirely inside the browser?
The result is Timeline Studio, an MIT-licensed video editor built with React, browser media APIs, ONNX Runtime Web, WebGPU, WASM, and WebCodecs.
The interesting part was not adding another timeline UI. It was getting media playback, AI inference, editable timeline state, audio mixing, and deterministic export to agree on the same definition of time — without relying on an editing backend.
The architecture in one view
Timeline Studio separates interactive editing from expensive media and AI work:
React UI and timeline state
|
+-- Native media playback for responsive preview
|
+-- Web Workers
| +-- ONNX Runtime Web
| +-- WebGPU inference
| +-- WASM fallback
|
+-- OfflineAudioContext for audio mixing
|
+-- Deterministic composition renderer
+-- exact timeline timestamps
+-- Canvas composition
+-- WebCodecs encoding
+-- MP4 / WebM muxing
Large models are loaded only when a feature needs them. Model artifacts are revision-pinned and cached locally, while inference runs in long-lived workers so repeated operations do not rebuild the entire runtime.
The editor remains responsive because React owns the editing interface and declarative project state, not every decoded video frame.
One timeline, two playback paths
A real-time preview and a high-quality export have different requirements.
During editing, the active video element follows the browser's native playback path. Trying to continuously seek media elements from delayed React state produced visible stuttering, especially with multiple layers.
Export takes the opposite approach. It resolves every frame from an exact output timestamp. For each timestamp, the renderer determines which clips are active and composes:
- main visual media;
- picture-in-picture overlays;
- captions and stickers;
- masks, filters, and effects;
- keyframed transforms;
- animations and transitions.
This split gives the editor smooth playback without making the exported file depend on real-time UI performance.
Deterministic export with WebCodecs
MediaRecorder is useful when compatibility matters, but it records a real-time stream. If the tab stalls or the device cannot keep up, the output can inherit timing problems from the recording session.
Timeline Studio therefore treats WebCodecs as the primary export path.
For each output frame, the renderer:
- converts the frame index into an exact timeline timestamp;
- resolves every active visual and its source timestamp;
- draws the shared composition geometry to the export canvas;
- passes the frame to the selected video encoder;
- mixes audio separately and muxes it into the final container.
The same geometry rules are shared by preview and export. A caption, mask, overlay, or transform should not move merely because the output resolution changes.
MediaRecorder remains a compatibility fallback, but it is not allowed to silently change the requested format. Active exports also expose a real cancel path that stops further work, releases media resources, and never downloads a partial file.
Running ONNX models with WebGPU and WASM
The editor currently uses browser-side models for several workflows:
- Piper/VITS and Kokoro for multilingual voice generation;
- Whisper small q8 for automatic captions;
- YOLOS and MODNet for subject detection and portrait matting;
- vocal separation and AI music generation;
- JoyVASA and LivePortrait for talking-avatar generation.
WebGPU is preferred when a model path has been verified with it. WASM is an explicit fallback rather than a hidden change of engine.
A lesson from this work was that model availability is not the same as workflow reliability. A voice is not added to the catalog just because an ONNX file exists; it needs to pass generation, timeline insertion, and downloaded-WAV validation through the editor itself.
Heavy inference runs outside the UI thread. For model bundles containing several artifacts, downloads happen in parallel, while GPU sessions are created serially to avoid unreliable initialization and memory pressure. Once initialized, a worker stays alive for repeated generations on the same page.
Caching large model artifacts
Downloading a model again on every visit would make a browser AI editor impractical.
Timeline Studio uses a service worker and versioned browser caches for its application shell and large runtime assets. Model URLs are revision-pinned so a cached artifact has a stable identity. When hosting locations change, legacy cache keys can still be recognized to avoid forcing users to download identical weights again.
This changes the perceived workflow:
- the first run performs model setup;
- later runs reuse the initialized or cached assets;
- repeated generation is presented as generation, not another fake "model download."
The application is also installable as a PWA, so the editor shell can start independently of a full network round trip.
Audio is its own timeline problem
Video export is only half of the job. Voiceovers, music, separated source audio, and enabled embedded video audio all need to agree with clip ranges and source offsets.
Timeline Studio prepares audio independently with OfflineAudioContext. Each source is cropped and scheduled against the export range, then mixed before muxing.
This becomes particularly important for custom in/out exports. Cropping only the visuals is not enough: captions, overlays, voice, music, embedded audio, and SRT timing all have to be rebased to zero consistently.
Waveforms are derived from decoded peaks when available. While decoding or restoration is still in progress, clips render a stable low-emphasis placeholder instead of becoming blank and visually ambiguous.
AI output should become editable state
The most useful architectural decision was to avoid treating AI output as a final opaque artifact.
Instead:
- speech recognition becomes timed caption clips;
- generated speech becomes editable audio linked to a caption;
- subject detection becomes a timestamped analysis track;
- vocal separation becomes independent vocal and instrumental clips;
- avatar generation becomes replaceable visual media;
- AI music is added to the asset library instead of being inserted automatically.
This keeps AI operations reversible. Users can move, trim, replace, unlink, or regenerate individual results without repeating the entire workflow.
Making the timeline agent-editable
The repository also includes an edit-timeline-studio skill and a versioned command runner for tools such as Codex, Claude Code, Copilot, and Gemini CLI.
The command layer can inspect portable .timeline projects, validate revisioned edit plans, show field-level diffs, apply supported edits transactionally, and render a documented portable subset to H.264/AAC MP4.
For example:
npm run agent -- project.inspect /absolute/path/project.timeline
npm run agent -- project.diff /absolute/path/edit-plan.json
npm run agent -- project.run /absolute/path/edit-plan.json
npm run agent -- project.render /absolute/path/render-request.json
Dry runs, stable clip IDs, explicit timestamps, revision checks, and idempotent operation IDs matter here. An agent should not turn a user's editable project into an irreversible render or apply the same mutation twice after a retry.
What I learned
Building a browser video editor is less about drawing frames than maintaining one consistent interpretation of time.
The timeline, media elements, waveform data, captions, keyframes, source offsets, transitions, audio mixer, and export renderer must all answer the same questions:
- Which clips are active now?
- Which source frame or audio sample belongs here?
- Which values are interpolated at this timestamp?
- What should happen before the first keyframe and after the last one?
AI adds another layer, but the same principle applies: generated results become much more useful when they enter a deterministic, editable timeline model.
The browser platform is now capable of handling far more of this stack than I expected. WebCodecs, WebGPU, WASM, Workers, OfflineAudioContext, Cache Storage, and installable PWAs can support a substantial local-first media workflow — as long as their responsibilities remain clearly separated.
Try it
Timeline Studio is open source and actively developed:
You can run it locally with Node.js 20 or newer:
git clone https://github.com/MartinDelophy/ai-video-editor.git
cd ai-video-editor
npm install
npm run dev
If you work with browser media, WebCodecs, WebGPU, ONNX, timeline UX, or deterministic rendering, feedback and focused contributions are welcome.
Top comments (0)