DEV Community

Vinamra Sareen
Vinamra Sareen

Posted on

Miraiclip: an open-source engine for building video editors in the browser

#webdev #typescript #opensource #showdev

I've been building Miraiclip — an open-source, framework-agnostic library for building video editors in the browser — and it's now on npm under MIT. This post covers what it is, the one design decision everything hangs on, and a few hard-won lessons from making video actually work in a browser.

The problem

Every browser video editor ends up rebuilding the same difficult parts: frame-accurate playback, an undo/redo system that never corrupts state, and exports that match what the preview showed. These are engine problems, not product problems — but there's no engine to reach for, so every team pays for them again.

Miraiclip is that engine. It is deliberately not an editor app. There's no timeline UI, no buttons — the core is headless: project state, a microsecond-precision timeline, and a full command history, with rendering delivered as a separate layer. You bring the UI (React, Vue, Svelte, vanilla — the core has zero UI dependencies).

One design decision: everything is a command

Instead of calling imperative methods that mutate state, you dispatch descriptive commands. Every command is deterministic, validated against a schema, serializable, and invertible.

import { createProject } from "@miraiclip/core";

const project = createProject({ width: 1920, height: 1080, fps: 30 });

project.dispatch({
  type: "asset/add",
  payload: { id: "intro", kind: "video", src: "/media/intro.mp4", durationUs: 12_000_000 },
});
project.dispatch({ type: "track/add", payload: { id: "video-1", kind: "video" } });
project.dispatch({
  type: "clip/add",
  payload: {
    kind: "video", id: "clip-1", trackId: "video-1", assetId: "intro",
    startUs: 0,            // timeline position in microseconds
    durationUs: 5_000_000, // 5 seconds
  },
});

// Batch commands into one undoable transaction
project.transaction(() => {
  project.dispatch({ type: "clip/split", payload: { clipId: "clip-1", atUs: 2_000_000, newClipId: "clip-1b" } });
  project.dispatch({ type: "clip/move", payload: { clipId: "clip-1", startUs: 1_000_000 } });
});

project.undo();
project.redo();
Enter fullscreen mode Exit fullscreen mode

Three things fall out of this decision without being bolted on:

Undo/redo — commands are invertible, so history is exact, replayable, and inspectable.

Collaboration — every state change is emitted as granular RFC-6902 JSON patches. Command and patch streams are the substrate for multiplayer editing:

project.events.on("patches", ({ patches }) => {
  // [{ op: "replace", path: "/clips/clip-1/startUs", value: 1000000 }]
});
Enter fullscreen mode Exit fullscreen mode

AI control — commands are plain descriptive data with a published catalog and JSON schemas. An LLM can read project state and generate valid edit sequences; the command catalog doubles as a tool definition. A human clicking a UI, an LLM planning an edit, and a sync layer replaying a collaborator's changes all speak the same language.

Playback and export

The rendering layer (@miraiclip/renderer) is a WebCodecs decode pipeline plus a WebGL compositor (PixiJS), with an audio-master clock:

import {
  createPixiBackend, createPlayer, exportProject,
  openMediabunnyDemuxer, createWebCodecsDecoderFactory,
  createWebAudioOutput, openMediabunnyAudio,
} from "@miraiclip/renderer";

const backend = await createPixiBackend({ canvas, width: 1280, height: 720 });
const player = createPlayer(project, {
  backend,
  openDemuxer: openMediabunnyDemuxer,
  createDecoder: createWebCodecsDecoderFactory({ maxOutputDimensionPx: 1920 }), // proxy preview
  audioOutput: createWebAudioOutput(),
  openAudio: openMediabunnyAudio,
});
player.play();

// Offline export, faster than realtime — same compositor as the preview
const bytes = await exportProject(project, { format: "mp4", quality: "high" });
Enter fullscreen mode Exit fullscreen mode

The export renders through the same compositor as the preview, so preview/export parity holds by construction rather than by testing luck. Output goes to MP4 or WebM, and it can stream straight into a file (showSaveFilePicker writable) so long exports don't accumulate in memory. There's also @miraiclip/server-export, which runs the identical pipeline in headless Chrome from Node:

miraiclip-export project.json --out final.mp4 --quality high
Enter fullscreen mode Exit fullscreen mode

Keyframe animations (cubic-bézier easings), effects (color adjust, blur, chroma key), transitions with equal-power audio crossfades, and karaoke-style captions with SRT/VTT and ASR word-timestamp import are built in — and they render identically in preview, browser export, and server export, because it's one compositor everywhere.

Things the browser made me learn

A few lessons that shaped the engine, in case you're building anything media-heavy:

"Frame-accurate" means presented frames, not scheduled ones. Callbacks tell you a frame was scheduled; only reading pixels back tells you what the user actually saw. Miraiclip's test fixtures encode each frame's index as its color, so tests assert on what's really on the canvas — that's how several seek and playback bugs were caught that callback-based tests would have blessed.

Export memory must be independent of timeline length. Mixing an hour of audio in one OfflineAudioContext buffer costs ~1.4 GB; encoding into an in-memory buffer costs the whole file. Miraiclip mixes audio in bounded sequential chunks interleaved with the frame walk and streams encoded output to disk — a demonstrated one-hour export (86,400 frames plus a full hour of audio) peaked at 165 MB of JS heap, running at 1.6× realtime on a dev laptop.

Trust nothing you can't verify with a second decoder. Exports are validated by a corpus that re-decodes every file with ffmpeg — an independent decoder — checking every frame's timestamp and pixels, audio gain, and A/V sync (measured offset: 0 ms at both ends of the file). A muxer bug can't hide by round-tripping through the encoder that produced it.

Status

Miraiclip is pre-1.0: the core model is stable in shape, but expect API changes between minor versions. Current packages: @miraiclip/core, @miraiclip/renderer, and @miraiclip/server-export. Chromium-family browsers are the target for rendering and export (WebCodecs-based); the core runs anywhere JS runs, including Node.

If you're building anything with video in the browser — an editor, a clip tool, automated video generation — I'd genuinely value your eyes on the API before it hardens.

Questions and issues welcome — and if you try it, I'd love to hear what you build.

Top comments (0)