DEV Community

MartinDelophy
MartinDelophy

Posted on

Building a Browser-Native AI Video Editor: Timeline Synchronization, WebGPU, and Worker Architecture

#webdev #ai #react #webgpu

Modern browsers can now do work that once required a native desktop application. WebGPU exposes GPU compute, WebAssembly brings mature runtimes to the web, WebCodecs provides lower-level media primitives, and Web Workers let us move expensive tasks away from the UI thread.

But getting an AI model to run once in a browser is very different from building a video editor that remains predictable during a long editing session.

I recently completed a substantial infrastructure update to Timeline Studio, a local-first browser AI video editor:

The release changed 37 files and added roughly 1,400 lines. The work focused on:

  • stable dragging, splitting, reordering, and cross-track movement;
  • synchronization between project time and media time;
  • reusable timeline frame data;
  • explicit high-performance WebGPU adapter selection;
  • persistent AI workers and reusable inference sessions;
  • centralized model caching through a service worker.

This article explains the engineering decisions behind that update.

1. A timeline is a constrained data model, not a row of draggable rectangles

A first timeline prototype can position clips with a simple pixel conversion:

const left = startTime * pixelsPerSecond;
const width = duration * pixelsPerSecond;
Enter fullscreen mode Exit fullscreen mode

Dragging appears equally straightforward:

const nextStartTime =
  originalStartTime + deltaX / pixelsPerSecond;
Enter fullscreen mode Exit fullscreen mode

A production clip, however, contains more than a visual position:

const segment = {
  id: "segment-001",
  trackId: "visual-track",
  startTime: 30,
  duration: 10,
  trimStart: 10,
  trimEnd: 20,
  sourceDuration: 60,
  playbackRate: 1,
  locked: false
};
Enter fullscreen mode Exit fullscreen mode

A single move may need to enforce all of these rules:

  • the clip cannot begin before zero;
  • it cannot exceed the project boundary;
  • its position may snap to other edit points;
  • a locked track cannot accept it;
  • it may not overlap another clip on the same track;
  • its media type must be compatible with the target track;
  • related caption and audio state must remain valid.

When each React component implements its own version of those rules, dragging, duplication, cutting, and reordering eventually disagree.

The update moves timeline decisions into a shared domain layer:

pointer coordinates
       ↓
timeline coordinates
       ↓
candidate start time
       ↓
bounds and snapping constraints
       ↓
track routing and collision checks
       ↓
atomic project-state update
Enter fullscreen mode Exit fullscreen mode

The UI collects input and renders a preview. Domain logic determines the final valid result.

2. New clips must not rearrange existing work

A common way to handle overlapping audio is to redistribute every clip until no overlap remains. The algorithm succeeds, but the user's earlier track arrangement changes unexpectedly.

The new rule is deliberately asymmetric:

A new clip may search for an available track, but existing clips keep their lanes.

function findTrackForNewAudio(newClip, tracks) {
  for (const track of tracks) {
    if (!hasOverlap(track.clips, newClip)) {
      return track.id;
    }
  }

  return createAudioTrack();
}
Enter fullscreen mode Exit fullscreen mode

Asset semantics also matter. AI-generated music belongs on the dedicated music track rather than whichever voice track happens to be empty:

function resolveAudioTrack(asset) {
  if (asset.type === "ai-music") {
    return MUSIC_TRACK_ID;
  }

  return findAvailableVoiceTrack(asset);
}
Enter fullscreen mode Exit fullscreen mode

A track is not merely a visual group. It may define mixing behavior, caption relationships, mute and solo behavior, volume defaults, and export rules.

Automatic routing should therefore preserve both user intent and media meaning.

3. Separate project time from media time

A video editor has at least two clocks:

  • Project time describes where a clip appears in the final composition.
  • Media time describes which position in the source file should be decoded.

Suppose seconds 10–20 of a source video are placed at seconds 30–40 of the project. At project time 33, the video element should display source time 13.

function getMediaTimeAtTimelineTime(segment, timelineTime) {
  const localTime = timelineTime - segment.startTime;
  const mediaTime =
    segment.trimStart +
    localTime * segment.playbackRate;

  return Math.max(
    segment.trimStart,
    Math.min(segment.trimEnd, mediaTime)
  );
}
Enter fullscreen mode Exit fullscreen mode

Keeping this transformation independent from the DOM makes it reusable for trimming, splitting, playback-rate changes, clip movement, and multiple clips referencing the same source asset.

4. Do not write video.currentTime on every frame

The obvious synchronization operation is:

video.currentTime = targetTime;
Enter fullscreen mode Exit fullscreen mode

Doing this continuously forces the browser to seek repeatedly. That can cause decoder churn, visual jitter, black frames, and unnecessary CPU use.

Instead, compare the current media position with the calculated target:

const drift = Math.abs(video.currentTime - targetTime);

if (isSeeking || drift > MAX_ALLOWED_DRIFT) {
  video.currentTime = targetTime;
}
Enter fullscreen mode Exit fullscreen mode

During normal playback, the media element advances on its own and small drift is tolerated. During timeline scrubbing, exact frame feedback matters more, so synchronization happens immediately.

Both paths are called “synchronization,” but they optimize for different outcomes:

  • playback optimizes for continuity;
  • scrubbing optimizes for precision.

5. Make timeline frames part of the media asset

Thumbnail frames are not just decoration. They help users identify content and find cut points.

If those frames live only in temporary component state, they are easy to lose after splitting, copying, or moving a clip. A generated video may even appear as a blank block or one stretched cover frame.

The asset now carries compact sampled frames:

const videoAsset = {
  id: "video-001",
  duration: 12,
  trackFrameDuration: 0.5,
  trackFrames: [
    { time: 0, image: "..." },
    { time: 0.5, image: "..." },
    { time: 1.0, image: "..." }
  ]
};
Enter fullscreen mode Exit fullscreen mode

The same data can be reused by:

  • the media-library card;
  • the main visuals track;
  • overlay tracks;
  • multiple clips created from a split;
  • browser-generated video assets.

Rendering code selects frames based on the clip's trim range and visual width without rewriting the source frame data.

6. WebGPU does not necessarily choose the fastest GPU

A typical WebGPU setup begins with:

const adapter =
  await navigator.gpu.requestAdapter();
Enter fullscreen mode Exit fullscreen mode

On a dual-GPU machine, the browser may prefer an integrated adapter to save power. That can be reasonable for ordinary UI rendering but expensive for ONNX Runtime and generative workloads.

For compute-heavy paths, the project now uses an explicit default:

const adapter =
  await navigator.gpu.requestAdapter({
    powerPreference: "high-performance"
  });
Enter fullscreen mode Exit fullscreen mode

An explicit caller override must still win:

function normalizeAdapterOptions(options = {}) {
  return {
    ...options,
    powerPreference:
      options.powerPreference ?? "high-performance"
  };
}
Enter fullscreen mode Exit fullscreen mode

This provides a high-performance default without breaking callers that intentionally request low-power.

The policy is shared across AI music, speech, face processing, video repair, and super-resolution workers.

7. Handling requestAdapter calls inside third-party runtimes

Not every adapter request is made by application code. A pinned runtime may internally call requestAdapter() without options.

When upgrading the dependency immediately would introduce compatibility risk, a narrowly scoped initialization wrapper can provide the missing default:

const originalRequestAdapter =
  navigator.gpu.requestAdapter.bind(navigator.gpu);

navigator.gpu.requestAdapter = (options = {}) =>
  originalRequestAdapter({
    powerPreference: "high-performance",
    ...options
  });
Enter fullscreen mode Exit fullscreen mode

Two details are essential:

  1. Keep the patch scoped to the relevant worker or initialization phase.
  2. Spread explicit options after the default so the caller retains control.

The original method should be restored after initialization. This is a compatibility strategy, not a preferred permanent API.

8. AI latency is more than inference time

The delay users experience usually contains several stages:

download artifacts
       ↓
write cache
       ↓
read model files
       ↓
create inference sessions
       ↓
preprocess inputs
       ↓
run inference
       ↓
postprocess outputs
Enter fullscreen mode Exit fullscreen mode

Optimizing only model execution may leave most of the perceived delay untouched.

Download independent artifacts in parallel 

const artifacts = await Promise.all(
  modelFiles.map(downloadModelFile)
);
Enter fullscreen mode Exit fullscreen mode

Create large GPU sessions serially

Initializing several large WebGPU sessions at once can create a sharp memory and GPU-resource peak. Parallel downloading combined with serial session creation is usually a safer balance.

Keep initialized workers alive

Terminating a worker after every generation discards the expensive model sessions:

first use
   ↓
start worker
   ↓
initialize models
   ↓
generation 1
   ↓
generation 2
   ↓
release when the page closes
Enter fullscreen mode Exit fullscreen mode

The UI should also distinguish model setup from content generation. A repeated generation should not look like another model download.

9. Give one component ownership of persistent model caching

If the page, inference workers, and service worker all write to Cache Storage, a large model can be stored more than once.

Timeline Studio makes the shared service worker the only persistent cache writer:

inference worker
       ↓ model request
service worker
       ├─ normalize provider URLs
       ├─ resolve immutable model identity
       ├─ preflight storage capacity
       ├─ evict stale model families
       └─ write Cache Storage
Enter fullscreen mode Exit fullscreen mode

Hugging Face and ModelScope may use different URLs for equivalent artifacts. Those URLs are normalized to one cache identity based on the model, immutable revision, and file path.

A cache write failure also does not necessarily mean the current inference must fail. If the artifact is already available in memory, the task can continue; only the next session may need to download it again.

That distinction separates a performance degradation from a functional failure.

10. Never expose “Failed to fetch” as the complete error

A browser's generic Failed to fetch message is nearly useless to an end user.

The application layer should distinguish at least:

  • no network connection;
  • an unavailable model mirror;
  • insufficient browser storage;
  • unavailable WebGPU support;
  • model initialization failure;
  • user cancellation. 
function toUserFacingError(error) {
  if (isNetworkError(error)) {
    return "The model server is unavailable. Check your connection and retry.";
  }

  if (isStorageError(error)) {
    return "Browser storage is full. This run will try an in-memory fallback.";
  }

  if (isAbortError(error)) {
    return "The operation was cancelled.";
  }

  return "The model could not be initialized.";
}
Enter fullscreen mode Exit fullscreen mode

A useful error should answer three questions:

  1. What failed?
  2. Can the current task continue?
  3. What can the user do next?

Resulting architecture

The refactored flow is easier to reason about:

user interaction
       ↓
timeline commands and constraints
       ↓
project state
       ├─ segment placement
       ├─ track routing
       ├─ caption relationships
       └─ sampled track frames
                ↓
media synchronization
       ├─ project time
       ├─ source-media time
       └─ drift correction
                ↓
AI workers
       ├─ model lifecycle
       ├─ WebGPU adapter policy
       └─ cancellation
                ↓
centralized model cache
Enter fullscreen mode Exit fullscreen mode

The most important outcome is not the number of new features. It is the clearer separation of responsibilities:

  • timeline components no longer invent their own placement rules;
  • media-time conversion is independent from UI state;
  • WebGPU, worker lifecycle, and model caching form shared infrastructure.

Validation

The release was checked with:

npm run check
Enter fullscreen mode Exit fullscreen mode

This runs ESLint, TypeScript checking, and the Vite production build. The release completed with:

  • zero ESLint errors;
  • successful TypeScript validation;
  • successful production build;
  • GitHub release v1.0.2;
  • successful production deployment.

Final takeaways

The difficult part of a browser-native AI video editor is not running a single model. It is keeping timeline state, media elements, React state, workers, GPU sessions, and local caches consistent throughout a real editing session.

The most reusable lessons from this update are:

  1. Treat the timeline as a constrained domain model.
  2. Do not let new assets silently rearrange existing work.
  3. Separate project time from source-media time.
  4. Use different synchronization policies for playback and scrubbing.
  5. Explicitly request a high-performance WebGPU adapter for compute workloads.
  6. Download artifacts in parallel but initialize large GPU sessions serially.
  7. Reuse initialized workers and inference sessions.
  8. Give the service worker exclusive ownership of persistent model caching.

Browsers are becoming local compute platforms, not just document viewers. Turning those capabilities into a dependable creative tool requires careful boundaries around time, state, storage, and GPU resources.

Project links

Top comments (1)

Collapse
 
talha_ramzan_3878156fea8c profile image
Talha Ramzan •

The asymmetric routing rule (new clips find space, existing clips never move) is the most transferable insight, auto-rearranging everything to resolve conflicts optimize for clean internal state at the cost of the user's mental model silently diverging from reality.

Playback-vs-scrubbing sync being two separate code paths instead of one tuned drift threshold is the sharper call, continuity and precision are different optimization targets, not different intensities of the same one.

Giving the service worker exclusive cache-writing ownership sidesteps a whole class of race-condition bugs by removing the distribution instead of coordinating around it.