Building Cinematic Depth in the Browser with Depth Anything V2 Small and WebGPU

Introduction

We recently added Cinematic Depth to Timeline Studio, an open-source browser video editor.

The feature runs Depth Anything V2 Small through WebGPU, analyzes images and video frames locally, and turns the resulting depth data into an adjustable depth-of-field effect. Source media never needs to leave the user's device.

• GitHub: MartinDelophy/ai-video-editor
• Live demo: Timeline Studio

This post focuses on the engineering work required to turn a depth-estimation model into a real editing capability—not just a demo that outputs a grayscale depth map.

Background blur is not depth of field

A typical background-blur feature uses person segmentation:

Person pixels     -> keep sharp
Everything else   -> apply one blur radius

That is useful for video calls, but it does not model the spatial structure of a scene.

A frame may contain leaves close to the camera, a person in the middle, furniture behind the person, and distant buildings. A person mask only answers “person or not.” It cannot tell us how far each region is from the lens.

Depth Anything V2 Small estimates continuous relative depth across the whole frame. That lets the editor:

• blur foreground and background differently;
• keep a configurable depth range sharp;
• move focus through the scene;
• build a foundation for animated rack focus.

Why the Small model?

A browser runtime has constraints that a GPU server does not:

• initial download size;
• GPU and system memory;
• WebGPU availability;
• initialization latency;
• the cost of processing many video frames;
• mobile hardware;
• cache and model-version management.

We chose the Q4F16 configuration of Depth Anything V2 Small as a practical balance.

The project already used @huggingface/transformers, so the conceptual initialization path is straightforward:

import { pipeline } from "@huggingface/transformers";

let depthEstimator;

export async function getDepthEstimator(
  modelId,
  onProgress
) {
  if (depthEstimator) return depthEstimator;

  depthEstimator = await pipeline(
    "depth-estimation",
    modelId,
    {
      device: "webgpu",
      dtype: "q4f16",
      progress_callback: onProgress,
    }
  );

  return depthEstimator;
}

The important production detail is reuse. We keep the initialized worker and WebGPU session alive instead of rebuilding the pipeline for every analysis.

Designing an editing feature, not a model demo

Cinematic Depth appears as the fifth card in the editor's Effects workspace.

The card:

• shows the untreated shot by default;
• switches to an unmistakable depth-of-field preview on hover or focus;
• auto-previews on touch devices without hover;
• opens a dedicated Effects inspector on desktop;
• opens a focused property drawer on mobile.

Users can adjust:

• focus distance;
• focus range;
• lens blur;
• bokeh highlights;
• Fast, Standard, or Fine analysis quality;
• enable, cancel, and reset.

Depth analysis and visual styling are separate stages. Moving the focus or blur sliders re-composites the existing depth data—it does not rerun the model.

Turning depth into blur

For each pixel, we calculate how far its depth is from the selected focus plane:

export function calculateBlurAmount({
  depth,
  focusDistance,
  focusRange,
  lensBlur,
}) {
  const distance = Math.abs(
    depth - focusDistance
  );

  return Math.max(
    0,
    distance - focusRange
  ) * lensBlur;
}

Pixels inside the focus range stay sharp. Blur increases as depth moves away from that range.

Canvas does not provide a single operation for assigning a different blur radius to every pixel. A practical implementation builds several blurred versions of the source and composites them with depth masks:

Original
├── light blur
├── medium blur
└── strong blur

The masks also need smoothing and feathering to reduce halos around depth discontinuities.

This is not a full physical lens simulation, but it produces a far more convincing spatial transition than a binary person/background mask.

One depth map is not enough for video

A still image needs one inference. Video changes over time.

If we reuse the first frame's depth for the entire clip, motion quickly causes the depth map and source frame to diverge. Running inference on every original frame, however, is too expensive for many browser devices.

We use quality-dependent temporal sampling:

Decode video
    ↓
Sample frames along the selected clip range
    ↓
Run depth estimation with WebGPU
    ↓
Store timestamped depth frames
    ↓
Reuse them during playback and export

The resulting data conceptually looks like this:

const depthFrames = [
  { time: 0.0, depth: depth0 },
  { time: 0.5, depth: depth1 },
  { time: 1.0, depth: depth2 },
];

At render time, the editor selects the depth frame matching the current clip-relative time. Interpolation between neighboring samples can make transitions smoother.

Clip-scoped depth caching

Depth inference is expensive. Re-compositing already computed depth is comparatively cheap.

These changes should therefore not invalidate analysis:

• focus distance;
• focus range;
• blur strength;
• bokeh settings;
• temporarily disabling the effect.

We invalidate the cache only when the source, analyzed range, quality, or model revision changes.

function createDepthCacheKey({
  assetId,
  clipStart,
  clipEnd,
  quality,
  modelRevision,
}) {
  return [
    assetId,
    clipStart,
    clipEnd,
    quality,
    modelRevision,
  ].join(":");
}

The cache is bound to the exact Visuals or Overlay clip. Every preview and export path carries an explicit clip ID so an Overlay effect cannot accidentally alter the main track.

Preview and export must match

AI editing features often look correct in the editor but change during export.

To avoid that, preview and export share:

• the same timestamped depth frames;
• the same clip identity;
• the same focus distance and range;
• the same lens blur;
• the same bokeh settings;
• the same enabled state.

The feature is integrated with:

• the main Visuals track;
• Overlay / picture-in-picture clips;
• real-time preview;
• project save and restore;
• deterministic export;
• compatibility export.

We reuse the same composition logic wherever possible instead of maintaining a separate “preview approximation.”

ModelScope and Hugging Face mirrors

Local inference still requires an initial model download.

To support users in different network environments, the runtime can prefer ModelScope for Chinese and domestic sessions, with Hugging Face as a fallback.

Chinese / domestic session
    ↓
Try ModelScope
    ↓
Fall back to Hugging Face

There is an important cache problem here: the same model has different provider URLs. If the URL becomes the cache identity, switching providers downloads identical files twice.

We use a provider-independent cache identity. The files on both mirrors are checksum-verified, and production URLs are pinned to immutable revisions rather than a mutable main branch.

That prevents:

• duplicate downloads after provider switching;
• sudden incompatibility after a repository update;
• mixing old code with new weights;
• deployments that cannot be reproduced.

Progress, cancellation, and useful errors

Video analysis needs more than a spinner.

The UI distinguishes:

1. model preparation;
2. WebGPU session initialization;
3. video-frame analysis;
4. completed frame count and percentage;
5. result finalization.

Cancel is also real. It stops further decode and inference work through an AbortController and worker messages instead of merely hiding a dialog.

Low-level messages such as Failed to fetch are converted into actionable, localized errors for:

• WebGPU unavailable;
• network failure;
• model download failure;
• user cancellation.

How long does a 10-second clip take?

There is no honest device-independent number.

Processing time depends on:

• GPU hardware;
• the browser's WebGPU implementation;
• video resolution;
• temporal sampling density;
• analysis quality;
• whether the model is cached;
• whether the WebGPU session is already initialized.

The first run includes download, initialization, and analysis. Later runs mostly need cache access and analysis.

The product optimizations that matter most are:

• download the model once;
• reuse the initialized session;
• sample video according to quality;
• reuse depth frames across preview and export;
• never rerun inference for a style-only change;
• expose real progress and cancellation.

What else can time-varying depth enable?

Cinematic Depth is only the first use of this data.

The same timestamped depth frames can support:

• 2.5D photo animation with foreground/background parallax;
• text and stickers placed correctly in front of or behind subjects;
• near-to-far spatial transitions;
• depth-aware Smart Frame;
• keyframed rack focus.

A reusable temporal depth representation becomes an editing primitive rather than a one-off effect.

Final thoughts

Running a model once in the browser can be a short demo. Turning it into an editing feature requires model delivery, caching, temporal mapping, cancellation, responsive UI, persistent state, and export consistency.

Timeline Studio integrates Depth Anything V2 Small as something users can analyze once, adjust repeatedly, save, restore, and export.

• GitHub: https://github.com/MartinDelophy/ai-video-editor
• Live demo: https://video-editor.ai-creator.top/

If you find the project useful, a GitHub star is appreciated. Issues and implementation feedback are welcome.

---

This feature is intended only for lawful editing of media the user is authorized to use. It must not be used for illegal, infringing, false, misleading, or identity-impersonation content, and AI-generated or edited output must not be presented as authentic footage. Users are responsible for misuse.