DEV Community: MartinDelophy

I Built an AI Skill That Turns a URL or Raw Footage Into an Editable Video Project

MartinDelophy — Wed, 05 Aug 2026 11:56:32 +0000

AI video tools are getting better at producing output, but many of them still have the same limitation: they give you a finished MP4 and very little control over what happens next.

If one subtitle is too fast, one voiceover sentence sounds wrong, or one product shot needs a longer pause, the easiest option is often to regenerate the whole video.

I wanted a different workflow.

I recently shipped a major upgrade to Timeline Studio, an open-source, browser-based AI video editor. The new workflow can analyze a website, images, or raw footage, create a first cut, and export both:

a finished video ready to publish
an editable .timeline project for future revisions

The idea is simple: let AI handle the time-consuming first draft, while the creator keeps control of the final edit.

What the Skill can do

After installing the Timeline Studio Skill, you can describe the result you want in natural language.

For example:

Analyze this website and create a product introduction video.
Use a professional and energetic English female voice.
Target roughly 60 seconds in 16:9.
If login is required, ask me before continuing.
Export the final video and an editable .timeline project.

The Skill can then plan the content, create a narration, generate synchronized subtitles, select scenes, add background music, and apply visual emphasis.

It supports several editing scenarios:

talking-head and voice-driven videos
website and SaaS walkthroughs
product or brand promotion
image-based videos
long-form content condensation
tutorials
highlights and short-form clips

The editing strategy changes with the content instead of applying the same template to every project.

Website promotion is more than screen recording

For a website introduction, recording a browser session is not enough.

The system first explores the publicly accessible pages and tries to understand the product, information structure, and most important features. If the key experience requires an account, it asks the user for access instead of inventing conclusions from the landing page.

The resulting video can combine:

stable page recordings
smooth keyframed zooms
lines that guide the viewer's attention
a small number of accurately positioned highlights
subtitles and narration aligned with the visible action

We deliberately avoid drawing boxes around everything. Too many highlights make the video feel noisy and reduce their value.

Instead, the Skill chooses between three visual treatments:

Zoom for a core feature, result, or chart
Guide line for buttons, text, and reading direction
Frame for a clearly bounded area that genuinely needs emphasis

A typical move is a smooth zoom in, a short hold, and a controlled zoom out. The camera should feel intentional and stable rather than shaky.

Voiceover becomes the timing source

One of the biggest lessons from earlier versions was that forcing narration into an exact duration can make it sound rushed and unnatural.

The upgraded workflow generates the voiceover at a natural pace first. It then uses the real speech timing to determine:

caption boundaries
scene changes
zoom and highlight timing
music volume automation
ending space

A requested duration such as 60 seconds is treated as a target, not a reason to damage the delivery. A clear 54-second or 66-second video is often better than an unnaturally compressed 60-second one.

An editable `.timeline` v3 project

The most important technical change is the project format.

In .timeline v3, voiceover is stored as separate sentence-level audio clips rather than one long, difficult-to-edit file. Each caption can reference its corresponding voice clip.

A simplified representation looks like this:

{
  "caption": {
    "text": "Career planning should not depend on headlines alone.",
    "audioClipId": "voice-001"
  },
  "audioClip": {
    "id": "voice-001",
    "start": 0.8,
    "duration": 4.2
  }
}

This means you can replace one sentence, regenerate one voice clip, or adjust one subtitle without rebuilding the entire project.

The project also preserves scenes, captions, voice clips, music, overlays, effects, and animation settings. The MP4 solves today's publishing need; the project file solves tomorrow's revision.

Fixing playback loops at clip boundaries

The upgrade also addresses an editor playback issue where a segment could repeat near a clip boundary.

Individual media elements are not always reliable as the master timeline clock. Decode delays and small rounding differences can cause time to move backward or make two neighboring clips appear active at the same boundary.

Timeline Studio now uses a monotonic timeline wall clock as the master and synchronizes media against it. Clip activity is evaluated using half-open intervals:

[start, end)

That prevents the previous clip and next clip from both being selected at the exact same timestamp.

Install the Skill

You can install the current release with:

npx skills add MartinDelophy/ai-video-editor@v0.9.1

GitHub:

https://github.com/MartinDelophy/ai-video-editor

Live demo:

https://video-editor.ai-creator.top

The goal is to produce a strong first cut in the time it takes to finish a coffee, without turning the result into a black box.

AI performs the repetitive first-pass work. The creator still owns the timeline, the details, and the final decision.

Feedback and contributions are welcome.

Building Cinematic Depth in the Browser with Depth Anything V2 Small and WebGPU

MartinDelophy — Wed, 05 Aug 2026 03:08:09 +0000

Introduction

We recently added Cinematic Depth to Timeline Studio, an open-source browser video editor.

The feature runs Depth Anything V2 Small through WebGPU, analyzes images and video frames locally, and turns the resulting depth data into an adjustable depth-of-field effect. Source media never needs to leave the user's device.

GitHub: MartinDelophy/ai-video-editor
Live demo: Timeline Studio

This post focuses on the engineering work required to turn a depth-estimation model into a real editing capability—not just a demo that outputs a grayscale depth map.

Background blur is not depth of field

A typical background-blur feature uses person segmentation:

Person pixels     -> keep sharp
Everything else   -> apply one blur radius

That is useful for video calls, but it does not model the spatial structure of a scene.

A frame may contain leaves close to the camera, a person in the middle, furniture behind the person, and distant buildings. A person mask only answers “person or not.” It cannot tell us how far each region is from the lens.

Depth Anything V2 Small estimates continuous relative depth across the whole frame. That lets the editor:

blur foreground and background differently;
keep a configurable depth range sharp;
move focus through the scene;
build a foundation for animated rack focus.

Why the Small model?

A browser runtime has constraints that a GPU server does not:

initial download size;
GPU and system memory;
WebGPU availability;
initialization latency;
the cost of processing many video frames;
mobile hardware;
cache and model-version management.

We chose the Q4F16 configuration of Depth Anything V2 Small as a practical balance.

The project already used @huggingface/transformers, so the conceptual initialization path is straightforward:

import { pipeline } from "@huggingface/transformers";

let depthEstimator;

export async function getDepthEstimator(
  modelId,
  onProgress
) {
  if (depthEstimator) return depthEstimator;

  depthEstimator = await pipeline(
    "depth-estimation",
    modelId,
    {
      device: "webgpu",
      dtype: "q4f16",
      progress_callback: onProgress,
    }
  );

  return depthEstimator;
}

The important production detail is reuse. We keep the initialized worker and WebGPU session alive instead of rebuilding the pipeline for every analysis.

Designing an editing feature, not a model demo

Cinematic Depth appears as the fifth card in the editor's Effects workspace.

The card:

shows the untreated shot by default;
switches to an unmistakable depth-of-field preview on hover or focus;
auto-previews on touch devices without hover;
opens a dedicated Effects inspector on desktop;
opens a focused property drawer on mobile.

Users can adjust:

focus distance;
focus range;
lens blur;
bokeh highlights;
Fast, Standard, or Fine analysis quality;
enable, cancel, and reset.

Depth analysis and visual styling are separate stages. Moving the focus or blur sliders re-composites the existing depth data—it does not rerun the model.

Turning depth into blur

For each pixel, we calculate how far its depth is from the selected focus plane:

export function calculateBlurAmount({
  depth,
  focusDistance,
  focusRange,
  lensBlur,
}) {
  const distance = Math.abs(
    depth - focusDistance
  );

  return Math.max(
    0,
    distance - focusRange
  ) * lensBlur;
}

Pixels inside the focus range stay sharp. Blur increases as depth moves away from that range.

Canvas does not provide a single operation for assigning a different blur radius to every pixel. A practical implementation builds several blurred versions of the source and composites them with depth masks:

Original
├── light blur
├── medium blur
└── strong blur

The masks also need smoothing and feathering to reduce halos around depth discontinuities.

This is not a full physical lens simulation, but it produces a far more convincing spatial transition than a binary person/background mask.

One depth map is not enough for video

A still image needs one inference. Video changes over time.

If we reuse the first frame's depth for the entire clip, motion quickly causes the depth map and source frame to diverge. Running inference on every original frame, however, is too expensive for many browser devices.

We use quality-dependent temporal sampling:

Decode video
    ↓
Sample frames along the selected clip range
    ↓
Run depth estimation with WebGPU
    ↓
Store timestamped depth frames
    ↓
Reuse them during playback and export

The resulting data conceptually looks like this:

const depthFrames = [
  { time: 0.0, depth: depth0 },
  { time: 0.5, depth: depth1 },
  { time: 1.0, depth: depth2 },
];

At render time, the editor selects the depth frame matching the current clip-relative time. Interpolation between neighboring samples can make transitions smoother.

Clip-scoped depth caching

Depth inference is expensive. Re-compositing already computed depth is comparatively cheap.

These changes should therefore not invalidate analysis:

focus distance;
focus range;
blur strength;
bokeh settings;
temporarily disabling the effect.

We invalidate the cache only when the source, analyzed range, quality, or model revision changes.

function createDepthCacheKey({
  assetId,
  clipStart,
  clipEnd,
  quality,
  modelRevision,
}) {
  return [
    assetId,
    clipStart,
    clipEnd,
    quality,
    modelRevision,
  ].join(":");
}

The cache is bound to the exact Visuals or Overlay clip. Every preview and export path carries an explicit clip ID so an Overlay effect cannot accidentally alter the main track.

Preview and export must match

AI editing features often look correct in the editor but change during export.

To avoid that, preview and export share:

the same timestamped depth frames;
the same clip identity;
the same focus distance and range;
the same lens blur;
the same bokeh settings;
the same enabled state.

The feature is integrated with:

the main Visuals track;
Overlay / picture-in-picture clips;
real-time preview;
project save and restore;
deterministic export;
compatibility export.

We reuse the same composition logic wherever possible instead of maintaining a separate “preview approximation.”

ModelScope and Hugging Face mirrors

Local inference still requires an initial model download.

To support users in different network environments, the runtime can prefer ModelScope for Chinese and domestic sessions, with Hugging Face as a fallback.

Chinese / domestic session
    ↓
Try ModelScope
    ↓
Fall back to Hugging Face

There is an important cache problem here: the same model has different provider URLs. If the URL becomes the cache identity, switching providers downloads identical files twice.

We use a provider-independent cache identity. The files on both mirrors are checksum-verified, and production URLs are pinned to immutable revisions rather than a mutable main branch.

That prevents:

duplicate downloads after provider switching;
sudden incompatibility after a repository update;
mixing old code with new weights;
deployments that cannot be reproduced.

Progress, cancellation, and useful errors

Video analysis needs more than a spinner.

The UI distinguishes:

model preparation;
WebGPU session initialization;
video-frame analysis;
completed frame count and percentage;
result finalization.

Cancel is also real. It stops further decode and inference work through an AbortController and worker messages instead of merely hiding a dialog.

Low-level messages such as Failed to fetch are converted into actionable, localized errors for:

WebGPU unavailable;
network failure;
model download failure;
user cancellation.

How long does a 10-second clip take?

There is no honest device-independent number.

Processing time depends on:

GPU hardware;
the browser's WebGPU implementation;
video resolution;
temporal sampling density;
analysis quality;
whether the model is cached;
whether the WebGPU session is already initialized.

The first run includes download, initialization, and analysis. Later runs mostly need cache access and analysis.

The product optimizations that matter most are:

download the model once;
reuse the initialized session;
sample video according to quality;
reuse depth frames across preview and export;
never rerun inference for a style-only change;
expose real progress and cancellation.

What else can time-varying depth enable?

Cinematic Depth is only the first use of this data.

The same timestamped depth frames can support:

2.5D photo animation with foreground/background parallax;
text and stickers placed correctly in front of or behind subjects;
near-to-far spatial transitions;
depth-aware Smart Frame;
keyframed rack focus.

A reusable temporal depth representation becomes an editing primitive rather than a one-off effect.

Final thoughts

Running a model once in the browser can be a short demo. Turning it into an editing feature requires model delivery, caching, temporal mapping, cancellation, responsive UI, persistent state, and export consistency.

Timeline Studio integrates Depth Anything V2 Small as something users can analyze once, adjust repeatedly, save, restore, and export.

GitHub: https://github.com/MartinDelophy/ai-video-editor
Live demo: https://video-editor.ai-creator.top/

If you find the project useful, a GitHub star is appreciated. Issues and implementation feedback are welcome.

This feature is intended only for lawful editing of media the user is authorized to use. It must not be used for illegal, infringing, false, misleading, or identity-impersonation content, and AI-generated or edited output must not be presented as authentic footage. Users are responsible for misuse.

Why Browser-Based Multilingual TTS Kept Freezing at 86% — and How We Fixed It

MartinDelophy — Tue, 04 Aug 2026 03:24:26 +0000

Running text-to-speech entirely in the browser sounds simple: download an ONNX model, create an inference session, and synthesize audio without sending user text or media to a server.

In production, it is much harder.

While building multilingual voice generation for Timeline Studio, we repeatedly saw the same failure: the model-loading UI stopped at 86%, the Generate button stayed busy, and switching from Chinese to English, German, Korean, Thai, or Japanese made the issue more likely.

The most useful console message was:

Unable to cache file QuotaExceededError: Quota exceeded.

QuotaExceededError:
The operation failed because it would cause the application
to exceed its storage quota.

The progress bar was only the symptom. The real problem was the interaction between large model files, multiple TTS runtimes, browser storage quotas, service-worker caching, WebGPU initialization, and region-specific model mirrors.

This post explains the architecture changes that made the multilingual pipeline reliable.

Project: https://github.com/MartinDelophy/ai-video-editor
Live demo: https://video-editor.ai-creator.top/

The browser was doing more than the progress bar showed

A browser TTS model normally goes through several stages:

Download configuration, phonemizer, tokenizer, and vocabulary files.
Download one or more ONNX model files.
Store artifacts in Cache Storage.
Create an ONNX Runtime session.
Compile the graph for WebGPU or initialize WASM.
Run a warm-up inference.
Finally synthesize the requested speech.

Our original progress calculation mainly represented network downloads. If the files finished downloading but cache insertion or session creation failed, the UI retained the last reported value — often 86%.

So the browser was not necessarily still downloading anything. It had already moved into an unrepresented stage and thrown an exception before the state machine could reach either success or a useful error state.

The fix started by treating model setup as a real multi-stage operation:

Checking local model cache
Downloading model artifacts
Initializing the local inference engine
Preparing the selected voice
Generating speech

During initialization, the UI now says what is actually happening instead of pretending another file is still downloading.

One origin, many competing caches

Timeline Studio supports several browser-local voice runtimes because one model family is not the best choice for every language:

Language group	Runtime	Execution path
Chinese	Piper	WebGPU first, WASM fallback
English	Kokoro Q8	WASM
Selected European languages	Piper	WASM
Korean, Thai, Vietnamese, Russian	MMS	WASM
Japanese	Supertonic	WASM

All of these runtimes operate under the same browser origin. Cache Storage, IndexedDB, and service-worker caches therefore compete for the same site quota.

After a user tried several voices, the origin could contain:

Current model files
Older model revisions
Multiple quantization variants
Duplicate files from different mirrors
Service-worker response copies
Runtime-specific cache entries

Each individual cache looked reasonable. Together, they could exceed the browser's storage allowance.

Separate model identity from download URL

The most important caching change was to stop treating a model URL as the model's identity.

This is fragile:

const cacheKey = modelDownloadUrl;

The same immutable artifact can be served by ModelScope in China and Hugging Face elsewhere. If the full URL becomes the cache key, identical bytes from two providers occupy two independent cache entries.

Instead, we generate a provider-independent identity:

const cacheIdentity = [
  modelFamily,
  immutableRevision,
  language,
  voice,
  quantization,
].join(":");

For example:

kokoro:revision-20260804:en:female:q8

Both mirror URLs resolve to that same logical entry. The provider can change without forcing a redownload or duplicating hundreds of megabytes.

This also makes cache migration predictable: an artifact revision is explicit, and a new revision naturally receives a new identity.

Use regional mirrors without fragmenting the cache

Browser-local inference still needs a network connection the first time a voice is used.

For Chinese-language and domestic sessions, Timeline Studio tries the owned ModelScope mirror first. Other sessions prefer the owned Hugging Face repository. If the preferred source fails, the loader tries the fallback source automatically.

The simplified flow looks like this:

async function loadVoiceArtifact(artifact: VoiceArtifact) {
  const cached = await readSharedVoiceCache(artifact.cacheIdentity);

  if (cached) return cached;

  for (const source of getPreferredSources()) {
    try {
      const bytes = await downloadArtifact(source, artifact);
      await writeSharedVoiceCache(artifact.cacheIdentity, bytes);
      return bytes;
    } catch (error) {
      reportSourceFailure(source, error);
    }
  }

  throw new VoiceModelUnavailableError();
}

Every production artifact is pinned to an immutable provider revision. That prevents a remote main branch from silently changing model bytes or breaking the browser runtime.

The product also never displays a raw Failed to fetch message. Users receive a localized explanation that the model could not be downloaded and that the fallback source was attempted.

A smaller model can be the faster product decision

The English pipeline originally used a roughly 325 MB FP32 Kokoro model with a WebGPU-first path.

That configuration looked attractive in benchmarks, but it created several real-world problems:

Long first download
High Cache Storage pressure
Unpredictable WebGPU graph compilation
Driver-specific failures
More competition with other language models

We switched the English path to a roughly 92 MB Q8 model and a stable WASM execution provider:

const session = await ort.InferenceSession.create(modelBuffer, {
  executionProviders: ["wasm"],
  graphOptimizationLevel: "all",
});

For an editor, voice generation is usually an occasional operation, not a continuously saturated inference workload. A smaller quantized model that reliably loads on more devices creates a better user experience than a theoretically faster GPU path that frequently fails before inference begins.

The design goal became:

First generation succeeds
Repeat generation reuses the cache
Switching languages does not break existing voices
Mid-range devices remain supported
WebGPU has a bounded WASM fallback where appropriate

Evict stale voice models, not the entire application cache

Deleting every cache when storage is full is easy, but it punishes the user by removing the model they just downloaded.

Instead, the cache manager protects the active voice and removes stale voice artifacts first:

Determine the cache identity required by the current voice.
Mark that entry as protected.
Remove obsolete model revisions.
Evict least-recently-used inactive voice models.
Preserve application assets and the active voice.
Retry the cache write.
If persistent storage still fails, allow the current inference to continue from memory when possible.

A simplified quota check is:

async function ensureVoiceStorage(activeIdentity: string) {
  const estimate = await navigator.storage.estimate();

  if (!estimate.quota || !estimate.usage) return;

  const usageRatio = estimate.usage / estimate.quota;

  if (usageRatio >= 0.8) {
    await evictStaleVoiceModels({
      preserve: [activeIdentity],
      strategy: "least-recently-used",
    });
  }
}

This turns a hard quota failure into a recoverable resource-management event.

Give large model files one caching owner

Service workers are excellent for JavaScript bundles, styles, icons, and ordinary static resources. They become dangerous when they independently cache very large ONNX responses that are already managed by a model loader.

Cloning and caching the same response in both layers can silently double storage use.

We established a single-responsibility rule:

The voice model manager owns large model artifacts.
The service worker does not duplicate large ONNX files.
Normal application assets remain under service-worker control.
Every TTS runtime uses the shared voice-artifact manifest.

This made storage usage measurable and model cleanup deterministic.

Progress should be byte-weighted and stage-aware

A 5 KB configuration file and a 92 MB model should not contribute equally to progress.

The downloader now reports progress using actual bytes:

const progress = loadedBytes / totalBytes;
onProgress(Math.round(progress * 100));

When the network phase ends, the UI resets into the initialization phase rather than holding at an arbitrary download percentage.

Not every runtime exposes identical internal progress, so the application uses a shared high-level contract while allowing each adapter to provide the best signals it has. The UI remains honest even when graph compilation itself cannot provide byte-level progress.

Let the browser paint before heavy WASM work

There was another small but important issue.

React state updates are asynchronous. If we set the generation status and immediately begin synchronous or CPU-heavy WASM work, the main thread may not paint the new state. To the user, the page appears frozen before it ever shows a useful message.

We now yield one frame before starting inference:

setGenerationState({
  status: "generating",
  progress: 0,
});

await new Promise<void>((resolve) => {
  requestAnimationFrame(() => resolve());
});

await generateVoice();

This does not make inference faster, but it makes the product feel responsive because the browser can display the transition before entering the heavy task.

Hide runtime diversity behind one interface

A multilingual product should not force the editor UI to understand every model family.

Each adapter implements a common contract:

interface VoiceRuntime {
  prepare(options: VoiceOptions): Promise<void>;
  synthesize(text: string): Promise<AudioBuffer>;
  dispose(): Promise<void>;
}

The timeline, asset library, and export pipeline work with generated audio regardless of whether the source was Piper, Kokoro, MMS, or Supertonic.

That separation also makes it possible to change quantization, execution providers, or mirror routing without rewriting product-level editing features.

What we verified

After the changes, we tested:

First-time English model setup and repeated generation
Repeated German generation
Korean local inference
Thai model download and synthesis
Japanese Supertonic initialization
Chinese mirror preference and provider fallback
Automatic cleanup near the browser quota
Cache reuse after a page refresh

Repeated generation no longer presents itself as another full model download, and quota failures no longer leave the interface permanently stuck at 86%.

Lessons for browser AI applications

The biggest lesson is that a model running once in a local prototype is not the same as a reliable browser AI feature.

A production implementation should answer all of these questions:

Are model files pinned to immutable revisions?
Do regional mirrors share a cache identity?
Can the service worker duplicate large artifacts?
What happens when persistent storage is almost full?
Does progress represent real bytes and real stages?
Does the UI get a chance to paint before heavy inference?
Is there a stable fallback when WebGPU is unavailable?
Are unused language models eventually evicted?

The original “stuck at 86%” report looked like a progress-bar bug. In reality, it exposed an architectural problem spanning storage, networking, inference, and UI scheduling.

By introducing quantized models, shared artifact identities, regional mirror fallback, quota-aware eviction, stage-aware progress, and stable WASM paths, multilingual voice generation became far more predictable across browsers and regions.

If you are building local-first AI in the browser, treat model distribution and storage as first-class infrastructure. Successful inference is only the beginning; reliable recovery across devices, networks, and storage conditions is what turns it into a product.

Open-source project: https://github.com/MartinDelophy/ai-video-editor

Try it online: https://video-editor.ai-creator.top/

Building Semantic Optical Flow Tracking in the Browser with WebCodecs

MartinDelophy — Mon, 03 Aug 2026 06:04:28 +0000

Project links

GitHub: https://github.com/MartinDelophy/ai-video-editor

Live demo: https://video-editor.ai-creator.top/

Timeline Studio is a local-first AI video editor that runs in the browser. I recently added an experimental Optical Flow Tracking capability to it.

The goal was not to cover the frame with impressive-looking arrows. The feature first detects a person or object, calculates optical flow only inside that semantic region, aggregates the local vectors into a motion cohort, and accumulates the cohort into a visible trajectory.

When processing finishes, the browser renders a reusable WebM result and adds it to My assets. The generated result is a real editing asset rather than a temporary analysis preview.

Why raw optical flow was not enough

Optical flow describes pixel displacement between adjacent frames:

v = (deltaX, deltaY)

Drawing every vector gives us a familiar motion-field visualization, but those vectors do not understand the scene.

They may come from:

camera shake;
moving background textures;
subtitle changes;
lighting variation;
compression artifacts;
hair and clothing deformation.

Clustering all of those vectors can easily turn background texture into a supposed “motion group.”

So I changed the question from:

Which pixels moved?

To:

Which person or object moved, in which direction, and along what trajectory?

The processing pipeline

The browser-local pipeline looks like this:

Selected video clip
        ↓
Detect people and objects
        ↓
Create large semantic regions
        ↓
Calculate local optical flow inside each ROI
        ↓
Aggregate vectors into motion cohorts
        ↓
Accumulate trajectories over time
        ↓
Render on clear source frames
        ↓
Encode a WebM result
        ↓
Add the result to My assets

Frame extraction, model inference, flow calculation, Canvas rendering, and video encoding all happen in the browser. The source video does not need to be uploaded to an analysis server.

Detecting the subject before calculating flow

The first sampled frame acts as the semantic anchor. The implementation uses bounded local fallbacks:

NanoDet for lightweight person and object proposals;
MediaPipe segmentation to confirm a person region;
YOLOS as another detection fallback;
a conservative centered semantic prior when the model runtimes fail on an otherwise valid video.

A “group” does not require multiple people. One clearly visible person or one main object is already a valid semantic motion cohort.

Close-up footage also needs special handling. A person may occupy almost the complete frame, so rejecting every large detection box would incorrectly fail on normal portrait composition.

Representing a semantic motion cohort

Each detected subject becomes a large region that can be propagated through time.

const cohort = {
  id: "cohort-1",
  label: "person",
  box: {
    xmin: 0.2,
    ymin: 0.1,
    xmax: 0.8,
    ymax: 0.95
  },
  center: {
    x: 0.5,
    y: 0.525
  },
  dx: 0,
  dy: 0,
  confidence: 0,
  stability: 0
};

The box is normalized to the 0–1 range. This makes it possible to calculate motion on a small analysis frame and later map the result onto a higher-resolution rendering canvas.

Nearby people may be merged into a larger cohort when their regions and motion agree. A single principal object remains its own cohort.

Calculating flow inside the semantic ROI

The sampled video frames are converted to grayscale. A local block-matching pass searches for the best displacement between the current frame and the next one.

For a candidate displacement, the matching error can be described as:

E(dx, dy) = average(abs(frameA(pixel) - frameB(pixel + displacement)))

The displacement with the lowest error becomes a candidate motion vector.

Not every candidate is accepted. The implementation rejects vectors that are:

too small to represent meaningful motion;
too uncertain;
outside the semantic ROI;
too far from the cohort’s dominant direction;
produced by a poor block match.

The important rule is that background vectors cannot create a person or object cohort on their own.

Aggregating local vectors into one group direction

Inside a semantic region, the accepted local vectors are combined using their confidence as a weight:

cohortVector = sum(confidence[i] * vector[i]) / sum(confidence[i])

A simplified implementation looks like this:

function aggregateVectors(vectors) {
  let dx = 0;
  let dy = 0;
  let totalWeight = 0;

  for (const vector of vectors) {
    const weight = vector.confidence;

    dx += vector.dx * weight;
    dy += vector.dy * weight;
    totalWeight += weight;
  }

  if (!totalWeight) {
    return { dx: 0, dy: 0 };
  }

  return {
    dx: dx / totalWeight,
    dy: dy / totalWeight
  };
}

I also compare how consistently the local vectors agree with the group direction. That becomes the cohort’s stability score.

The UI exposes several useful diagnostics:

valid vector count;
motion cohort count;
dominant direction;
trajectory stability;
active detector;
processed frame count;
analysis and encoding progress.

These metrics make the experiment easier to understand and help identify weak tracks.

Turning frame-by-frame motion into a trajectory

One optical-flow step only explains the motion at that instant. To show movement from point A to point B, the cohort center is updated after each sampled frame.

const nextPoint = {
  x: previousPoint.x + cohortVector.dx,
  y: previousPoint.y + cohortVector.dy,
  time: frameTime
};

track.points.push(nextPoint);

The visible trail is a configurable window over the accumulated points:

const visiblePoints = track.points.slice(-trailLength);

This supports short motion tails as well as longer experiment-style trajectories.

Rendering the visualization

The result Canvas contains:

the original video frame;
a subtle analysis grid;
cyan local-flow arrows;
unlabeled cohort bounds;
distinct accumulated trails;
the main cohort direction;
an experiment timecode in the upper-right corner.

I intentionally removed the text label from the cohort boundary. On close-up footage, a “Person · 1” label covered the subject and made the generated video feel more like a debug screen than a finished experiment.

The cohort count and detector information still appear in the inspector, where they do not obscure the image.

A simplified arrow renderer uses the Canvas 2D API:

function drawArrow(ctx, startX, startY, endX, endY, color) {
  const angle = Math.atan2(endY - startY, endX - startX);
  const head = 5;

  ctx.strokeStyle = color;
  ctx.fillStyle = color;

  ctx.beginPath();
  ctx.moveTo(startX, startY);
  ctx.lineTo(endX, endY);
  ctx.stroke();

  ctx.beginPath();
  ctx.moveTo(endX, endY);
  ctx.lineTo(
    endX - head * Math.cos(angle - Math.PI / 6),
    endY - head * Math.sin(angle - Math.PI / 6)
  );
  ctx.lineTo(
    endX - head * Math.cos(angle + Math.PI / 6),
    endY - head * Math.sin(angle + Math.PI / 6)
  );
  ctx.closePath();
  ctx.fill();
}

Adding the timecode

A clip-relative timecode is burned into the upper-right corner of each generated frame.

const timecode = formatTime(frame.time);

ctx.font = "700 16px ui-monospace, monospace";

const padding = 10;
const width = ctx.measureText(timecode).width + padding * 2;
const x = canvas.width - width - 16;
const y = 16;

ctx.fillStyle = "rgba(3, 10, 13, 0.78)";
ctx.fillRect(x, y, width, 34);

ctx.fillStyle = "#55f3e1";
ctx.textAlign = "center";
ctx.textBaseline = "middle";
ctx.fillText(timecode, x + width / 2, y + 17);

Besides improving readability, this gives the output the feel of a completed motion-analysis experiment.

Low-resolution analysis, high-resolution rendering

The first version stored the 192-pixel-wide analysis frames and enlarged them during result generation. The motion vectors looked correct, but the underlying video became visibly blurry.

The fix was to separate the analysis and rendering paths.

Analysis path

resize the frame to a small working resolution;
convert it to grayscale;
calculate local flow;
update cohorts and trajectories.

Rendering path

retain clear samples from the source video;
cap the render width at 1280 pixels;
map analysis coordinates to the render Canvas;
redraw vectors, bounds, trails, and timecode;
encode the high-resolution result.

The coordinate mapping is straightforward:

const scaleX = renderWidth / analysisWidth;
const scaleY = renderHeight / analysisHeight;

const renderX = vector.x * scaleX;
const renderY = vector.y * scaleY;

This keeps the optical-flow calculation inexpensive without sacrificing the clarity of the generated asset.

Encoding the result with WebCodecs

After analysis, every sampled moment is rendered to a Canvas frame. The frames are then encoded as WebM using WebCodecs.

const resultBlob = await encodeFrames(
  renderedFrameBlobs,
  renderWidth,
  renderHeight,
  sampleRate,
  keyframeTimes,
  duration,
  {
    signal: abortController.signal,
    onProgress(value) {
      updateProgress(92 + Math.round(value * 8));
    }
  }
);

The same abort signal is checked during frame rendering and encoding, so cancellation stops further processing and releases the encoder resources.

Generating real timeline frames

A generated video should not become a blank rectangle when it is placed on the timeline. Compact timeline frames are therefore created from the rendered output frames.

const trackFrames = await createVideoTrackFramesFromBlobs(
  renderedFrameBlobs,
  {
    duration,
    width: renderWidth,
    height: renderHeight,
    signal
  }
);

The asset stores a matching track-frame duration:

const asset = {
  id: crypto.randomUUID(),
  type: "video",
  src: URL.createObjectURL(resultBlob),
  blob: resultBlob,
  duration,
  width: renderWidth,
  height: renderHeight,
  trackFrames,
  trackFrameDuration: duration,
  generatedBy: "optical-flow-tracking",
  diagnostics: {
    detector,
    sampleRate,
    vectors: summary.vectorCount,
    cohorts: summary.cohortCount,
    dominantAngle: summary.dominantAngle,
    stability: summary.stability
  }
};

Adding the result to My assets

When encoding finishes, the result is added to the media library and selected automatically.

setUserAssets(current => [asset, ...current]);
setSelectedLibraryAssetId(asset.id);
setActiveTool("media");
setMediaTab("mine");

It is deliberately not inserted into the timeline automatically. The user can inspect the result first, then decide whether it belongs on the main visual track or as an overlay.

A small browser benchmark

I tested the pipeline with a two-second, 852×480 video at a 4 fps sampling rate.

The result contained:

9 sampled analysis frames;
268 valid motion vectors;
1 semantic motion cohort;
a dominant direction of roughly 235 degrees;
a trajectory stability score of roughly 78%;
an 852×480 WebM output.

Because low-resolution analysis and clear-frame rendering are separated, the result preserves the source dimensions instead of enlarging the analysis frames.

Benefits and limitations of browser-local processing

The complete pipeline runs locally:

video frame extraction;
person and object detection;
person-region estimation;
optical-flow calculation;
cohort tracking;
Canvas rendering;
WebM encoding;
timeline thumbnail generation.

This provides several advantages:

the source video does not need to be uploaded;
there is no backend inference queue;
parameter changes can be tested immediately;
the result becomes an editing asset right away;
server-side video storage and processing costs are reduced.

There are also real limitations. Long clips, high sampling rates, and 4K sources increase memory use and processing time. Heavy motion blur, long occlusion, and scene cuts can reduce tracking stability. Browser support for WebCodecs and local model runtimes also varies.

Conclusion

The most important change was not a more complicated arrow renderer. It was giving optical flow a semantic boundary and a complete product workflow.

Person/object detection
  +
Semantic ROI
  +
Local optical flow
  +
Cohort vector aggregation
  +
Trajectory accumulation
  +
High-resolution Canvas rendering
  +
WebCodecs encoding
  +
Reusable media asset

Three decisions made the largest difference:

Do not create people or object groups from background vectors.
Separate low-resolution analysis from high-resolution output rendering.
Treat analysis as complete only after it becomes a reusable video asset.

The result feels less like a raw computer-vision demo and more like a finished browser-based motion experiment that can continue through the normal editing workflow.

If you are interested in browser AI, WebCodecs, video editing, or computer vision, take a look at the repository:

GitHub: https://github.com/MartinDelophy/ai-video-editor

Building Browser-Local Video Face Swapping: Lessons from a WebGPU Inference Pipeline

MartinDelophy — Fri, 31 Jul 2026 08:10:51 +0000

Video face swapping is usually associated with cloud GPUs: upload a video, process it frame by frame on a server, and download the result.

Modern browser APIs make another architecture possible. WebGPU, WebCodecs, Web Workers, and ONNX Runtime Web can move a meaningful part of the pipeline onto the user's device. Source media does not have to be uploaded to an inference server, models can be cached locally, and completed results can go straight into an editable video project.

Once we built the pipeline, however, model inference turned out to be only part of the problem. Pixel conversion, cross-thread transfers, GPU initialization, target tracking, mask cleanup, memory ownership, cancellation, and reproducible benchmarking all had a direct impact on whether the feature felt usable.

This article focuses on those engineering details rather than the UI.

The implementation discussed here is part of the open-source Timeline Studio project.

Responsible-use notice

This article is a technical discussion of browser image processing and machine-learning inference, not legal advice. Use only face images and videos for which you have clear authorization from the depicted people and relevant rights holders. Do not use face synthesis for illegal, infringing, deceptive, misleading, harassing, fraudulent, or identity-abuse purposes. Do not present generated output as authentic footage or a factual record. Label AI-altered content when appropriate, and comply with applicable laws, contracts, media licenses, and platform policies.

1. Follow one frame through the entire pipeline

A decoded frame travels through roughly the following sequence:

VideoFrame / Canvas
    ↓
RGBA Uint8ClampedArray
    ↓
NCHW Float32Array
    ↓
ONNX Tensor
    ↓
Generated RGB and Alpha Mask
    ↓
Canvas compositing
    ↓
WebP intermediate frame
    ↓
WebM encoding

Every arrow costs time and memory.

SCRFD and MobileFaceSwap expect NCHW tensors, while pixels read from a Canvas are interleaved RGBA values. Before inference, the three color channels must be separated:

const plane = width * height;
const tensor = new Float32Array(plane * 3);

for (let i = 0; i < plane; i += 1) {
  tensor[i] = normalize(rgba[i * 4]);
  tensor[plane + i] = normalize(rgba[i * 4 + 1]);
  tensor[plane * 2 + i] = normalize(rgba[i * 4 + 2]);
}

This loop looks harmless in isolation. In a video pipeline it runs repeatedly, alongside Canvas readback, Tensor allocation, and worker communication. Together, these operations can consume substantial CPU time and memory bandwidth.

That leads to an important rule: choose model input sizes based on the cost of the entire data path, not model accuracy alone.

2. Do not send a full 1080p frame through the generator

Face detection and face generation solve different problems, so they should not use the same resolution.

Stage	Input size	Purpose
SCRFD detection	640×640	Locate faces and landmarks across the complete frame
Identity extraction	112×112	Extract a source identity representation that is relatively pose-independent
Face generation	224×224	Generate the new identity in the target pose
Optical flow	Long edge at most 720px	Track five facial landmarks at lower cost
Final composition	Original video resolution	Preserve the background and source detail

Detection needs the complete image. A 640×640 analysis frame helps retain smaller faces. Generation only needs an aligned face crop, so it stays at 224×224.

Sending a complete 1080p frame into the generator would spend most of its compute on background pixels that never change. Detecting globally, aligning the face, and generating only the region of interest is one of the main reasons this workload can run in a browser.

3. Download models in parallel, initialize WebGPU sessions serially

The first run has two expensive phases: downloading the model files and creating ONNX sessions. They benefit from different scheduling strategies.

Model downloads are independent, so they can run concurrently. Session creation, however, may involve:

parsing the ONNX graph;
graph optimization and operator fusion;
WebGPU shader generation;
pipeline compilation;
weight upload;
GPU buffer allocation.

Creating several sessions at once can produce a burst of shader compilation and GPU allocation. On resource-constrained devices, this increases startup jitter and peak memory use, and may even contribute to a lost GPU device.

We use parallel download followed by serial initialization:

const [
  detectorBuffer,
  identityBuffer,
  conditionerBuffer,
  generatorBuffer,
] = await Promise.all(modelDownloads);

const detector = await createSession(detectorBuffer);
const identity = await createSession(identityBuffer);
const conditioner = await createSession(conditionerBuffer);
const generator = await createSession(generatorBuffer);

This preserves network concurrency without concentrating GPU initialization into the same moment.

4. Transfer buffers instead of cloning them

Detection and generation should not block the UI thread, so frames and tensors move through Web Workers.

If an ArrayBuffer is posted without a transfer list, the browser may use structured cloning. A single 640×640 Float32 RGB tensor is already about 4.69 MB:

640 × 640 × 3 × 4 bytes ≈ 4.69 MB

Copying that buffer at every detection anchor quickly creates memory-bandwidth and garbage-collection pressure.

Transfer ownership instead:

worker.postMessage(
  {
    type: "detect",
    pixels: tensor.buffer,
  },
  [tensor.buffer],
);

After the call, the original thread's buffer is detached and the worker owns it. Generated RGB output and alpha masks can return through the same mechanism.

Transferables do not make inference itself faster. They remove avoidable copies and short-lived large objects, which matters just as much in a continuous video workload.

5. Optical flow is useful only when you can reject bad tracks

Running full face detection on every frame is expensive. A practical pipeline detects anchor frames and propagates landmarks between them with Lucas–Kanade optical flow.

Optical flow estimates local pixel movement; it does not guarantee that the match is correct. We therefore use a forward-backward consistency check.

Given a point p_t in the previous frame, forward tracking produces:

p_{t+1}=F(I_t,I_{t+1},p_t)

Track that result backward:

\hat{p_t}=F(I_{t+1},I_t,p_{t+1})

The forward-backward error is:

e_{fb}=\lVert \hat{p_t}-p_t \rVert_2

A stable track should return close to the original point. In our case, a frame is accepted only when at least four of the five landmarks remain valid and their average error stays below a threshold.

This check helps reject:

incorrect matches during fast movement;
hands or objects covering the face;
motion blur;
a subject leaving the frame;
abrupt lighting changes;
landmarks drifting onto background texture.

Optical flow is still limited to short propagation windows. SCRFD runs again at fixed intervals to prevent error from accumulating indefinitely.

6. The highest-confidence face is not always the target

Selecting the highest-confidence detection on each frame works poorly in multi-person footage. A new person may enter closer to the camera and receive a better detector score, even though they are not the editing target.

Target selection becomes a temporal matching problem. Candidate scoring can consider:

distance from the previous target center;
change in bounding-box area;
current detector confidence;
distance from the frame center on the first frame.

One simplified score is:

S=w_cC-w_dD-w_aA

Here, C is detector confidence, D is center distance, A is the area change, and the w values are weights.

The first frame favors a confident, reasonably large face near the center. Later frames give more weight to spatial and scale continuity.

This is not full face re-identification, but it is substantially more stable than picking the highest-confidence box on every frame. When a match is not trustworthy, keeping the original frame is safer than applying the effect to the wrong person.

7. Traditional image processing still matters after generation

Mask morphology

The generator's mask may contain holes, spikes, or discontinuous edges. Using it directly can make the blend boundary flicker.

A typical cleanup sequence is:

Raw mask
  ↓
Threshold
  ↓
Dilation: close local gaps
  ↓
Erosion: remove outer spikes
  ↓
Additional erosion: pull the blend region inward
  ↓
Box blur: create a smooth alpha
  ↓
Boundary safety mask: remove crop-box edges

Dilation and erosion can use a separable sliding-window implementation: filter horizontally, then vertically.

A direct two-dimensional morphology operation with radius r is roughly:

O(W \times H \times r^2)

A separable sliding-window implementation can approach:

O(W \times H)

The neural network may run on the GPU, but conventional algorithms that execute on every frame still deserve careful optimization.

Bounded color matching

When the source identity and target video have different color temperatures, the generated face can look disconnected from the forehead or neck.

Within the valid mask, we calculate the mean and standard deviation of the generated and target pixels, then apply a per-channel correction:

I'=\frac{\sigma_t}{\sigma_s}(I-\mu_s)+\mu_t

Unbounded correction may amplify noise or produce extreme colors under unusual lighting. Clamp both scale and offset:

const scale = clamp(targetStd / sourceStd, 0.78, 1.22);
const shift = clamp(targetMean - sourceMean * scale, -0.12, 0.12);

The corrected result is blended with the original output rather than replacing it completely. This reduces skin-tone discontinuities without discarding texture already reconstructed by the generator.

8. A model URL is part of its cache identity

Loading production models from a mutable path is risky:

repository/resolve/main/model.onnx

The URL can remain unchanged while the remote content changes. That can lead to:

different inference results from the same frontend version;
stale browser cache entries alongside new server files;
regressions with no identifiable model version;
temporarily inconsistent mirrors.

Production URLs should point to immutable revisions:

repository/resolve/<immutable-revision>/model.onnx

We also record the expected file size, SHA-256 hash, purpose, license, and input/output tensor definitions.

After download, validate size or hash before creating a session. Otherwise, a CDN error page or partial response may be passed to the ONNX loader as if it were a model.

9. Memory ownership is harder than one inference timing

A browser video task may hold all of these at once:

decoded frames;
Canvas pixels;
Float32 input tensors;
ONNX output tensors;
grayscale optical-flow images;
intermediate frame blobs;
encoder buffers.

Relying only on JavaScript garbage collection can produce steady memory growth after a relatively small number of frames. Several resource types need explicit cleanup:

tensor.dispose?.();
bitmap.close();
mat.delete();
input.dispose();
URL.revokeObjectURL(url);
worker.terminate();

ONNX tensors, ImageBitmap, OpenCV matrices, and object URLs belong to different runtimes and have different lifetime rules.

OpenCV.js deserves particular attention: Mat instances allocate from a WebAssembly heap. Losing the JavaScript reference does not guarantee prompt release of the underlying memory, so delete() must be called explicitly.

10. Cancellation must reach the deepest stage

Closing a progress dialog is not cancellation. A real cancel signal must propagate through:

model downloads;
video-frame reading;
face detection;
optical-flow tracking;
per-frame generation;
intermediate compression;
final video encoding.

The main thread can own an AbortController and send a request-scoped cancel message to the worker:

controller.abort();

worker.postMessage({
  type: "cancel",
  requestId,
});

The worker checks cancellation during download loops, before and after inference, between frames, and when switching requests. A canceled task does not continue into encoding and never adds a partial result to the asset library.

Without this propagation, the UI may report that a task ended while the GPU and CPU continue working in the background.

11. Report cold and warm performance separately

"Completed in seconds" is not a reproducible benchmark.

A browser AI performance report should capture at least:

Device:
GPU:
Operating system:
Browser and version:
WebGPU adapter:
Video codec:
Video resolution:
Video duration:
Output FPS:
Detection-anchor FPS:
First run:
Model initialization time:
Per-frame processing time:
Generator cumulative inference time:
Encoding time:
Peak memory:

It should also separate two scenarios.

Cold start

Cold start includes model download, file validation, ONNX session creation, shader compilation, source-identity extraction, video processing, and encoding. It measures distribution and device initialization as much as inference.

Warm start

Warm start assumes that models are cached and sessions and source-identity weights can be reused. It measures video decoding, anchor detection, optical flow, generation, post-processing, and encoding.

Warm-start performance is closer to the experience of editing several clips on the same page. Cold and warm numbers should not be mixed, and a report should not publish only the faster one.

12. Local processing does not remove consent and safety obligations

Keeping inference in the browser can reduce the need to upload source media to a remote inference service. It does not remove the responsibilities attached to face images and synthetic output.

Before releasing or operating this kind of feature, verify that:

the depicted people and rights holders authorized the intended use;
photos, videos, model weights, and sample code are used under valid licenses;
identifiable face images and embeddings are handled lawfully and securely;
generated output cannot easily be mistaken for real footage, news, or a person's actual behavior;
AI alteration is disclosed in a manner appropriate to the use case;
minors, public figures, and third parties are not used without authorization;
the workflow is not being used for fraud, impersonation, harassment, defamation, or sexual abuse;
users can cancel processing and remove generated results.

The safest failure mode is to emit no effect when target identity cannot be maintained reliably.

Closing thoughts

Browser-local video face swapping is not a matter of dropping an ONNX file into a web page. Moving from a demo to a repeatable workflow requires at least four kinds of engineering:

Bound the computation: run models only at the required resolution and on the required ROI.
Schedule resources deliberately: combine parallel downloads, serial session initialization, and transferable buffers.
Measure result confidence: reject bad optical-flow tracks, temporal identity switches, and poor masks.
Manage the runtime: pin model versions, release heterogeneous resources, and propagate cancellation.

WebGPU provides the compute, but the surrounding system determines whether that compute becomes a stable user experience.

The same lessons apply to browser-local person segmentation, super-resolution, object tracking, video restoration, and style transfer.

You can explore the implementation in the Timeline Studio repository.

Disclosure: AI assistance was used for translation and editorial restructuring of this article. The technical content is based on the linked project implementation.

Building Browser-Only Object Outlines with Detection, Prompted Segmentation, and Optical Flow

MartinDelophy — Thu, 30 Jul 2026 08:02:23 +0000

Introduction

I recently added an object-outline effect to Timeline Studio, an open-source browser-based video editor. The feature can isolate objects such as shoes, products, or toys and render a reusable outline around them—without uploading the source image or video to a server.

The complete implementation is available on GitHub:

https://github.com/MartinDelophy/ai-video-editor

This post focuses on three engineering decisions:

separating object detection from pixel-level segmentation;
keeping the same object identity across video frames;
rejecting unreliable masks instead of propagating bad results.

The processing pipeline

A conventional edge detector is not enough. It also picks up background texture, shadows, and internal details. I split the workflow into distinct stages:

Detect candidate objects
        ↓
Select and lock the editing target
        ↓
Create a prompt point inside the target
        ↓
Generate a pixel-level mask
        ↓
Validate mask quality
        ↓
Cache alpha and render the outline

The detector answers where the object is. Prompted segmentation answers which pixels belong to it. Keeping these responsibilities separate makes target selection more stable in crowded scenes.

Run inference outside the UI thread

Model inference on the main thread can freeze React controls and timeline interaction. Detection runs in a Web Worker, while the UI matches asynchronous responses by request ID.

worker.postMessage({
  requestId,
  type: "detect",
  blob,
  scoreThreshold: 0.24,
});

The ONNX Runtime session prefers WebGPU and falls back to WASM when GPU initialization is unavailable or fails:

async function createSession(modelBytes) {
  if (navigator.gpu) {
    try {
      return await ort.InferenceSession.create(modelBytes, {
        executionProviders: ["webgpu"],
        graphOptimizationLevel: "all",
      });
    } catch (error) {
      console.warn("WebGPU failed; falling back to WASM");
    }
  }

  return ort.InferenceSession.create(modelBytes, {
    executionProviders: ["wasm"],
    graphOptimizationLevel: "all",
  });
}

This keeps the fast path on supported devices without making WebGPU a hard requirement.

Selecting and locking the target

Choosing the highest-confidence detection on every frame can make the effect jump to another object. I rank candidates using confidence, area, distance from the center, overlap with the previous target, and class continuity.

function calculateRank(item, previous) {
  const area = getArea(item.box);
  const centered = getCenterScore(item.box);
  const previousIoU = boxIoU(item.box, previous?.box);
  const identityBonus = previous?.label === item.label ? 0.5 : 0;

  return (
    item.score * 0.55 +
    Math.min(area, 0.25) +
    centered * 0.2 +
    previousIoU * 1.2 +
    identityBonus
  );
}

For video, overlap with the previously accepted box receives a larger weight. A new object entering the frame should not silently become the editing target.

Prompted segmentation and alpha generation

Once the target is locked, I convert its center into a normalized prompt point and pass it to the segmentation model.

const result = segmenter.segment(image, {
  keypoint: {
    x: clamp(point.x, 0, 1),
    y: clamp(point.y, 0, 1),
  },
});

const probabilities = result.confidenceMasks.at(-1)
  .getAsFloat32Array();

const alpha = new Uint8ClampedArray(probabilities.length);
for (let i = 0; i < probabilities.length; i += 1) {
  const value = clamp(probabilities[i], 0, 1);
  alpha[i] = value >= threshold ? Math.round(value * 255) : 0;
}

The raw output is never trusted immediately. I validate mask area, connected components, target-box overflow, and changes relative to the previous accepted frame. A mask that includes background or another object is rejected.

Sparse anchors plus optical flow

Running detection and segmentation on every video frame is expensive in a browser. Full analysis runs only on sparse anchor frames; optical flow propagates the alpha between anchors.

const needsAnchor =
  index === 0 ||
  index - lastAnchorIndex >= anchorInterval ||
  !trackedMaskBox;

if (needsAnchor) {
  currentAlpha = await analyzeObjectAnchor(frame);
  lastAnchorIndex = index;
} else {
  currentAlpha = await propagateAlphaWithFlow(
    previousFrame,
    currentFrame,
    currentAlpha,
  );
}

If the propagated area changes implausibly, tracking is discarded and detection starts again:

const areaRatio = nextArea / previousArea;
if (!nextBox || areaRatio < 0.35 || areaRatio > 2.8) {
  currentAlpha = null;
  trackedMaskBox = null;
}

Failing closed matters here. Showing no effect for one uncertain frame is safer than legitimizing a wrong mask and carrying it forward.

Separate AI analysis from style rendering

Analysis is expensive, but changing color, width, glow, or material should feel immediate. I store the analyzed alpha separately from the effect style:

const objectOutline = {
  targetKind: "object",
  outline: {
    color: "#f3efe4",
    width: 12,
    opacity: 1,
    glow: 0.35,
  },
  material: {
    id: "paper",
    edgeDensity: 0.5,
    shadowDepth: 0.32,
  },
};

Style edits recompose the cached alpha instead of rerunning the models. Preview and export consume the same alpha and effect state, which reduces visual differences between the editor and the final video.

Takeaways

A browser-native object-outline tool is not a single-model feature. It is a pipeline built from:

Worker-based inference;
WebGPU with a WASM fallback;
object detection plus prompted segmentation;
temporal target locking;
sparse anchors and optical flow;
conservative quality gates;
separate analysis and rendering state.

Together, these pieces make image and video outlines editable while keeping the original media on the user's device.

Explore the implementation and try the editor on GitHub:

https://github.com/MartinDelophy/ai-video-editor

Building Stable Video Portrait Outlines in the Browser with MODNet, SlimSAM, MediaPipe, and Optical Flow

MartinDelophy — Wed, 29 Jul 2026 09:58:29 +0000

A portrait outline effect looks simple: find a person and draw a line around them. In real video, however, the problem quickly expands into subject identity, fine matting, temporal stability, occlusion, model latency, and memory pressure.

This article walks through the portrait-outline pipeline currently being developed for Timeline Studio, an open-source, local-first video editor that runs directly in the browser. The implementation combines YOLOS tiny, MODNet, MediaPipe, SlimSAM, and OpenCV Farneback optical flow. Source footage stays on the user's device during analysis and preview.

GitHub: https://github.com/MartinDelophy/ai-video-editor
Live editor: https://video-editor.ai-creator.top/

Why not run MODNet on every frame?

The first prototype was straightforward: run portrait matting on every sampled frame, extract the alpha boundary, and render an outline. It worked for still frames, but an editor-quality implementation exposed four problems:

Full inference at 25 or 30 FPS is expensive in a browser.
Independent predictions make hair and clothing edges flicker.
Nearby people, sofas, and cushions can leak into the portrait mask.
Style changes should not require the AI pipeline to run again.

The solution was to split the job into subject locking, high-quality anchor matting, fast subject gating, guarded segmentation recovery, and temporal propagation.

Pipeline overview

Video decoding
  ↓
YOLOS tiny detects and locks one person
  ↓
MODNet produces an alpha inside the person ROI
  ↓
MediaPipe constrains the subject and removes spill
  ↓
SlimSAM repairs difficult anchors only when needed
  ↓
Farneback optical flow propagates alpha between anchors
  ↓
Area, position, and motion quality checks
  ↓
SVG filters render a material-aware outline

Each component has one primary responsibility:

Component	Responsibility
YOLOS tiny	Decide which person is the effect target
MODNet	Produce a soft portrait alpha with fine edges
MediaPipe	Refresh the person region quickly and suppress spill
SlimSAM	Recover difficult segmentation anchors
Farneback optical flow	Propagate accepted alpha masks through time

1. Lock the target person with YOLOS tiny

Portrait matting answers which pixels look like a person, but it does not necessarily answer which person the user wants in a crowded scene.

The pipeline starts with YOLOS tiny and ranks person detections using confidence, box area, body completeness, and the previous tracked location. Once a primary person is selected, the downstream MODNet, MediaPipe, and SlimSAM stages operate around that person's ROI.

This identity-first step is important when the main person is touching a sofa, sitting beside another person, or surrounded by background people.

2. Use MODNet as the alpha-quality baseline

A binary segmentation mask is usually not enough for a polished outline. Hair, clothing, and motion-blurred boundaries contain partially transparent pixels. If those values are collapsed to only zero and one, the rendered outline becomes jagged and unstable.

Timeline Studio runs MODNet on an expanded person ROI rather than the full source frame. On browsers with WebGPU, the worker uses the WebGPU path; otherwise it falls back to quantized WASM.

Processing an ROI lowers the amount of work while preserving more useful resolution around the target person.

3. Use MediaPipe as a fast soft gate

MediaPipe person segmentation is fast and mobile-friendly, but it does not replace MODNet for every fine-edge case. Instead, the pipeline uses it as a fast subject constraint:

refresh the approximate person region between expensive anchors,
remove sofa or background spill from the MODNet alpha,
reduce contamination from nearby people.

The MediaPipe confidence mask is slightly dilated and blurred before it is applied to MODNet:

fusedAlpha[i] = Math.round(modnetAlpha[i] * mediaPipeGate[i] / 255);

A soft gate is deliberate. A hard binary intersection would easily remove hair strands, fingers, and translucent clothing edges. MediaPipe also has a GPU-to-CPU fallback, while the rest of the pipeline continues if MediaPipe is unavailable.

4. Run SlimSAM only as a guarded fallback

SlimSAM is not part of every frame. It is invoked only when an anchor fails quality validation—for example, when MODNet produces an incomplete person, the connected component no longer matches the detector box, or the mask includes a large spill region.

The SlimSAM request includes:

the YOLOS person box,
a positive point inside the selected person,
negative points derived from other detected people,
negative points derived from suspicious MODNet spill.

The current worker pins Xenova/slimsam-77-uniform to a fixed revision. It uses FP16 on WebGPU and quantized WASM when WebGPU is not available.

Depending on the frame, the SlimSAM mask can be used as the selected subject or as a soft gate over MODNet. That preserves MODNet's smooth alpha edges while benefiting from SAM's prompt-guided subject separation.

5. Connect frames with Farneback optical flow

Instead of running a heavy model on every sampled frame, an OpenCV.js worker calculates Farneback dense optical flow and warps the previous alpha toward the current frame.

Optical flow is limited to short temporal windows. The propagated mask is never trusted blindly. The pipeline monitors:

sudden mask-area expansion or collapse,
large position changes relative to tracking history,
abnormal mean motion,
ROI fallback to the full frame,
loss of the selected person.

When a check fails, the propagated alpha is rejected and a new model anchor is scheduled early. This prevents one tracking error from contaminating the rest of the clip.

Current analysis profile

The editor currently calls the portrait pipeline with a profile similar to this:

const analysis = await analyzePersonOutlineVideo({
  src: videoSource,
  duration,
  flowFps: 8,
  anchorFps: 1 / 3.5,
  maxSamples: 360,
  maxDimension: 360,
  onProgress,
  onSample,
});

That means:

effect samples are generated at roughly 8 FPS,
routine model anchors are approximately 3.5 seconds apart,
the analysis frame's longest side is capped near 360 pixels,
one run produces at most 360 samples,
degraded tracking can trigger an earlier anchor.

A 30 FPS source does not need 30 full model executions per second for an editor preview. Preview and export resolve the cached analysis sample for the current source timestamp.

Optimizing more than inference

Model inference is only one part of browser video processing. Seeking, decoding, canvas reads, worker copies, and transparent-image encoding can all become bottlenecks.

The current implementation includes several engineering optimizations:

Prefer sequential WebCodecs decoding.
Fall back to precise video seeking for unsupported codecs.
Warm YOLOS/MODNet, MediaPipe, and the optical-flow worker in parallel.
Transfer RGBA and alpha buffers with transferable ArrayBuffer objects.
Keep initialized models and workers alive for reuse.
Commit each completed sample to the UI instead of waiting for the whole video.

The last point matters for product experience. Users can see the current cutout, outline, timeline position, processed-frame count, and tracking state advance together. The analysis can also be canceled without leaving a hidden task running.

Turning alpha into material-aware outlines

The simplest outside border is:

Outline = Dilate(Alpha) - Alpha

Timeline Studio builds on that ring with an SVG filter pipeline using feMorphology, feComposite, feTurbulence, feDisplacementMap, texture images, feBlend, feDropShadow, and feGaussianBlur.

The current material presets are intentionally different from one another:

crumpled paper,
heavy frost,
layered light halo,
liquid chrome,
impasto paint,
ink bleed.

Users can tune outline color, width, opacity, grain, texture scale, irregularity, relief, diffusion, ring count, ring spacing, and glow radius.

The important architectural detail is that the person alpha is cached separately from the visual style. Changing a paper outline to chrome does not rerun YOLOS, MODNet, or optical flow.

Delivering browser AI models reliably

The first model download is another practical challenge for local browser AI. Timeline Studio mirrors model artifacts across Hugging Face and ModelScope. Chinese and domestic sessions can prefer ModelScope, with Hugging Face as a fallback, while equivalent files share the same cache identity.

ModelScope: https://www.modelscope.cn/models/martindelophy/timeline-studio-onnx-models
Hugging Face: https://huggingface.co/haixin/timeline-studio-onnx-models

Models are downloaded on first use and reused from browser storage afterward.

Run the project locally

Timeline Studio is open source under the MIT License:

git clone https://github.com/MartinDelophy/ai-video-editor.git
cd ai-video-editor
npm install
npm run dev

Use Node.js 20+ and a modern Chromium-based browser. A WebGPU-capable device improves the MODNet and SlimSAM paths.

For a production build:

npm run build
npm run preview

Current limitations and next steps

The hybrid pipeline improves speed and temporal consistency, but difficult cases remain:

people fully crossing in front of each other,
a target leaving and re-entering the frame,
fast shot changes,
severe motion blur,
low contrast between hair and background,
complex overlap between a person and a held object,
long clips on low-memory mobile devices.

Next steps include shot-boundary detection, stronger person-ID continuity, manual subject selection, a larger repeatable test set, and extending the same effect system to object outlines.

The portrait-outline workspace is still under active development, so the repository README, releases, and roadmap remain the source of truth for current availability.

Why Valid SVG Is Not Good SVG: Improving Gemini Nano for Browser-Local Vector Design

MartinDelophy — Wed, 29 Jul 2026 02:31:46 +0000

A parser can tell me whether an SVG is valid and safe. It cannot tell me whether the design is good.

That distinction has become one of the most interesting problems in Timeline Studio, the open-source, local-first video editor I am building for the browser.

Timeline Studio can generate editable vector graphics with Gemini Nano through Chrome’s built-in Prompt API. The whole workflow stays in the browser: the request is detected and translated locally, the model generates SVG, and the result is sanitized before it is added to the user’s asset library.

The pipeline works. But “works” is not the same as “produces a useful design.”

The current browser-local pipeline

The generation flow is intentionally small:

Detect the actual language of the request.
Translate non-English input to English in the browser.
Ask Gemini Nano for one standard SVG document.
Extract the first complete SVG from Markdown fences or surrounding prose.
Sanitize elements, attributes, links, scripts, and external resources.
Normalize the result into a 1200 × 1200 viewBox.
Add the vector to My Assets without inserting it into the timeline automatically.

The shipped code is roughly this simple:

const raw = await session.prompt(buildVectorDesignPrompt(englishRequest));
const vectorXml = extractVectorXml(raw);
const vectorBody = sanitizeGeneratedVectorXml(vectorXml, scope);

The parser is deliberately tolerant. Gemini Nano may return a clean SVG, wrap it in a Markdown code fence, add a sentence before it, or use an older custom wrapper. The extractor handles those cases before the sanitizer applies the real safety boundary.

What validation catches

The current validation layer is good at rejecting structural and security problems:

missing or malformed SVG markup
scripts and event handlers
unsafe or unsupported elements
external URLs and embedded remote content
attributes that do not belong in an editable vector asset
output that cannot be normalized into the editor’s coordinate system

This is necessary. Generated markup should never be trusted just because it came from a local model.

But a safe SVG can still be a bad SVG.

What validation misses

Visual quality failures are much harder to express as pass or fail:

the artwork is clipped even though the viewBox is valid
the composition technically matches the request but feels generic
a supposedly transparent asset contains a full-canvas background rectangle
the illustration is almost empty or uses too few meaningful shapes
dozens of tiny shapes make the vector difficult to edit
groups exist, but their IDs and hierarchy are not useful
typography, spacing, or contrast makes the asset unusable in a video
translation preserves the words but loses the design intent

None of these necessarily produces invalid XML. A sanitizer can accept all of them.

This is the part I want to improve next.

A reproducible SVG quality benchmark

Before changing prompts or adding repair logic, I want a small benchmark that makes regressions visible.

The first version should contain at least 24 prompts across:

icons and simple symbols
lower thirds and title graphics
charts and infographic elements
callouts, labels, and badges
frames and mask-like compositions
abstract motion-graphics assets
at least four non-English requests

Each result can be reviewed with a compact human rubric:

Request fidelity — does the image represent what was asked for?
Composition — are spacing, balance, and hierarchy usable?
Polish — does it look intentional rather than merely generated?
Editability — are groups and shapes structured for later editing?
Safety and transparency — does it stay within the editor’s SVG rules?

The goal is not to invent one magical quality score. It is to create a repeatable set of prompts and examples so that a prompt change can be compared with the previous behavior.

Deterministic checks before another model call

Some visual failures can be approximated cheaply without running a second inference:

empty or nearly empty visible bounds
shapes extending far outside the viewBox
a large opaque element covering almost the entire canvas
extreme coordinate values
excessive shape or filter complexity
duplicate, missing, or unhelpful group IDs
content clustered into a tiny fraction of the canvas

These checks will never replace visual review, but they can produce a machine-readable report. More importantly, they can explain a failure instead of returning a vague “generation failed” message.

One bounded local repair pass

The next experiment is a single optional repair attempt. This is a proposal, not a shipped feature yet:

const report = analyzeSvgQuality(vectorXml);

if (report.repairable && experiments.svgRepair) {
  const repaired = await repairOnce({
    request: englishRequest,
    svg: vectorXml,
    report,
  });

  return sanitizeGeneratedVectorXml(
    extractVectorXml(repaired),
    scope,
  );
}

The repair prompt would receive the original English request, the generated SVG, and a short list of deterministic warnings. It would keep the same element and safety constraints and would run at most once.

Why only once?

Because an unlimited generate-check-repair loop is unpredictable on a user’s laptop. Browser-local software has to respect memory, battery, model latency, cancellation, and weaker hardware. A bounded repair step is easier to understand, test, and disable.

The repaired result must also pass through the exact same extractor and sanitizer. Repair is not a security exception.

Why the local-first constraint matters

Using a hosted vision model could make evaluation and repair easier, but it would change the product boundary. Timeline Studio is exploring what a capable editor can do without uploading project media or design prompts to a backend.

That creates useful engineering constraints:

no server-side SVG cleanup service
no hidden cloud fallback
no unbounded retries
clear errors when browser AI or local translation is unavailable
generated assets remain editable and are never silently placed on the timeline

The constraint is not just about privacy. It forces the quality system to be understandable and reproducible.

I am looking for contributors

I opened two research issues for this work:

These are good contributions for someone interested in browser AI, SVG internals, evaluation, prompt design, or developer tooling. You do not need to solve the entire system. A useful prompt set, a measurable static check, a failure taxonomy, or a small repair prototype would already move the project forward.

You can try Timeline Studio in the browser or explore the source on GitHub.

If you have worked on generated SVG quality before, I would especially like to hear how you separate “valid output” from “useful design.”

ブラウザだけで動画を4倍高画質化する：NanoVSR WebGPUの実装

MartinDelophy — Tue, 28 Jul 2026 06:42:42 +0000

動画の超解像はサーバーGPUで処理する構成が一般的ですが、動画のアップロード時間やプライバシー、GPUコストが課題になります。

Timeline Studio では、NanoVSR・ONNX Runtime Web・WebGPU・FFmpeg.wasmを組み合わせ、動画の4倍高画質化をブラウザ内で完結させています。

GitHub: https://github.com/MartinDelophy/ai-video-editor
Demo: https://video-editor.ai-creator.top/

処理パイプライン

全体の流れは次のとおりです。

動画を読み込む
  ↓
12fpsでフレームを抽出
  ↓
5フレーム単位でNanoVSRへ入力
  ↓
WebGPUで4倍超解像
  ↓
PNGフレームを生成
  ↓
FFmpeg.wasmでH.264/AACのMP4に再構成

素材は端末内で処理され、動画フレームをサーバーへ送信しません。

1枚ずつではなく、5フレームをまとめて処理する

画像用モデルは1フレームを入力しますが、動画用モデルは5フレームの時系列ウィンドウを使用します。

用途	入力	出力
画像	`[1, 1, 3, 180, 320]`	`[1, 1, 3, 720, 1280]`
動画	`[1, 5, 3, 180, 320]`	`[1, 5, 3, 720, 1280]`

単純な1フレーム超解像では、髪や細線などの生成結果がフレームごとに変わり、再生時にちらつきやすくなります。隣接する5フレームを同時に参照することで、時間方向の一貫性を改善します。

実装では動画全体を一度にGPUへ渡さず、5フレームずつ処理します。これにより、GPUメモリのピークを抑え、キャンセルにも比較的早く反応できます。

Web Workerで推論を分離する

ONNX推論をUIスレッドで実行すると、Reactの描画やタイムライン操作が重くなります。そのため推論処理は専用のWeb Workerへ分離しています。

const session = await ort.InferenceSession.create(model, {
  executionProviders: ["webgpu"],
  graphOptimizationLevel: "all",
});

メインスレッドからは ImageBitmap をTransferableとして送信し、余分なピクセルコピーを避けます。処理後はBitmapとONNX Tensorを明示的に解放します。

bitmap.close?.();
result.sr.dispose?.();

ブラウザ動画処理では、モデルだけでなくBitmap、Canvas、GPU Bufferの解放も重要です。

アスペクト比を維持する

モデル入力は320×180固定ですが、実際の素材には縦動画や正方形動画もあります。

Timeline Studioでは元画像を引き伸ばさず、320×180へcontain配置します。推論後の1280×720画像から有効領域だけを切り出すことで、元のアスペクト比を維持しています。

また、入力素材がすでにモデルの有効出力より大きい場合は、原画像の解像度を保護し、AI結果と元画像のディテールを保守的に合成します。

モデルは固定Revisionでキャッシュする

モデルはHugging FaceとModelScopeの両方から取得でき、中国語環境ではModelScopeを優先します。

ただしキャッシュキーは配信元URLではなく、リポジトリ・Revision・モデルパスから生成します。ミラーが切り替わっても同じモデルを再ダウンロードしません。

const cacheKey =
  "/__model-cache__/haixin/timeline-studio-onnx-models/" +
  MODEL_REVISION + "/" + modelPath;

本番環境では main ではなく不変のコミットRevisionへ固定しています。ONNX SessionもWorker内で再利用するため、2回目以降の実行ではダウンロードと初期化を繰り返しません。

元の音声を残してMP4を生成する

超解像モデルが返すのは画像フレームです。そこで生成したPNGをFFmpeg.wasmへ渡し、H.264動画としてエンコードします。

元動画に音声があれば、クリップの開始位置と長さに合わせて音声を切り出し、AAC 192kbpsで再結合します。

最終的な出力は次の構成です。

H.264 / CRF 18
AAC 192kbps
yuv420p
faststart MP4
処理フレームレートは12fps

これにより、Canvas上のプレビューだけでなく、音声付きのMP4として編集タイムラインへ戻せます。

現在の制約

現在の実装には次の制約があります。

WebGPU対応ブラウザが必要
処理フレームレートは12fps
モデル入力は320×180固定
PNG中間フレームとFFmpeg.wasmが長尺動画でメモリを消費する

今後は VideoFrame → WebGPU → WebCodecs のストリーミング構成に移行することで、PNG中間ファイルを減らし、より長い動画に対応できます。

まとめ

ブラウザ動画超解像では、モデルを動かすだけでなく、フレーム抽出、時系列処理、リソース解放、モデルキャッシュ、音声保持、再エンコードまで一つのパイプラインとして設計する必要があります。

Timeline Studioでは、この一連の処理をローカルファーストな動画編集機能として実装しています。WebGPUやブラウザAIに興味があれば、ぜひリポジトリをチェックしてください。

GitHub: https://github.com/MartinDelophy/ai-video-editor

Removing Video Watermarks Entirely in the Browser with MI-GAN, ONNX Runtime Web, and WebGPU

MartinDelophy — Mon, 27 Jul 2026 08:45:31 +0000

Watermark and unwanted-object removal is one of those features that looks simple in a model demo and becomes much more complicated when you put it inside a real video editor.

The obvious implementation is to upload the media to a GPU server, run an inpainting model, and return the result. That works, but it introduces upload latency, infrastructure cost, and privacy concerns—especially for large or unpublished videos.

For our browser video editor, we took a different approach:

Run MI-GAN locally with ONNX Runtime Web and WebGPU, so images and videos never need to leave the user's device.

Source code: github.com/MartinDelophy/ai-video-editor
Live demo: video-editor.ai-creator.top

This article covers more than model inference. It explains the product-level work required for multiple repair regions, time ranges, moving watermarks, progressive frame previews, before/after comparison, cancellation, and final video encoding.

Use object-removal tools only on media you own or are authorized to edit. Removing ownership or attribution marks from third-party content may violate licenses or platform rules.

Why browser-local repair?

A server-side pipeline usually requires:

Uploading the original media.
Waiting for a large video transfer.
Paying for storage and GPU compute.
Sending potentially sensitive footage to another system.
Depending on a reliable network connection.

Browser-local inference moves the initial cost to downloading the model. Once the runtime and model artifacts are cached, repeated edits are much faster to start, and the original media remains local.

Our implementation uses:

MI-GAN for lightweight image inpainting
ONNX Runtime Web for browser inference
WebGPU as the preferred execution backend
Web Workers to keep inference off the UI thread
Canvas / OffscreenCanvas for frame extraction, masks, and compositing
WebCodecs or a compatibility encoder path for the final video
Cache Storage to avoid downloading the model again

The complete pipeline

Image repair is a single inference task. Video repair has a time axis, so the pipeline is substantially different:

Source media
  -> user defines repair regions and time ranges
  -> resolve regions active at the current timestamp
  -> decode a video frame
  -> build a mask
  -> run MI-GAN inference
  -> composite the repaired pixels over the source frame
  -> commit the progressive preview
  -> encode the processed frame
  -> create a new media asset

The model is only one stage. A production-quality experience also requires the canvas, playhead, processed-frame count, and progress indicator to share the same clock.

Image repair with normalized regions

In image mode, the user draws one or more rectangles over the canvas. We store each rectangle in normalized coordinates so the selection remains correct across different preview sizes.

function createMask(width, height, regions) {
  const canvas = new OffscreenCanvas(width, height);
  const context = canvas.getContext("2d");

  context.fillStyle = "#000";
  context.fillRect(0, 0, width, height);

  context.fillStyle = "#fff";
  for (const region of regions) {
    context.fillRect(
      region.x * width,
      region.y * height,
      region.width * width,
      region.height * height
    );
  }

  return context.getImageData(0, 0, width, height);
}

Each region uses values between 0 and 1. The same rectangle can therefore be mapped back to the full-resolution source instead of the CSS-sized preview.

After inference, we do not immediately commit the result. The editor displays a draggable vertical comparison line:

<ComparisonView
  before={sourceUrl}
  after={repairedUrl}
  position={comparePosition}
  onPositionChange={setComparePosition}
/>

That distinction matters because inpainting does not recover a hidden ground truth. It synthesizes plausible pixels from the surrounding context, so users need a clear way to inspect the result before applying it.

Video repair is a time-range problem

Our first video prototype repaired only the frame currently visible in the editor. It looked correct during a single-frame test but was obviously wrong for a video.

A watermark exists for a time range, not for one frame. It may also move: for example, it can appear in the bottom-left corner for the first few seconds and later switch to the top-right.

We represent every repair region as an independent time-scoped object:

const repairRegion = {
  id: "region-1",
  startTime: 0,
  endTime: 4,
  keyframes: [
    { time: 0, x: 0.72, y: 0.82, width: 0.22, height: 0.10 },
    { time: 4, x: 0.08, y: 0.08, width: 0.22, height: 0.10 }
  ]
};

The rectangle between two recorded positions is interpolated:

function interpolateRegion(a, b, ratio) {
  return {
    x: a.x + (b.x - a.x) * ratio,
    y: a.y + (b.y - a.y) * ratio,
    width: a.width + (b.width - a.width) * ratio,
    height: a.height + (b.height - a.height) * ratio
  };
}

This supports fixed watermarks, watermarks that jump to another corner, and slowly moving overlays without creating a separate repair job for every frame.

Multiple regions for both images and videos

Real media may contain a logo and a text overlay at the same time. Both image and video workflows therefore need multiple selections.

At each video timestamp, we resolve all active regions and combine them into one mask:

const activeRegions = regions
  .filter((region) => {
    return time >= region.startTime && time <= region.endTime;
  })
  .map((region) => resolveRegionAtTime(region, time));

This data model also gives us a practical editing interface:

Every region has its own start and end time.
A selected region can be moved and resized independently.
Positions can be recorded at different timestamps.
Individual regions can be deleted.
Region edits can participate in undo and redo history.

Keeping MI-GAN off the main thread

Creating the model session or running inference on the main thread can freeze the entire editor. We keep a long-lived model session inside a Web Worker and transfer frame data to it.

const session = await ort.InferenceSession.create(modelUrl, {
  executionProviders: ["webgpu", "wasm"]
});

WebGPU is preferred. If it is unavailable or initialization fails, we explicitly fall back to WASM and report the actual backend in the UI.

worker.postMessage(
  {
    type: "repair",
    requestId,
    imageBitmap,
    regions
  },
  [imageBitmap]
);

The Worker and inference session remain alive for later requests. Model and runtime assets are stored in a versioned Cache Storage entry, so a second repair should never present itself as another model download.

Preprocessing and compositing without degrading the full frame

Inpainting models commonly expect a fixed input size. Resizing and regenerating the entire video frame for a small watermark can unintentionally change faces, text, and fine texture outside the selected area.

Our processing rules are:

Transform the frame and mask into the model's expected input.
Include enough context around the masked area.
Restore only the required patch at source resolution.
Preserve every pixel outside the repair mask.
Blend the mask boundary to avoid a visible seam.

context.drawImage(originalFrame, 0, 0);
context.save();
context.globalCompositeOperation = "source-over";
context.drawImage(
  repairedPatch,
  patchX,
  patchY,
  patchWidth,
  patchHeight
);
context.restore();

Keeping regions reasonably tight improves both performance and output stability.

The subtle bug: progress and preview were out of sync

An early version used three independent notions of time:

Progress followed the frame-processing loop.
Preview followed the frame the browser had actually decoded and rendered.
Timeline followed a separate playback clock.

The result was confusing: the progress bar might show 40% while the preview was still near 25%.

The fix was to make the committed visible frame the single UI clock:

for (let frameIndex = 0; frameIndex < totalFrames; frameIndex += 1) {
  const frame = await decodeFrame(frameIndex);
  const repairedFrame = await repairFrameIfNeeded(frame, frameIndex);

  await onFrameCommitted(repairedFrame, frameIndex);

  reportProgress({
    completedFrames: frameIndex + 1,
    totalFrames
  });
}

Frames outside every repair range must still call onFrameCommitted. Otherwise, the preview and playhead appear frozen whenever inference is skipped.

We also decode the progressive preview image and give the browser one paint before advancing the completed-frame count:

await image.decode();
setPreviewFrame(image.src);
await new Promise(requestAnimationFrame);
setCompletedFrame(frameIndex + 1);

During processing, the left side now displays the source frame for the current timestamp while the right side displays the repaired frame for that same timestamp. The comparison line stays interactive throughout the process.

Separate repair progress from encoding progress

Reaching 100% repaired frames does not mean the output file is ready. We still need to initialize the encoder, encode the frames, and create the final media asset.

We expose these as distinct phases:

1. Preparing the model
2. Repairing frames
3. Loading the video encoder
4. Encoding the repaired video
5. Creating the new asset
6. Complete

Trying to compress all of these into a single unexplained percentage makes the UI appear stuck at the end. A phase label tells the user what the browser is actually doing.

Cancellation must release real resources

For long videos, cancellation is not optional. Changing the button text to "Cancelling..." is not enough.

A real cancel path must:

Stop scheduling new frame decodes.
Invalidate pending Worker inference requests.
Close VideoFrame and ImageBitmap objects.
Revoke temporary Blob URLs.
Stop encoders and release media resources.
Avoid downloading or inserting a partial result.

if (abortSignal.aborted) {
  throw new DOMException("Repair cancelled", "AbortError");
}

We disable duplicate cancel requests immediately, but return to an editable state only after cleanup has completed. The original media remains unchanged until the user explicitly applies the new result.

A dedicated repair workspace works better than a crowded inspector

Our first UI placed all controls in the editor's right-side property panel. That panel quickly became too small for region drawing, time ranges, keyframes, preview, comparison, and progress.

The final product keeps AI Repair as a lightweight capability entry point and opens the actual work in a larger modal:

Top toolbar: move, redraw, presets, undo, redo
Center: source/repaired comparison canvas
Bottom: playback, time scale, range handles, playhead
Right side: repair-region list and time/position controls
Footer: cancel, single-frame repair/test, process range, apply result

The mobile layout uses the same mental model in a vertical workspace. Applying closes both the repair modal and the properties drawer; cancelling closes only the modal.

Performance lessons

Browser AI performance varies widely with GPU, resolution, codec, and browser implementation. Rather than promising a universal frame rate, we focused on these principles:

Download model artifacts in parallel and store them in a versioned cache.
Create ONNX sessions serially and only once.
Reuse the Worker and session for repeated jobs.
Skip inference for frames without an active mask.
Keep repair regions as small as practical.
Give the UI a chance to paint between committed frames.
Revoke temporary URLs and close GPU/video resources promptly.
Fall back to WASM clearly when WebGPU is unavailable.

The first model load is still a cost, but cached repeat operations become a much better fit for an interactive editor.

Current limitations

MI-GAN is lightweight and fast enough to be practical in a browser, but it is not perfect:

A watermark covering a face or detailed text may produce unnatural content.
Very large selected areas can generate unstable structures.
Fast motion may reveal temporal inconsistency between frames.
Processing can be significantly slower without WebGPU.
Total time includes decoding and encoding, not just model inference.

Frame-by-frame inpainting does not guarantee temporal consistency. Optical flow, region tracking, and temporally aware restoration models are natural next steps.

Final thoughts

The biggest lesson from building this feature was:

Successful model inference is only the beginning. The real product is the editing flow that lets users inspect, trust, cancel, and apply the result.

Repairing one image in a demo and shipping multi-region, time-aware, progressively previewed video repair are completely different engineering problems.

This implementation runs locally in the browser and inserts the result as a new media asset, leaving the original recoverable.

GitHub: https://github.com/MartinDelophy/ai-video-editor
Live demo: https://video-editor.ai-creator.top/

If you are experimenting with WebGPU, ONNX Runtime Web, browser-side media processing, or video-editor UX, I hope these implementation details save you a few iterations.

Running AI Music Generation Entirely in the Browser: ONNX, WebGPU, Samplers, and Long Audio Stitching

MartinDelophy — Sun, 26 Jul 2026 03:03:41 +0000

Many developers think running AI models in the browser is hard because of inference performance.

After shipping local browser AI music generation in our open-source video editor Timeline Studio, I can confirm:

Model inference is the easy part.

The real difficulty lies in solving browser-specific constraints: model size, WebGPU memory limits, cache quotas, sampler correctness, long audio generation, and stable editor integration.

This article breaks down our complete production-grade browser AI music pipeline, including all pitfalls and fixes.

1. Why We Chose Stable Audio 3 Small (Q4 ONNX)

We use the Stable Audio 3 Small Music Q4 ONNX model.

Total size: 683MB

It consists of four core components:

Module	Purpose	Size
Text Encoder	Encode prompt into conditioning vectors	213MB
Number Conditioner	Encode target audio duration	<1MB
DiT Diffusion	Main denoising computation	380MB
Audio Decoder	Decode latents to 44.1kHz stereo audio	45MB

Browser-specific optimizations

Model sharding
We split model weights into chunks under 100MB to avoid browser network timeout and parsing failures.
4-bit quantization
MatMul / Linear → MatMulNBits
Embedding → GatherBlockQuantized
Other parameters remain FP32

Q4 drastically reduces download size and runtime memory usage.

Tradeoff: minor grainy artifacts in high-frequency and reverb tails — acceptable for fully local browser generation and runnable on 16GB M1 devices.

2. Parallel Download + Serial WebGPU Initialization

Parallel network fetch

Model files are downloaded in parallel to reduce first-time startup latency:

const responses = await Promise.all(
  paths.map(path => fetchModelFile(path))
);

Serial GPU initialization (critical fix)

While network requests are parallelizable, WebGPU session creation cannot.

Parallel initialization of multiple large ONNX sessions causes instant unified memory spikes, especially on Apple Silicon, often leading to WebGPU allocation failure or tab crash.

We enforce a strict sequential pipeline:

Text Encoder → Number Conditioner → DiT → Audio Decoder

Core engineering rule

Network I/O can be parallelized, but WebGPU resource initialization must be serialized.

Once initialized in a Web Worker, all GPU sessions are reused for subsequent generations, avoiding reloading overhead.

3. Multilingual Prompt Pipeline (Auto Translate + Structured Prompts)

Stable Audio models natively understand only English.

To support global users, we built a fully automatic multilingual prompt pipeline:

User native prompt
→ Language detection
→ Chrome built-in translation
→ Structured English music prompt

We also auto-append standardized metadata: style, mood, instrument, BPM, and no-vocal constraints.

Example transformation:

Raw Chinese input
melancholic piano in a rainy café

Final structured prompt

melancholic jazz piano in a rainy café,
cinematic soundtrack,
dreamy,
piano,
90 BPM,
instrumental music,
clean production,
no vocals

For browsers without translation support, we gracefully fall back and ask users for English input, avoiding silent generation failures.

4. Most Audio Quality Issues Are Not From Quantization

Initially, our generated audio suffered from:

Grainy high-frequency noise
Blurry instrument texture
Unstable reverb tails
Loose rhythm structure

We initially suspected Q4 quantization quality loss.

After deep investigation, the real root cause was incorrect sampler schedule implementation.

Correct Rectified Flow sampling logic

denoised = x - tCurrent * velocity;
xNext = (1 - tNext) * denoised + tNext * randomNoise;

The time parameter t must decay strictly from 1 to 0.

Our original schedule only decayed to ~0.27, leaving 27% residual noise in the final latent before decoding — this destroyed audio quality.

Fixed LogSNR schedule

const logSnr = 2 - t * 8.2;
const sigma = 1 / (1 + Math.exp(logSnr));

// Hard enforce clean endpoints
schedule[0] = 1;
schedule[steps] = 0;

// Zero-noise final step
if (tNext === 0) {
  x = denoised;
}

This single fix improved audio quality far more than increasing sampling steps.

Key takeaway

Poor local model generation quality is usually caused by preprocessing/scheduler bugs, not model quantization.

5. From Latent Output to Final WAV

Latent length is dynamically calculated based on target audio duration:

latentLength =
  Math.ceil((seconds + 6) * 44100 / 8192) * 2;

DiT outputs latent tensors, and the decoder produces:

Shape: [1, 2, audioFrames]
Format: Float32
Range: -1 ~ 1
Sample rate: 44100Hz

We convert float samples to standard 16-bit PCM WAV:

sample< 0 ? sample * 32768 : sample * 32767;

Finally, the audio asset is automatically:

Converted to WAV blob
Decoded for accurate duration
Waveform-analyzed
Saved with prompt/model metadata
Imported into the editor asset library

6. Long Audio Generation: Why We Avoid Direct 120s Inference

Direct long-form audio inference (90s/120s) in browsers causes:

WebGPU buffer allocation failures
Excessive memory usage
Tab freezing or system process killing
Peak decoder memory pressure

To solve this, we use a segmented generation + intelligent looping strategy:

90s target → generate 45s segment
120s target → generate 60s segment

The final long audio is stitched from high-quality short segments, cutting VRAM and inference time nearly in half.

7. Seamless Audio Stitching (No Clicks, No Jumps)

Naive loop splicing creates obvious artifacts:

Volume jumps
Misaligned drum hits
Phase discontinuity
Audible clicks/pauses

We analyze the last 5 seconds of the source segment and score candidate cut points using:

RMS energy
Amplitude jump difference
Waveform slope jump
Shorten ratio penalty

Unified scoring formula:

score =
  rms * 0.7
  + amplitudeJump * 0.8
  + slopeJump * 0.25
  + shortenedRatio * 0.08;

Lower score = better loop boundary.

We also use dynamic fade duration (0.25s ~ 1.5s) based on local audio energy:

fadeSeconds = clamp(
  0.25 + localRms * 4,
  0.25,
  1.5
);

High-energy regions get longer fades; quiet regions use short transitions for natural looping.

8. Model Caching: Harder Than Inference

We use Service Worker + Cache Storage to cache the 683MB model persistently.

We fixed two production-critical browser cache issues.

Issue 1: Cache hit misjudgment

Custom HTTP headers cannot be reliably exposed cross-origin after Service Worker proxying.
This caused UI to show “downloading” even when models were loaded from local cache.

Fix: Directly query Cache Storage from the AI worker instead of relying on response headers.

Issue 2: QuotaExceededError

Dual-write from both AI Worker and Service Worker caused temporary duplicate model occupancy, exceeding browser storage quota.

Final cache architecture:

Only Service Worker writes cache
AI Worker only reads/validates cache
Cache failure does not block generation
Auto clean old model cache versions

Large model caching requires a single source of truth for writes. Cache logic must never break core inference.

9. Model Mirroring & Immutable Version Locking

Public Hugging Face repos are unstable for production use — authors may delete, overwrite, or update weights.

We fully mirrored and frozen the model:

Mirror repo: haixin/stable-audio-3-small-music-onnx
Fixed immutable commit hash: 0b8a05e0bc3511e674b4cb3413d3ef6c48880cdb

We verified:

26 model files
Full 683MB byte integrity
SHA256 hashes
ONNX graph & weight shards
License & notice files

Old cache entries are compatible, so existing users do not need to re-download.

10. Real Boundaries & Value of Browser-Native AI Music

Limitations

Q4 quantization cannot match FP16/FP32 server-side quality
Long audio relies on intelligent looping instead of full inference
WebGPU support depends on device/browser implementation

Unique strengths

Zero server-side inference cost
100% local processing (prompt + audio never uploaded)
One-time model download + persistent cache
Hot GPU session reuse for fast subsequent generation
Native integration into video editor workflow
Deployable via static hosting

Conclusion

Running production-ready AI music in the browser is not just about getting ONNX inference running.

It requires careful system-level engineering:

Correct sampler scheduling
WebGPU memory & concurrency control
Intelligent long audio stitching
Robust caching strategy
Immutable model supply chain

With these fixes, a 683MB quantized music model can become a usable, stable, production-grade browser AI feature.

Open Source Project

If you are interested in browser-side AI, WebGPU inference, or local media generation:

GitHub: martindelophy/ai-video-editor

How I Built a Local-First AI Video Editor with WebCodecs, WebGPU, and WASM

MartinDelophy — Sat, 25 Jul 2026 08:35:21 +0000

Modern video editors are powerful, but AI-assisted workflows often begin with the same trade-off: upload your media to someone else's server, wait for processing, and accept that the editable state lives somewhere else.

I wanted to explore a different constraint:

How much of a serious AI video editing workflow can run entirely inside the browser?

The result is Timeline Studio, an MIT-licensed video editor built with React, browser media APIs, ONNX Runtime Web, WebGPU, WASM, and WebCodecs.

The interesting part was not adding another timeline UI. It was getting media playback, AI inference, editable timeline state, audio mixing, and deterministic export to agree on the same definition of time — without relying on an editing backend.

The architecture in one view

Timeline Studio separates interactive editing from expensive media and AI work:

React UI and timeline state
          |
          +-- Native media playback for responsive preview
          |
          +-- Web Workers
          |      +-- ONNX Runtime Web
          |      +-- WebGPU inference
          |      +-- WASM fallback
          |
          +-- OfflineAudioContext for audio mixing
          |
          +-- Deterministic composition renderer
                 +-- exact timeline timestamps
                 +-- Canvas composition
                 +-- WebCodecs encoding
                 +-- MP4 / WebM muxing

Large models are loaded only when a feature needs them. Model artifacts are revision-pinned and cached locally, while inference runs in long-lived workers so repeated operations do not rebuild the entire runtime.

The editor remains responsive because React owns the editing interface and declarative project state, not every decoded video frame.

One timeline, two playback paths

A real-time preview and a high-quality export have different requirements.

During editing, the active video element follows the browser's native playback path. Trying to continuously seek media elements from delayed React state produced visible stuttering, especially with multiple layers.

Export takes the opposite approach. It resolves every frame from an exact output timestamp. For each timestamp, the renderer determines which clips are active and composes:

main visual media;
picture-in-picture overlays;
captions and stickers;
masks, filters, and effects;
keyframed transforms;
animations and transitions.

This split gives the editor smooth playback without making the exported file depend on real-time UI performance.

Deterministic export with WebCodecs

MediaRecorder is useful when compatibility matters, but it records a real-time stream. If the tab stalls or the device cannot keep up, the output can inherit timing problems from the recording session.

Timeline Studio therefore treats WebCodecs as the primary export path.

For each output frame, the renderer:

converts the frame index into an exact timeline timestamp;
resolves every active visual and its source timestamp;
draws the shared composition geometry to the export canvas;
passes the frame to the selected video encoder;
mixes audio separately and muxes it into the final container.

The same geometry rules are shared by preview and export. A caption, mask, overlay, or transform should not move merely because the output resolution changes.

MediaRecorder remains a compatibility fallback, but it is not allowed to silently change the requested format. Active exports also expose a real cancel path that stops further work, releases media resources, and never downloads a partial file.

Running ONNX models with WebGPU and WASM

The editor currently uses browser-side models for several workflows:

Piper/VITS and Kokoro for multilingual voice generation;
Whisper small q8 for automatic captions;
YOLOS and MODNet for subject detection and portrait matting;
vocal separation and AI music generation;
JoyVASA and LivePortrait for talking-avatar generation.

WebGPU is preferred when a model path has been verified with it. WASM is an explicit fallback rather than a hidden change of engine.

A lesson from this work was that model availability is not the same as workflow reliability. A voice is not added to the catalog just because an ONNX file exists; it needs to pass generation, timeline insertion, and downloaded-WAV validation through the editor itself.

Heavy inference runs outside the UI thread. For model bundles containing several artifacts, downloads happen in parallel, while GPU sessions are created serially to avoid unreliable initialization and memory pressure. Once initialized, a worker stays alive for repeated generations on the same page.

Caching large model artifacts

Downloading a model again on every visit would make a browser AI editor impractical.

Timeline Studio uses a service worker and versioned browser caches for its application shell and large runtime assets. Model URLs are revision-pinned so a cached artifact has a stable identity. When hosting locations change, legacy cache keys can still be recognized to avoid forcing users to download identical weights again.

This changes the perceived workflow:

the first run performs model setup;
later runs reuse the initialized or cached assets;
repeated generation is presented as generation, not another fake "model download."

The application is also installable as a PWA, so the editor shell can start independently of a full network round trip.

Audio is its own timeline problem

Video export is only half of the job. Voiceovers, music, separated source audio, and enabled embedded video audio all need to agree with clip ranges and source offsets.

Timeline Studio prepares audio independently with OfflineAudioContext. Each source is cropped and scheduled against the export range, then mixed before muxing.

This becomes particularly important for custom in/out exports. Cropping only the visuals is not enough: captions, overlays, voice, music, embedded audio, and SRT timing all have to be rebased to zero consistently.

Waveforms are derived from decoded peaks when available. While decoding or restoration is still in progress, clips render a stable low-emphasis placeholder instead of becoming blank and visually ambiguous.

AI output should become editable state

The most useful architectural decision was to avoid treating AI output as a final opaque artifact.

Instead:

speech recognition becomes timed caption clips;
generated speech becomes editable audio linked to a caption;
subject detection becomes a timestamped analysis track;
vocal separation becomes independent vocal and instrumental clips;
avatar generation becomes replaceable visual media;
AI music is added to the asset library instead of being inserted automatically.

This keeps AI operations reversible. Users can move, trim, replace, unlink, or regenerate individual results without repeating the entire workflow.

Making the timeline agent-editable

The repository also includes an edit-timeline-studio skill and a versioned command runner for tools such as Codex, Claude Code, Copilot, and Gemini CLI.

The command layer can inspect portable .timeline projects, validate revisioned edit plans, show field-level diffs, apply supported edits transactionally, and render a documented portable subset to H.264/AAC MP4.

For example:

npm run agent -- project.inspect /absolute/path/project.timeline
npm run agent -- project.diff /absolute/path/edit-plan.json
npm run agent -- project.run /absolute/path/edit-plan.json
npm run agent -- project.render /absolute/path/render-request.json

Dry runs, stable clip IDs, explicit timestamps, revision checks, and idempotent operation IDs matter here. An agent should not turn a user's editable project into an irreversible render or apply the same mutation twice after a retry.

What I learned

Building a browser video editor is less about drawing frames than maintaining one consistent interpretation of time.

The timeline, media elements, waveform data, captions, keyframes, source offsets, transitions, audio mixer, and export renderer must all answer the same questions:

Which clips are active now?
Which source frame or audio sample belongs here?
Which values are interpolated at this timestamp?
What should happen before the first keyframe and after the last one?

AI adds another layer, but the same principle applies: generated results become much more useful when they enter a deterministic, editable timeline model.

The browser platform is now capable of handling far more of this stack than I expected. WebCodecs, WebGPU, WASM, Workers, OfflineAudioContext, Cache Storage, and installable PWAs can support a substantial local-first media workflow — as long as their responsibilities remain clearly separated.

Try it

Timeline Studio is open source and actively developed:

You can run it locally with Node.js 20 or newer:

git clone https://github.com/MartinDelophy/ai-video-editor.git
cd ai-video-editor
npm install
npm run dev

If you work with browser media, WebCodecs, WebGPU, ONNX, timeline UX, or deterministic rendering, feedback and focused contributions are welcome.

DEV Community: MartinDelophy

I Built an AI Skill That Turns a URL or Raw Footage Into an Editable Video Project

What the Skill can do

Website promotion is more than screen recording

Voiceover becomes the timing source

An editable .timeline v3 project

Fixing playback loops at clip boundaries

Install the Skill

Building Cinematic Depth in the Browser with Depth Anything V2 Small and WebGPU

Introduction

Background blur is not depth of field

Why the Small model?

Designing an editing feature, not a model demo

Turning depth into blur

One depth map is not enough for video

Clip-scoped depth caching

Preview and export must match

ModelScope and Hugging Face mirrors

Progress, cancellation, and useful errors

How long does a 10-second clip take?

What else can time-varying depth enable?

Final thoughts

Why Browser-Based Multilingual TTS Kept Freezing at 86% — and How We Fixed It

The browser was doing more than the progress bar showed

One origin, many competing caches

Separate model identity from download URL

Use regional mirrors without fragmenting the cache

A smaller model can be the faster product decision

Evict stale voice models, not the entire application cache

Give large model files one caching owner

Progress should be byte-weighted and stage-aware

Let the browser paint before heavy WASM work

Hide runtime diversity behind one interface

What we verified

Lessons for browser AI applications

Building Semantic Optical Flow Tracking in the Browser with WebCodecs

Project links

Why raw optical flow was not enough

The processing pipeline

Detecting the subject before calculating flow

Representing a semantic motion cohort

Calculating flow inside the semantic ROI

Aggregating local vectors into one group direction

Turning frame-by-frame motion into a trajectory

Rendering the visualization

Adding the timecode

Low-resolution analysis, high-resolution rendering

Analysis path

Rendering path

Encoding the result with WebCodecs

Generating real timeline frames

Adding the result to My assets

A small browser benchmark

Benefits and limitations of browser-local processing

Conclusion

Building Browser-Local Video Face Swapping: Lessons from a WebGPU Inference Pipeline

1. Follow one frame through the entire pipeline

2. Do not send a full 1080p frame through the generator

3. Download models in parallel, initialize WebGPU sessions serially

4. Transfer buffers instead of cloning them

5. Optical flow is useful only when you can reject bad tracks

6. The highest-confidence face is not always the target

7. Traditional image processing still matters after generation

Mask morphology

Bounded color matching

8. A model URL is part of its cache identity

9. Memory ownership is harder than one inference timing

10. Cancellation must reach the deepest stage

11. Report cold and warm performance separately

Cold start

Warm start

12. Local processing does not remove consent and safety obligations

Closing thoughts

Building Browser-Only Object Outlines with Detection, Prompted Segmentation, and Optical Flow

Introduction

The processing pipeline

Run inference outside the UI thread

Selecting and locking the target

Prompted segmentation and alpha generation

Sparse anchors plus optical flow

An editable `.timeline` v3 project