Building Browser-Local Video Face Swapping: Lessons from a WebGPU Inference Pipeline

Video face swapping is usually associated with cloud GPUs: upload a video, process it frame by frame on a server, and download the result.

Modern browser APIs make another architecture possible. WebGPU, WebCodecs, Web Workers, and ONNX Runtime Web can move a meaningful part of the pipeline onto the user's device. Source media does not have to be uploaded to an inference server, models can be cached locally, and completed results can go straight into an editable video project.

Once we built the pipeline, however, model inference turned out to be only part of the problem. Pixel conversion, cross-thread transfers, GPU initialization, target tracking, mask cleanup, memory ownership, cancellation, and reproducible benchmarking all had a direct impact on whether the feature felt usable.

This article focuses on those engineering details rather than the UI.

The implementation discussed here is part of the open-source Timeline Studio project.

Responsible-use notice

This article is a technical discussion of browser image processing and machine-learning inference, not legal advice. Use only face images and videos for which you have clear authorization from the depicted people and relevant rights holders. Do not use face synthesis for illegal, infringing, deceptive, misleading, harassing, fraudulent, or identity-abuse purposes. Do not present generated output as authentic footage or a factual record. Label AI-altered content when appropriate, and comply with applicable laws, contracts, media licenses, and platform policies.

1. Follow one frame through the entire pipeline

A decoded frame travels through roughly the following sequence:

VideoFrame / Canvas
    ↓
RGBA Uint8ClampedArray
    ↓
NCHW Float32Array
    ↓
ONNX Tensor
    ↓
Generated RGB and Alpha Mask
    ↓
Canvas compositing
    ↓
WebP intermediate frame
    ↓
WebM encoding

Every arrow costs time and memory.

SCRFD and MobileFaceSwap expect NCHW tensors, while pixels read from a Canvas are interleaved RGBA values. Before inference, the three color channels must be separated:

const plane = width * height;
const tensor = new Float32Array(plane * 3);

for (let i = 0; i < plane; i += 1) {
  tensor[i] = normalize(rgba[i * 4]);
  tensor[plane + i] = normalize(rgba[i * 4 + 1]);
  tensor[plane * 2 + i] = normalize(rgba[i * 4 + 2]);
}

This loop looks harmless in isolation. In a video pipeline it runs repeatedly, alongside Canvas readback, Tensor allocation, and worker communication. Together, these operations can consume substantial CPU time and memory bandwidth.

That leads to an important rule: choose model input sizes based on the cost of the entire data path, not model accuracy alone.

2. Do not send a full 1080p frame through the generator

Face detection and face generation solve different problems, so they should not use the same resolution.

Stage	Input size	Purpose
SCRFD detection	640×640	Locate faces and landmarks across the complete frame
Identity extraction	112×112	Extract a source identity representation that is relatively pose-independent
Face generation	224×224	Generate the new identity in the target pose
Optical flow	Long edge at most 720px	Track five facial landmarks at lower cost
Final composition	Original video resolution	Preserve the background and source detail

Detection needs the complete image. A 640×640 analysis frame helps retain smaller faces. Generation only needs an aligned face crop, so it stays at 224×224.

Sending a complete 1080p frame into the generator would spend most of its compute on background pixels that never change. Detecting globally, aligning the face, and generating only the region of interest is one of the main reasons this workload can run in a browser.

3. Download models in parallel, initialize WebGPU sessions serially

The first run has two expensive phases: downloading the model files and creating ONNX sessions. They benefit from different scheduling strategies.

Model downloads are independent, so they can run concurrently. Session creation, however, may involve:

• parsing the ONNX graph;
• graph optimization and operator fusion;
• WebGPU shader generation;
• pipeline compilation;
• weight upload;
• GPU buffer allocation.

Creating several sessions at once can produce a burst of shader compilation and GPU allocation. On resource-constrained devices, this increases startup jitter and peak memory use, and may even contribute to a lost GPU device.

We use parallel download followed by serial initialization:

const [
  detectorBuffer,
  identityBuffer,
  conditionerBuffer,
  generatorBuffer,
] = await Promise.all(modelDownloads);

const detector = await createSession(detectorBuffer);
const identity = await createSession(identityBuffer);
const conditioner = await createSession(conditionerBuffer);
const generator = await createSession(generatorBuffer);

This preserves network concurrency without concentrating GPU initialization into the same moment.

4. Transfer buffers instead of cloning them

Detection and generation should not block the UI thread, so frames and tensors move through Web Workers.

If an ArrayBuffer is posted without a transfer list, the browser may use structured cloning. A single 640×640 Float32 RGB tensor is already about 4.69 MB:

640 × 640 × 3 × 4 bytes ≈ 4.69 MB

Copying that buffer at every detection anchor quickly creates memory-bandwidth and garbage-collection pressure.

Transfer ownership instead:

worker.postMessage(
  {
    type: "detect",
    pixels: tensor.buffer,
  },
  [tensor.buffer],
);

After the call, the original thread's buffer is detached and the worker owns it. Generated RGB output and alpha masks can return through the same mechanism.

Transferables do not make inference itself faster. They remove avoidable copies and short-lived large objects, which matters just as much in a continuous video workload.

5. Optical flow is useful only when you can reject bad tracks

Running full face detection on every frame is expensive. A practical pipeline detects anchor frames and propagates landmarks between them with Lucas–Kanade optical flow.

Optical flow estimates local pixel movement; it does not guarantee that the match is correct. We therefore use a forward-backward consistency check.

Given a point p_t in the previous frame, forward tracking produces:

p_{t+1}=F(I_t,I_{t+1},p_t)

Track that result backward:

\hat{p_t}=F(I_{t+1},I_t,p_{t+1})

The forward-backward error is:

e_{fb}=\lVert \hat{p_t}-p_t \rVert_2

A stable track should return close to the original point. In our case, a frame is accepted only when at least four of the five landmarks remain valid and their average error stays below a threshold.

This check helps reject:

• incorrect matches during fast movement;
• hands or objects covering the face;
• motion blur;
• a subject leaving the frame;
• abrupt lighting changes;
• landmarks drifting onto background texture.

Optical flow is still limited to short propagation windows. SCRFD runs again at fixed intervals to prevent error from accumulating indefinitely.

6. The highest-confidence face is not always the target

Selecting the highest-confidence detection on each frame works poorly in multi-person footage. A new person may enter closer to the camera and receive a better detector score, even though they are not the editing target.

Target selection becomes a temporal matching problem. Candidate scoring can consider:

• distance from the previous target center;
• change in bounding-box area;
• current detector confidence;
• distance from the frame center on the first frame.

One simplified score is:

S=w_cC-w_dD-w_aA

Here, C is detector confidence, D is center distance, A is the area change, and the w values are weights.

The first frame favors a confident, reasonably large face near the center. Later frames give more weight to spatial and scale continuity.

This is not full face re-identification, but it is substantially more stable than picking the highest-confidence box on every frame. When a match is not trustworthy, keeping the original frame is safer than applying the effect to the wrong person.

7. Traditional image processing still matters after generation

Mask morphology

The generator's mask may contain holes, spikes, or discontinuous edges. Using it directly can make the blend boundary flicker.

A typical cleanup sequence is:

Raw mask
  ↓
Threshold
  ↓
Dilation: close local gaps
  ↓
Erosion: remove outer spikes
  ↓
Additional erosion: pull the blend region inward
  ↓
Box blur: create a smooth alpha
  ↓
Boundary safety mask: remove crop-box edges

Dilation and erosion can use a separable sliding-window implementation: filter horizontally, then vertically.

A direct two-dimensional morphology operation with radius r is roughly:

O(W \times H \times r^2)

A separable sliding-window implementation can approach:

O(W \times H)

The neural network may run on the GPU, but conventional algorithms that execute on every frame still deserve careful optimization.

Bounded color matching

When the source identity and target video have different color temperatures, the generated face can look disconnected from the forehead or neck.

Within the valid mask, we calculate the mean and standard deviation of the generated and target pixels, then apply a per-channel correction:

I'=\frac{\sigma_t}{\sigma_s}(I-\mu_s)+\mu_t

Unbounded correction may amplify noise or produce extreme colors under unusual lighting. Clamp both scale and offset:

const scale = clamp(targetStd / sourceStd, 0.78, 1.22);
const shift = clamp(targetMean - sourceMean * scale, -0.12, 0.12);

The corrected result is blended with the original output rather than replacing it completely. This reduces skin-tone discontinuities without discarding texture already reconstructed by the generator.

8. A model URL is part of its cache identity

Loading production models from a mutable path is risky:

repository/resolve/main/model.onnx

The URL can remain unchanged while the remote content changes. That can lead to:

• different inference results from the same frontend version;
• stale browser cache entries alongside new server files;
• regressions with no identifiable model version;
• temporarily inconsistent mirrors.

Production URLs should point to immutable revisions:

repository/resolve/<immutable-revision>/model.onnx

We also record the expected file size, SHA-256 hash, purpose, license, and input/output tensor definitions.

After download, validate size or hash before creating a session. Otherwise, a CDN error page or partial response may be passed to the ONNX loader as if it were a model.

9. Memory ownership is harder than one inference timing

A browser video task may hold all of these at once:

• decoded frames;
• Canvas pixels;
• Float32 input tensors;
• ONNX output tensors;
• grayscale optical-flow images;
• intermediate frame blobs;
• encoder buffers.

Relying only on JavaScript garbage collection can produce steady memory growth after a relatively small number of frames. Several resource types need explicit cleanup:

tensor.dispose?.();
bitmap.close();
mat.delete();
input.dispose();
URL.revokeObjectURL(url);
worker.terminate();

ONNX tensors, ImageBitmap, OpenCV matrices, and object URLs belong to different runtimes and have different lifetime rules.

OpenCV.js deserves particular attention: Mat instances allocate from a WebAssembly heap. Losing the JavaScript reference does not guarantee prompt release of the underlying memory, so delete() must be called explicitly.

10. Cancellation must reach the deepest stage

Closing a progress dialog is not cancellation. A real cancel signal must propagate through:

• model downloads;
• video-frame reading;
• face detection;
• optical-flow tracking;
• per-frame generation;
• intermediate compression;
• final video encoding.

The main thread can own an AbortController and send a request-scoped cancel message to the worker:

controller.abort();

worker.postMessage({
  type: "cancel",
  requestId,
});

The worker checks cancellation during download loops, before and after inference, between frames, and when switching requests. A canceled task does not continue into encoding and never adds a partial result to the asset library.

Without this propagation, the UI may report that a task ended while the GPU and CPU continue working in the background.

11. Report cold and warm performance separately

"Completed in seconds" is not a reproducible benchmark.

A browser AI performance report should capture at least:

Device:
GPU:
Operating system:
Browser and version:
WebGPU adapter:
Video codec:
Video resolution:
Video duration:
Output FPS:
Detection-anchor FPS:
First run:
Model initialization time:
Per-frame processing time:
Generator cumulative inference time:
Encoding time:
Peak memory:

It should also separate two scenarios.

Cold start

Cold start includes model download, file validation, ONNX session creation, shader compilation, source-identity extraction, video processing, and encoding. It measures distribution and device initialization as much as inference.

Warm start

Warm start assumes that models are cached and sessions and source-identity weights can be reused. It measures video decoding, anchor detection, optical flow, generation, post-processing, and encoding.

Warm-start performance is closer to the experience of editing several clips on the same page. Cold and warm numbers should not be mixed, and a report should not publish only the faster one.

12. Local processing does not remove consent and safety obligations

Keeping inference in the browser can reduce the need to upload source media to a remote inference service. It does not remove the responsibilities attached to face images and synthetic output.

Before releasing or operating this kind of feature, verify that:

• the depicted people and rights holders authorized the intended use;
• photos, videos, model weights, and sample code are used under valid licenses;
• identifiable face images and embeddings are handled lawfully and securely;
• generated output cannot easily be mistaken for real footage, news, or a person's actual behavior;
• AI alteration is disclosed in a manner appropriate to the use case;
• minors, public figures, and third parties are not used without authorization;
• the workflow is not being used for fraud, impersonation, harassment, defamation, or sexual abuse;
• users can cancel processing and remove generated results.

The safest failure mode is to emit no effect when target identity cannot be maintained reliably.

Closing thoughts

Browser-local video face swapping is not a matter of dropping an ONNX file into a web page. Moving from a demo to a repeatable workflow requires at least four kinds of engineering:

1. Bound the computation: run models only at the required resolution and on the required ROI.
2. Schedule resources deliberately: combine parallel downloads, serial session initialization, and transferable buffers.
3. Measure result confidence: reject bad optical-flow tracks, temporal identity switches, and poor masks.
4. Manage the runtime: pin model versions, release heterogeneous resources, and propagate cancellation.

WebGPU provides the compute, but the surrounding system determines whether that compute becomes a stable user experience.

The same lessons apply to browser-local person segmentation, super-resolution, object tracking, video restoration, and style transfer.

You can explore the implementation in the Timeline Studio repository.

---

Disclosure: AI assistance was used for translation and editorial restructuring of this article. The technical content is based on the linked project implementation.