DEV Community

Cover image for How We Built a Dual-Engine 4K AI Video Upscaler in the Browser with WebGPU, WebCodecs & Serverless RTX 4090 Tiling
beatsprom
beatsprom

Posted on

How We Built a Dual-Engine 4K AI Video Upscaler in the Browser with WebGPU, WebCodecs & Serverless RTX 4090 Tiling

#webgpu #ai #javascript

Anyone who works with AI-generated video clips (Sora, Runway Gen-3, Kling, Veo, Wan) or older compressed 720p H.264 footage hits the same wall: native output is almost always 720p or soft 1080p, while desktop upscalers cost $299+ and cloud SaaS tools force subscriptions, watermarks, or 20-minute upload queues.

Over the past few months we built UpscaleX Video — a hybrid dual-engine 4K AI video super-resolution platform combining:

  1. A 100% Free, Zero-Upload Client-Side Engine (upscalex.video) running custom WGSL compute shaders on WebGPU (with automatic WebGL2 fallback) and hardware WebCodecs H.264 + AAC MP4 muxing directly inside the browser.
  2. A Serverless Cloud NVIDIA RTX 4090 Engine (fast.upscalex.video) running Real-ESRGAN (Restore, Sharpen, and Anime weights) across 3×3 lossless frame montages for 10× faster 4K batch processing.

Here is a breakdown of the architecture and the trickiest browser/GPU bottlenecks we solved.

1. Why Browser-Based Video Super-Resolution Used to Be Impossible

Until recently, upscaling video inside a browser tab failed on three fronts:

  • WASM CPU bottlenecks: Running convolutional layers frame-by-frame on CPU takes seconds per frame.
  • No native MP4 + Audio encoder: MediaRecorder drops frames under heavy GPU load because it operates strictly in wall-clock real time.
  • VRAM exhaustion at 4K (3840×2160): Naive full-frame tensor allocation blows past browser Dawn/WebGPU buffer limits on 4 GB–8 GB integrated GPUs.

Solving Real-Time Frame Drops with Deterministic WebCodecs

Instead of MediaRecorder, we decouple decoding, neural inference, and encoding using the browser's hardware VideoDecoder and VideoEncoder APIs paired with an in-memory Mp4Muxer:

  1. Demux the source MP4/WebM/MOV stream and preserve the compressed AAC/Opus audio track untouched (or transcode cleanly to 128 kbps stereo AAC when needed).
  2. Decode video frames deterministically via VideoDecoder with backpressure control (never keeping more than 3 VideoFrame handles alive in GPU memory).
  3. Run our residual CNN super-resolution pass in WGSL (WebGPU) or fragment shaders (WebGL2) to reconstruct high-frequency edge detail and suppress H.264 8×8 macroblock ringing.
  4. Feed each upscaled VideoFrame with its exact microsecond presentation timestamp (timestamp = frameIndex * 1_000_000 / fps) into hardware VideoEncoder (avc1.640034 High Profile Level 5.2 for 4K).

Because timestamps are synthetic rather than wall-clock bound, even if a laptop GPU processes at 8 FPS on a 30 FPS clip, the exported 4K MP4 plays back at a silky-smooth, frame-accurate 30 FPS with zero audio drift. We documented the full pipeline diagram in our WebGPU & Cloud RTX Architecture Deep-Dive.

2. Eliminating HTTP Overhead on Cloud RTX 4090s with 3×3 Montage Tiling

Client-side WebGPU is unbeatable for privacy, short clips, and zero cost — your video never leaves your device. You can test it right away on our Free 4K Video Upscaler without creating an account.

However, when creators need to upscale a 15-minute 60fps clip or batch-process 25 clips on a lightweight MacBook Air or phone, a dedicated NVIDIA RTX 4090 (24 GB VRAM, Ada Lovelace FP16 Tensor Cores) is 10×–25× faster.

When we first built fast.upscalex.video on serverless RTX 4090 workers, sending individual PNG frames over HTTPS wasted up to 40% of wall-clock time on network round-trips and GPU kernel launch overhead.

The 3×3 Frame Montage Trick

Instead of dispatching 1 frame per inference job:

  • The browser packs 9 consecutive video frames into a single 3×3 grid image on an OffscreenCanvas.
  • A single serverless RTX 4090 Real-ESRGAN FP16 forward pass upscales all 9 frames simultaneously at near-100% Tensor Core utilization.
  • Our client demultiplexes the returned 3×3 4K sheet back into 9 exact frames, feeds them straight into hardware VideoEncoder, and caches the finished 4K MP4 inside IndexedDB (UpscaleXCloudVault) so a browser refresh never loses a paid render.

This cut per-frame HTTP overhead by 89% and let us drop pay-as-you-go pricing down to $0.08/minute (starting from $0.01 per short clip, with no monthly subscription).

3. Live RTX 4090 Crop Preview Before Spending a Cent

Different footage degrades differently:

To avoid blind full-video renders, we built an interactive Draggable Minimap & Live RTX 4090 Crop Preview inside the workspace: drag the ROI box over an actor's eyes or small text, hit preview, and inspect the exact 2× or 4× GPU output side-by-side in ~2 seconds before launching the full render.

Try It Out & Feedback Welcome

If you're working with WebGPU compute shaders, VideoDecoder/VideoEncoder backpressure, or serverless GPU inference, happy to answer any questions in the comments!

Top comments (0)