DEV Community

nareshipme
nareshipme

Posted on

How We Moved Video Rendering From the Server to the Browser

#typescript #nextjs #webdev #opensource

The Problem With Server-Side Video Rendering

If you have ever built a video editing tool, you know the pain of server-side rendering. Heavy FFmpeg processes, GPU provisioning, cold starts, job queues, and infrastructure costs that scale linearly with every export. For ClipCrafter -- an open-source app that turns long videos into short, captioned clips -- this was becoming unsustainable.

Our previous architecture relied on Remotion running inside a Docker container on Railway. Every time a user exported a clip, a serverless function would spin up, pull in Chromium, render frames, and stitch them with FFmpeg. It worked, but it was slow, expensive, and fragile.

So we ripped it all out.

Enter framewebworker

In PR #214, we replaced the entire backend rendering pipeline with framewebworker -- a library that renders video frames directly in the browser using Web Workers and the WebCodecs API.

The idea is simple: instead of shipping video data to a server, encode it right where the user already has the video loaded. Modern browsers have surprisingly capable media encoding APIs, and framewebworker wraps them into a clean, composable interface.

What We Removed

The migration was as much about deleting code as writing it. Here is what got removed:

  • 6 API routes related to render jobs, status polling, and artifact downloads
  • Inngest render functions that orchestrated async server-side renders
  • Remotion configuration including the composition, bundle, and all server rendering utilities
  • Chromium and FFmpeg from the Docker image, cutting roughly 200 MB from the container
  • Database columns tracking server-side export state via a new migration

The Dockerfile went on a diet too -- no more chromium, libgbm, or Remotion-specific system packages.

What We Added

The new rendering flow lives entirely in a React hook called useClipRender. Under the hood, it calls framewebworker's render function with a set of video segments built by a new buildVideoSegments helper.

Each segment maps a clip's time range to the source video and overlays caption data with precise timing. The render function handles frame extraction, caption compositing, encoding, and muxing -- all in a Web Worker so the main thread stays responsive.

// Simplified example of the new render flow
const segments = buildVideoSegments(clips, transcript);
const blob = await render({
  source: videoUrl,
  segments,
  format: 'mp4',
  onProgress: (p) => setProgress(p),
});
Enter fullscreen mode Exit fullscreen mode

The result is an MP4 blob that we hand directly to the browser's download API. No round-trips, no polling, no artifact storage.

Real Progress Reporting

One immediate benefit was progress tracking. Server-side renders gave us almost no useful progress data -- we were literally faking it with a simulated progress bar.

With framewebworker, we get genuine RichProgress events that include overall percentage, per-clip status, and phase information. We built a new RenderStatusPanel component that shows each clip's progress with spinner icons and a dynamic ETA that activates once rendering passes 2 percent.

Performance and Cost Impact

Early benchmarks are promising. For a typical 60-second clip with captions:

  • Server render: 45-90 seconds depending on cold start and queue depth
  • Browser render: 15-30 seconds on a mid-range laptop

The cost difference is even more dramatic. Server rendering required a persistent or on-demand container with enough CPU and memory to run Chromium. Browser rendering requires the user's existing browser tab. Our infrastructure bill for rendering dropped to zero.

Trade-offs

This approach is not without downsides. Browser rendering depends on the user's hardware -- a low-end Chromebook will be slower than our server was. The WebCodecs API is not available in all browsers yet, and Safari support is still catching up. Very long videos can hit memory limits.

For our use case -- short clips typically under 3 minutes -- the trade-offs are well worth it. We added a browser compatibility check that warns users if their browser does not support WebCodecs and falls back gracefully.

What This Means for Open Source Video Tools

This migration shows that browser-based video rendering is production-ready for many use cases. If you are building a video tool and dreading the infrastructure complexity of server-side rendering, consider whether your encoding work can happen client-side.

The key enablers are Web Workers for off-main-thread processing, WebCodecs for hardware-accelerated encoding, and libraries like framewebworker that abstract the complexity.

ClipCrafter is fully open source at github.com/clipcrafterapp/clipcrafter-app. Check out the PR and let us know what you think.

This post is part of our series on building ClipCrafter in the open. Follow along for more deep dives into real-world Next.js, video processing, and open-source development.

Top comments (0)