The Problem with Server-Side Video Rendering
If you've ever built a video editing app, you know the pain: users create clips, hit "render," and then... wait. The video gets shipped to a server, processed with ffmpeg or Remotion, and eventually comes back. It's slow, expensive, and creates a terrible user experience.
At ClipCrafter, we lived with this architecture for months. Our rendering pipeline involved Inngest background functions, six dedicated API routes, a beefy Docker image with Chromium and Remotion dependencies, and a Deno runtime layer. Every render meant a round-trip to the server, and our infrastructure costs were climbing.
Last week, we ripped it all out.
Enter framewebworker
The key insight was simple: modern browsers are powerful enough to handle video rendering directly. We replaced our entire backend rendering stack with framewebworker, a lightweight library that processes video frames in a Web Worker.
The migration touched almost every layer of the app. Here's what changed in our biggest PR of the quarter (#214):
What We Deleted
- All Inngest background render functions
- Six API routes dedicated to render orchestration
- Remotion bundler and renderer packages
- Chromium and system-level dependencies from our Docker image
- Deno runtime from the worker container
- Render-related database columns via a new migration
That last point is worth emphasizing. We didn't just swap libraries — we removed an entire category of server state. No more tracking render jobs, polling for completion, or handling failed renders on the backend.
What We Added
The replacement is a single React hook: useClientStitch. It calls framewebworker's API directly in the browser, passing video segments with caption overlays and timing data. The rendering happens in a Web Worker, so the main thread stays responsive.
// Before: complex server orchestration
const response = await fetch('/api/render', {
method: 'POST',
body: JSON.stringify({ clipIds, options })
});
// Then poll for completion...
// After: direct client-side rendering
const { render } = useClipRender();
await render(segments, {
onProgress: (progress) => updateUI(progress)
});
The difference in developer experience is night and day.
The Follow-Up: Rich Progress UI
Once rendering moved client-side, we could do something that was nearly impossible before: show real, granular progress. In the server model, the best we could offer was "processing..." with maybe a percentage that updated every few seconds.
With framewebworker running locally, we built a RenderStatusPanel (PR #216) that shows per-clip progress with spinner and check icons, an overall progress bar with calculated ETA based on elapsed time, and real RichProgress data with per-clip metrics.
We went from a vague loading spinner to a detailed breakdown of exactly what's happening and when it will finish.
Iterating on the API
The migration wasn't a single PR. After the initial swap, we refined the integration across several follow-up changes.
PR #217 bumped framewebworker to v0.1.1 and wired up the real RichProgress types, removing our local progress simulation. Then PR #218 migrated from the stitch() API to a new render() function that loads a single video once instead of re-loading per segment. This introduced buildVideoSegments() to map clips to framewebworker segments with captions and timing.
Results
Docker Image Size
Removing Chromium, Remotion, and Deno shaved roughly 200 MB off our worker Docker image (PR #215). Faster deploys, lower storage costs.
Infrastructure
Six API routes and all background job infrastructure: gone. Our server is simpler, cheaper to run, and has fewer failure modes.
User Experience
Renders start instantly — no upload, no queue, no polling. Users see exactly what's happening with per-clip status and a real ETA.
Takeaways for Your Projects
If you're building a media processing app, consider whether the browser can handle more than you think. Web Workers and modern APIs like OffscreenCanvas and WebCodecs are closing the gap with server-side processing for many use cases.
The key questions to ask: Does every user need to render, or just a few? (If most users render, client-side saves massive server costs.) Are your renders CPU-bound or GPU-bound? (Browsers have GPU access.) Can you tolerate slightly longer render times in exchange for zero infrastructure?
For ClipCrafter, the answer to all three made client-side rendering a clear win.
ClipCrafter is an open-source video clipping tool. Check out the repo at github.com/clipcrafterapp/clipcrafter-app and give it a star if this kind of engineering interests you.
Top comments (0)