DEV Community

Cover image for Don't Block the Event Loop: Scaling Heavy Video Rendering with Node.js, Redis & BullMQ
RodrigoHernandezVidal
RodrigoHernandezVidal

Posted on

Don't Block the Event Loop: Scaling Heavy Video Rendering with Node.js, Redis & BullMQ

#node #architecture #redis #backend

When I first started building Foog Animation Studio (part of our SaaS ecosystem at Foog Technology), I hit a massive wall: Video rendering is computationally expensive.

If you try to process a 1080p video with transitions using FFmpeg directly inside your main Express controller, your Node.js Event Loop will immediately block. Your server will stop responding to other users, APIs will timeout, and your app will effectively crash under pressure.

Here is how we solved this architectural nightmare and built a system that scales seamlessly.

❌ The Naive Approach (What NOT to do)

As developers, our first instinct is often to just exec a child process and wait for it.

// 🚨 Anti-Pattern: Blocking the main thread visually (even if async, it eats resources)
app.post('/api/render', async (req, res) => {
  const { images, text } = req.body;

  // Running heavy FFmpeg tasks right in the API layer
  await utils.renderVideoLocally(images, text); 

  res.json({ status: "done", videoUrl: "/videos/result.mp4" });
});
Enter fullscreen mode Exit fullscreen mode

The Problem: If 10 users click "Render" simultaneously, your server spins up 10 FFmpeg instances. Your CPU spikes to 100%, OOM (Out of Memory) kills the node process, and everyone gets a 502 Bad Gateway.

✅ The Architect Approach: Event-Driven Microservices

To fix this, we decoupled the API Layer from the Processing Layer. We introduced a message broker (Redis) and a robust queue system (BullMQ).

Here is the high-level architecture of our Video Engine:

  1. Client sends an HTTP request to create a video.
  2. Express Server validates the payload, saves a "Pending" record in a PostgreSQL database (via Sequelize), and pushes a Job to the BullMQ Queue.
  3. Express Server immediately replies to the Client: HTTP 202 Accepted + Job ID.
  4. Worker Server (a completely separate Node process) picks up the Job from Redis.
  5. Worker runs fluent-ffmpeg, renders the video, uploads it to S3/Cloud Storage, and updates the database.
  6. Client polls or receives a Webhook/WebSocket event notifying that the video is ready.

Why BullMQ + Redis?

We chose BullMQ because it’s backed by Redis and provides out-of-the-box features that are critical for Enterprise SaaS:

  • Concurrency Control: We can limit workers to process only 2 videos at a time per CPU core.
  • Retries & Backoff: If rendering fails (e.g., a corrupted image), BullMQ retries automatically 3 times with exponential backoff.
  • Resilience: If the Worker server crashes mid-render, the job is not lost. It goes back to the queue.

📦 The Minimal Implementation

Here is a simplified version of our decoupled architecture.

1. The API Controller (The Producer):

import { Queue } from 'bullmq';
import { connection } from '../config/redis';

// Create the Queue
const videoQueue = new Queue('videoRenderQueue', { connection });

export const requestVideoRender = async (req, res) => {
  const { projectId, assets } = req.body;

  // Add job to the queue
  const job = await videoQueue.add('renderTask', { projectId, assets });

  // Respond immediately! Don't wait for FFmpeg.
  return res.status(202).json({ 
    message: "Rendering started", 
    jobId: job.id 
  });
};
Enter fullscreen mode Exit fullscreen mode

2. The Worker Process (The Consumer):

import { Worker } from 'bullmq';
import { renderEngine } from './ffmpegTask';
import { connection } from '../config/redis';

// Note: Concurrency set to 2 to avoid melting the CPU
const videoWorker = new Worker('videoRenderQueue', async (job) => {
  console.log(`Processing job ${job.id} for project ${job.data.projectId}`);

  try {
     // Heavy FFmpeg lifting happens here
    const videoUrl = await renderEngine(job.data.assets);

    // Update DB to "Completed"
    await DB.Project.update({ status: 'DONE', url: videoUrl }, { where: { id: job.data.projectId }});

  } catch (error) {
    throw error; // Let BullMQ handle retries
  }
}, { connection, concurrency: 2 });

console.log("👷 Video Worker listening for jobs...");
Enter fullscreen mode Exit fullscreen mode

🎯 The Business Impact

By separating the API from the Worker, we achieved:

  1. Zero Downtime: The Express API responds in less than 50ms, regardless of how many videos are rendering.
  2. Infinite Scalability: If our client base doubles, we don't touch the API server. We just spin up a second Worker Server on AWS to consume the Redis queue.
  3. Better UX: Users see a nice progress bar instead of a spinning wheel of death waiting for a 2-minute HTTP request.

As SaaS Architects, our job isn't just to make things work; it's to make them resilient.

If you are building complex B2B systems or heavy processing engines, always treat your main thread like a king: Keep it free and let the workers do the heavy lifting.

I’m Rodrigo Hernández, CEO & Lead Architect at Foog Technology. We build high-performance B2B SaaS and Enterprise systems. If you found this architectural breakdown useful, let's connect!

Top comments (0)