Building a Smooth 60fps Video Player in Node.js.

I had to bring back Big Buck Bunny for this one.

If you were around during the 2018 Blender / creative-coding boom, you already know this video.

What you’re seeing below is Big Buck Bunny playing entirely inside Node.js.

Still holding 60 FPS, frame after frame.

You can clone the repo and try it yourself:

git clone https://github.com/sklyt/nodejs-videoplayer.git
cd nodejs-videoplayer && npm i
node video_player/main.js "path/to/any/video/file"

• But… how is this even possible?
• The Renderer
• The Decoder
• The Hard Part: Synchronization
• The Core Problem: Time Drift
  • Approach 1 (naïve)
  • Approach 2 (correct)
• The Fix: Absolute Time
• Final Result

---

But… how is this even possible?

Simple idea, really.

A video is just a sequence of images.

An image is just a grid of pixels.

A pixel looks like this:

const pixel = { r, g, b, a }

Or more accurately, as linear memory:

Uint8Array([r, g, b, a])

A collection of these pixels forms an image, a frame.

So if we can decode a video into raw frame pixels, all we need is a renderer that knows how to draw them fast.

We already have both.

---

The Renderer

I built tessera.js, a zero-copy, raw-buffer renderer for Node.js.

• 0 ms buffer swap
• ~0.02 ms draw time
• Takes pixels. That’s it.

How I Built a Renderer For Node.js
https://github.com/sklyt/tessera.js/tree/main

Here’s a simple example writing directly into a pixel buffer:

function generateSimplexNoise(data, width, height, options = {}) {
  const scale = options.scale || 0.01;

  for (let y = 0; y < height; y++) {
    for (let x = 0; x < width; x++) {
      const idx = (y * width + x) * 4;
      const value = Math.sin(x * scale) * Math.cos(y * scale);
      const normalized = (value + 1) * 0.5 * 255;

      data[idx]     = normalized;
      data[idx + 1] = normalized;
      data[idx + 2] = normalized;
      data[idx + 3] = 255;
    }
  }
}

generateSimplexNoise(cache.data, canvas.width, canvas.height);

If you can write pixels, tessera can render them.

---

The Decoder

ffmpeg, the greatest video tool ever created.

ffmpeg can decode a video straight into raw RGBA frames, piped directly to stdout:

// decoder.js
const ff = spawn("ffmpeg", [
  "-i", videoFile,
  "-vf", `scale=${width}:${height}`,
  "-r", String(fps),
  "-f", "rawvideo",
  "-pix_fmt", "rgba",
  "-threads", "1",
  "pipe:1"
], { stdio: ["ignore", "pipe", "pipe"] });

Node spawns ffmpeg, ffmpeg streams raw frames, and Node feeds them into tessera.

The “magic” lives in main.js, inside the loop:

if (available > 0) {
  const slotOffset = framesBase + readIdx * frameSize;
  const src = new Uint8Array(shared, slotOffset, frameSize);

  if (canvas.data.set) canvas.data.set(src);
  else Buffer.from(shared).copy(canvas.data, 0, slotOffset, slotOffset + frameSize);

  canvas.markRegion(0, 0, canvas.width, canvas.height);

  readIdx = (readIdx + 1) % FRAME_COUNT;
  Atomics.store(meta, 1, readIdx);
  lastFrameRead++;

  canvas.upload(); // mark for swap
}

This single line:

canvas.data.set(src);

copies whatever frame is most recent from ffmpeg straight into the renderer.

---

The Hard Part: Synchronization

Rendering wasn’t the hard part.

Timing was.

ffmpeg runs as fast as it possibly can.

On a good machine, it can decode hundreds of frames per second.

But we only want 60 FPS.

That creates a few problems:

• Buffers fill up
• Frames get dropped
• Playback jumps ahead
• Renderer starts chasing ffmpeg

The naïve solution is something like:

setTimeout(loop, 1000 / 60);

I tried that.

Bad idea.

---

The Core Problem: Time Drift

Imagine clapping once per second.

Approach 1 (naïve)

• Wait 1 second, clap
• Wait 1 second, clap
• Wait 1 second, clap

Approach 2 (correct)

• Note the start time
• Clap at start + 1s
• Clap at start + 2s
• Clap at start + 3s

Why does approach 1 fail?

Because setTimeout isn’t precise.

If “1 second” is actually 1.01s, after 100 claps you’re a full second late.

Your code starts dropping frames to “catch up”, but it’s still chasing a drifting clock, so it never truly syncs.

That’s exactly what was happening here.

---

The Fix: Absolute Time

Instead of waiting between frames, I anchor everything to a single absolute start time.

This runs in the worker:

const startTime = Date.now(); // absolute reference point

function tick() {
  const targetTime = startTime + (frameNumber * targetMs);
  const now = Date.now();
  const drift = now - targetTime;

  // If we're more than 2 frames behind, skip ahead
  if (drift > targetMs * 2) {
    const framesToSkip = Math.floor(drift / targetMs) - 1;
    const actualSkip = Math.min(framesToSkip, pendingFrames.length);

    if (actualSkip > 0) {
      pendingFrames.splice(0, actualSkip);
      frameNumber += actualSkip;
      Atomics.add(meta, 3, actualSkip);
    }
  }

  const frame = pendingFrames.shift();
  if (frame) {
    const writeIdx = Atomics.load(meta, 0);
    const nextWrite = (writeIdx + 1) % frameCount;
    const readIdx = Atomics.load(meta, 1);

    // Overwrite oldest frame if buffer is full
    if (nextWrite === readIdx) {
      Atomics.store(meta, 1, (readIdx + 1) % frameCount);
      Atomics.add(meta, 3, 1);
    }

    const slotOffset = framesBase + writeIdx * frameSize;
    new Uint8Array(shared, slotOffset, frameSize).set(frame);

    Atomics.store(meta, 0, nextWrite);
    Atomics.add(meta, 2, 1);
    frameNumber++;
  }

  // Resume ffmpeg if we’ve drained enough frames
  if (pendingFrames.length < 5 && ffmpegPaused) {
    ff.stdout.resume();
    ffmpegPaused = false;
  }

  const nextFrameTime = startTime + (frameNumber * targetMs);
  const delay = Math.max(0, nextFrameTime - Date.now());
  setTimeout(tick, delay);
}

tick();

Every frame knows exactly when it should exist relative to the start of playback.

No drifting.

No chasing.

---

Final Result

• ffmpeg decodes as fast as it wants
• Node buffers intelligently
• Renderer consumes frames at a fixed cadence
• Time never drifts

Once everything is anchored to absolute time…

…it just works.