I closed every VideoFrame and memory still climbed by gigabytes

I build small tools for a content team, and a lot of them come down to the same job: open a video, do something to every frame, save the result. In a browser you hand the compressed video to a VideoDecoder, which turns it back into full pictures, and each picture arrives as a VideoFrame. Every guide attaches one rule: call close() when you are done with a frame, because it is big. On my 1080p sample one frame reported 3,110,400 bytes, about 3.1 MB.

So I wrote the loop by the book. The sample is a clip of waves from Wikimedia Commons that I converted to H.264 MP4 with ffmpeg 7.1 and looped end to end into 60 seconds of 1920×1080, which makes 1,500 frames. It is a stitched loop, not real long footage. I unpacked it with mp4box.js 0.5.2 and handed every compressed chunk to the decoder in one go. Decoded frames went into an array, and a second loop took them out one at a time: draw on a small canvas, read one pixel, wait 10 ms to stand in for real work, close().

const waiting = [];
const decoder = new VideoDecoder({ output: f => waiting.push(f), error: console.error });
decoder.configure(cfg);
for (const chunk of chunks) decoder.decode(chunk); // all 1,500, no questions asked

In Chromium 149 all 1,500 frames came out, nothing was logged, and every one of them got its close(). System memory still peaked 3,752 MB above where it started. That reading is noisy, and the same scene can land more than 1 GB apart between two runs, but it is gigabytes either way.

The reason turned up when I ran the plainest version, the one that closes each frame the moment it arrives. The same 1,500 frames were through in 752 ms. Decoding the whole clip takes under a second, and my pretend work takes 10 ms per frame. The array in the middle is where the difference went: at its worst it held 1,421 frames that were decoded and not yet processed. Each one did get closed when its turn came. Until then it was a full picture in memory.

It reminds me of moving flats. The driver empties the van onto the pavement in minutes, and I carry one box upstairs at a time. Every box gets there in the end, but for most of the afternoon the pavement is covered in my stuff. close() is me putting a box away. It says nothing about how fast the driver unloads.

The red line is the loop above. It shoots up to about 1,400 waiting frames in the first couple of seconds and needs the rest of the 20 seconds to come back down. The blue line is the same work with the feed paced, which means I check two numbers before handing over the next chunk. One is decoder.decodeQueueSize, the count of chunks the decoder has accepted and not decoded yet. The other is the length of my own array.

for (const chunk of chunks) {
  while (decoder.decodeQueueSize > 8 || waiting.length > 4) await nap(2); // nap = setTimeout as a promise
  decoder.decode(chunk);
}

With that one extra line the backlog peaked at 17 frames instead of 1,421, and the system memory peak was 165 MB. The total time barely moved, 18.9 s against 20.3 s, which I read as "about the same" and nothing more.

What threw me is where the memory was. In the unpaced run the browser's own process memory, summed over all its processes, changed by 0 MB. On macOS the frames sat in graphics buffers, blocks of memory the system shares with the graphics side and calls IOSurface. Running ioclasscount IOSurface in a terminal counts them.

Look at the fourth and fifth bars: 1,488 extra buffers for the unpaced loop, 29 for the paced one. On a 4K clip of a river I only fed 400 frames to spare the machine, and the backlog peaked at 332 without pacing and 16 with it.

For comparison I wanted a per-frame pipeline that somebody had already shipped. For that I used ImgIng (https://imging.ai/). Its video matting tool is still labeled Beta, and the site says desktop Chrome and Edge are supported first. I ran it on the same 1,500-frame wave clip in Chromium 149. Sampled every 2 seconds for the whole job, the graphics-buffer count stayed between −2 and 79 above its starting level and did not rise with progress. Waves have no subject to cut out, so this is about memory and says nothing about matting quality.

This is what the tool looks like while it works. There is nobody in the wave clip, so I took the screenshot with a different one: 8 seconds of a street guitarist in Ljubljana, video by Isak Zvegelj on Wikimedia Commons (CC BY 3.0). I did not re-measure memory with it.

One thing I can't explain yet. In the unpaced run the backlog drained from 1,403 frames down to 4, but the buffer count stayed between 1,430 and 1,490 the whole time and only fell when the browser quit. I haven't looked into why. Everything here is from one Apple M4 with 16 GB on macOS, and I don't know where frames live on other systems. No run fell over, so I can't say where the ceiling is. What I do now is log the peak of waiting.length next to the frame count. If the two are close, the frames are queuing.

The wave footage is by David Shay (דוד שי) and the river footage by Nesnad, both from Wikimedia Commons under CC BY-SA 4.0.