I used to treat a backlog of decoded frames as a temporary cost. If the slow step falls behind, frames wait in an array, each one gets close() when its turn comes, and the memory comes back as the array shrinks. When I measured it on Chromium 149, the second half of that didn't hold. The array shrank to 4 frames, and the number of graphics buffers the operating system was holding stayed where it had peaked. Releasing late did not give the memory back right away, and that changed where I put the pacing check.
The clip is a sample video of ocean waves from Wikimedia Commons, transcoded to H.264 in ffmpeg 7.1 and looped end to end until it was 60 seconds long: 1920×1080, 1,500 frames. mp4box.js 0.5.2 pulls the compressed samples out of the MP4, and VideoDecoder turns them into frames. One decoded frame reports 3,110,400 bytes from allocationSize(), about 3.1 MB. The slow step is a stand-in for real work: draw the frame to a small canvas, read one pixel, wait 10 ms, then close(). Everything ran in Chromium 149 on an Apple M4 with 16 GB of RAM, on macOS.
Twice a second I sampled the browser's process memory, the system's memory use, and the number of graphics buffers on the system. A graphics buffer here is what macOS calls an IOSurface, a block of pixel memory that the system keeps track of, and ioclasscount IOSurface prints how many exist across the whole machine. The system memory figure could differ by more than 1 GB between two runs of the same scenario, so I lean on the buffer count, which was the steadiest of the three.
The backlog drained but the buffers stayed
The first version handed every compressed chunk to the decoder in one go:
const backlog = [];
const vdec = new VideoDecoder({
output: vf => backlog.push(vf), // waits here for the slow step
error: console.error,
});
for (const unit of units) vdec.decode(unit); // all 1,500 at once
Frames came out much faster than a 10 ms step could take them, so they piled up. The array of decoded frames waiting for the slow step peaked at 1,421. The extra graphics buffer count climbed to 1,488, while the browser's process memory showed an increase of 0 MB.
Then the slow step worked through the pile. Over about 20 seconds the waiting frames went from 1,403 down to 4, and each one was closed right after use. I expected the buffer count to follow that slope down. It stayed between 1,430 and 1,490 for the whole drain, and it only dropped when the browser exited. A separate run where I kept all 1,500 frames in an array and never closed any of them peaked at 1,508 extra buffers. Seen from the buffer count, closing late and never closing looked almost the same.
The red line is the unpaced run: roughly 1,400 frames waiting almost immediately, then a straight slide to near zero at the 20-second mark. The blue line is the paced run. What the chart cannot show is that the buffer count did not slide down with the red line.
Pacing from the first chunk kept the count at 29
The second version checks two numbers before it feeds each chunk. One is decodeQueueSize, the number of chunks the decoder has accepted but not decoded yet. The other is my own backlog.
for (const unit of units) {
while (vdec.decodeQueueSize > 8 || backlog.length > 4) await pause(2);
vdec.decode(unit);
}
Same clip, same 10 ms step. The waiting frames peaked at 17 and the extra buffer count peaked at 29. The job took 18.9 seconds, against 20.3 seconds for the unpaced run, so the two finished in about the same time.
For a reading from a complete pipeline instead of my 10 ms stand-in, I used ImgIng (https://imging.ai/). Its video matting feature is in Beta and supports desktop Chrome and Edge first. I ran the same 1,500-frame clip through it in Chromium 149 and sampled every 2 seconds. During processing the extra buffer count stayed between −2 and 79, and it did not rise as the job progressed. The clip is waves with nothing in it to cut out, so this tells you about memory over a long job and nothing about matting quality. The matting model is not my area, and I only read the counters, once, on this one clip.
The wave clip has no subject, so it shows nothing of the matting itself. For this screenshot I switched to an 8-second clip of a street guitarist in Ljubljana (video by Isak Zvegelj, Wikimedia Commons, CC BY 3.0). The readings above still come from the wave clip.
What I did not find out
I did not look into why the buffers stayed after close(). I have no explanation for it. I saw it in one scenario, on one browser, on one machine, and I kept the page open for only 6 seconds after the job finished before I stopped sampling. I didn't test other operating systems, where a decoded frame may not live in anything called an IOSurface at all.
The pacing check now goes in before the first decode() call, not after a backlog has already shown up, because in this run draining the backlog did not bring the count back down while the browser was open. And when I check a pipeline I watch the system-wide buffer count, since process memory read 0 MB extra through the whole unpaced run. On macOS that means running ioclasscount IOSurface in a terminal before the job and again while it runs.
The wave footage is by David Shay (דוד שי), from Wikimedia Commons, licensed CC BY-SA 4.0.
Top comments (0)