You write a loop that decodes a video and does something to every frame: a strip of thumbnails, a watermark, a filter. It runs fine on a short clip and the tab looks small in the browser's task manager. I run a small tool on my own and would rather do this work in the visitor's browser than rent a bigger server, so I wanted to know what that loop really costs. Here are the five things I now check, each with a number from my own test.
Some terms first. Decoding means turning compressed video back into full pictures, and VideoDecoder is the browser API that hands them to you one by one as VideoFrame objects. A graphics buffer is a block of image memory handed out by the operating system; macOS calls it an IOSurface and can count them system-wide. I tested on an Apple M4 with 16 GB running macOS, mostly in Chromium 149, reading the MP4 with mp4box.js 0.5.2. The clip is not a native long video: I took an example video from Wikimedia Commons, transcoded it with ffmpeg 7.1 and looped it end to end into 60 seconds of 1920×1080, which is 1500 frames.
1. Know how big one frame is. File size tells you nothing here. For a decoded 1920×1080 frame, allocationSize() read 3,110,400 bytes, about 3.1 MB. A 3840×2160 frame works out to 12,441,600 bytes by the same math, about 12.4 MB. That one is calculated, not read. Holding on to just 300 frames of a 4K clip pushed system memory up by 4546 MB, and I stopped at 300 to protect the machine.
The blue bar is the value I read. The red one is the calculated one, four times as long.
2. Close every frame. When I called close() on each frame as soon as it arrived, all 1500 frames left the system with 12 extra graphics buffers. Pushing them into an array instead left 1508 extra and system memory peaked 2694 MB higher. I trust the buffer count more than the megabytes, because the same scenario could differ by more than 1 GB of system memory between two runs.
3. Pace the decoder when your work is slower than decoding. I faked slow work by drawing each frame to a small canvas, reading one pixel, waiting 10 ms and then closing it. When I fed the whole file to the decoder in one go, the line of decoded frames waiting for me peaked at 1421, with 1488 extra buffers and 3752 MB more system memory. Then I made the feeding loop wait whenever too much was queued:
while (vd.decodeQueueSize > 8 || waiting.length > 4) {
await pause(2);
}
vd.decode(piece);
decodeQueueSize is how many compressed chunks the decoder has not got to yet, and waiting is my own array of unprocessed frames. With that check the line peaked at 17 frames, with 29 extra buffers and 165 MB more system memory, and the two runs finished in about the same total time. One thing I can't explain: in the unpaced run every frame did get closed after processing, yet the buffer count sat between 1430 and 1490 until the browser quit. I did not dig into why.
4. Don't count on garbage collection. I also tried neither closing a frame nor keeping a reference to it. In Chromium 149 the 1508 extra buffers were still there after the 6 seconds I kept sampling, and Chrome 154 and Firefox 151 did not give the memory back either. WebKit 26.5 (macOS) did reclaim it, down to 32 extra buffers, and printed "A VideoFrame was destroyed without having been closed explicitly" in the console. What happens after 6 seconds I don't know.
5. Watch system memory, not the browser process. In the Chromium run that kept all 1500 frames, the browser's process memory grew by 138 MB while 1508 buffers piled up where that number doesn't show them. Firefox 151 was the opposite, with process memory up 2291 MB and only 7 extra buffers. On macOS, ioclasscount IOSurface prints the system-wide count.
To compare with a pipeline somebody else built, I used ImgIng (https://imging.ai/), whose video matting goes through a clip frame by frame in the browser. The feature is in Beta and supports desktop Chrome and Edge first. I ran it on the same 1500-frame clip in Chromium 149. Its extra graphics buffers stayed between −2 and 79 throughout and did not climb with progress. The clip is ocean waves with nothing in it to cut out, so this tells me about memory and nothing about matting quality.
A look at the tool itself. Matting needs a subject and the wave clip has none, so the screenshots are from another clip, 8 seconds of a street guitarist (video by Isak Zvegelj, Wikimedia Commons, CC BY 3.0): imported on top, background replaced below. The buffer counts were not taken on this clip.
The line hits its peak of 79 at the start, then settles in the twenties and stays flat to the end.
Not covered: Windows, phones, codecs besides H.264. Where frames live on those systems I have not tested. Nothing crashed in any run, so I can't say where the limit is, and I did not measure frame rate. My habit from here is cheap: count the buffers, run 1500 frames, count again. If the difference looks like the frame count, a frame is not being closed. The wave footage is from a video by David Shay (דוד שי) and the 4K river footage from one by Nesnad, both on Wikimedia Commons under CC BY-SA 4.0.
Top comments (0)