I spent the last few months building a video upscaler that runs entirely in the browser — every frame is demuxed, decoded, pushed through a WebGPU shader or a small neural net, and re-encoded, without the file ever leaving the machine. WebCodecs is what makes that possible at all, and I want to say up front that it is a genuinely good API. The surface is small, the MDN pages are accurate, and VideoDecoder / VideoEncoder do exactly what they say.
What the docs don't cover is the shape of a pipeline. Each of the four things below cost me a day or more, and every one of them found me the same way: the code worked on my test clip and fell over on a real file.
None of these are WebCodecs bugs. Three of them are consequences of the API being honest about hardware, and one is a plain JavaScript mistake that only shows up under concurrency.
The pipeline, in shape
So we're talking about the same thing. Frames come from a demuxer feeding VideoDecoder, not from playing an off-screen <video> element:
for await (const sample of videoSink.samples()) {
const source = sample.toCanvasImageSource();
await engine.renderFrame(source); // GPU work
await videoSource.add(new VideoSample(engine.canvas, {
timestamp: sample.timestamp,
duration: sample.duration,
}));
if (source instanceof VideoFrame) source.close();
sample.close();
}
I'm using mediabunny for demuxing and muxing; the videoSink / videoSource names come from it. If you're driving VideoDecoder and VideoEncoder directly the structure is the same, and I'll point out the raw equivalent where it matters.
The <video> route is simpler and I started there. It has one root cause with three consequences: the audio track is unreachable, throughput is capped at 1× playback speed, and a backgrounded tab stalls forever, because requestVideoFrameCallback only fires while the tab is visible. Demuxing removes all three at once. It also puts you in charge of things the <video> element was quietly handling, which is most of the rest of this post.
1. Backpressure is a single await — and without it you run out of memory
Here is the entire backpressure mechanism in that loop:
await videoSource.add(sample);
That await resolves only once the encoder and the muxer's writer are ready for more. Drop it, and the loop pulls frames as fast as the decoder produces them. Decoding is much faster than encoding — especially when the encoder is writing frames four times the size of the source — so the gap becomes a queue of VideoFrame objects, each holding a GPU-side buffer, and the tab dies.
The failure is nastier than "slow": it scales with clip length, so a 5-second test passes and a 3-minute upload gets you a blank crash with no stack.
With raw WebCodecs, the lever is encoder.encodeQueueSize plus the dequeue event:
async function waitForRoom(encoder, limit = 4) {
while (encoder.encodeQueueSize > limit) {
await new Promise(r => encoder.addEventListener('dequeue', r, { once: true }));
}
}
The number matters less than the fact that there is one. The point is that pressure from the encoder has to travel back up to the decoder, and nothing in the API does that for you — encode() returns undefined, not a promise, and will happily accept a thousand frames.
There are actually three queues to bound in a full pipeline: decode depth, frames in flight through the GPU, and encode depth. I only ever needed to bound the third, because awaiting it stalls the loop and therefore starves the other two. That's worth knowing before you build three separate semaphores.
2. Your first audio packet has a negative timestamp
This one produced a spectacularly unhelpful error: adding the very first audio packet to the output track threw, on ordinary MP4s straight off a phone.
The cause is AAC encoder delay, usually called priming. An AAC encoder needs a run-up of samples before its first real output, so the encoded stream starts slightly before the media does. MP4 expresses this with an edit list, and a demuxer that honours the edit list reports the first packet at a negative presentation time. Output tracks won't accept one.
The obvious fix is to clamp it to zero. That's wrong: A/V sync is not the absolute timestamps, it's the offset between the two tracks. Clamp only the audio and you've shifted audio relative to video by ~20ms, which is small enough that you won't notice it in testing and large enough that someone will notice it on a talking-head clip.
Shift both tracks by the same amount:
const firstTimestamp = await input.getFirstTimestamp().catch(() => 0);
const shift = firstTimestamp < 0 ? -firstTimestamp : 0;
// video
new VideoSample(canvas, { timestamp: sample.timestamp + shift, duration: sample.duration })
// audio — same shift, no decode
const shifted = shift > 0 ? packet.clone({ timestamp: packet.timestamp + shift }) : packet;
While we're here: don't decode the audio at all. If the output container supports the source's audio codec, copy the encoded packets straight across. It costs nothing, loses nothing, and removes an entire class of resampling bugs:
const audioTrack = await input.getPrimaryAudioTrack();
const canCarry = audioTrack
&& output.format.getSupportedCodecs().includes(audioTrack.codec);
That check has teeth, and it's how I found the thing that became a whole separate post: if your video encoder falls back from H.264 to VP9, the container has to become WebM, and WebM can't hold AAC. The passthrough silently becomes impossible and the audio disappears. Which brings up the general rule — the codec decides the container, the container decides what audio you can keep, so those two decisions can't be made independently.
3. A VideoFrame held across an iteration deadlocks the decoder
Frame interpolation — synthesising a frame between two real ones to double the frame rate — needs frame N-1 and frame N at the same time. So the natural thing is to keep a reference:
// deadlocks
let previous = null;
for await (const sample of videoSink.samples()) {
if (previous) await engine.renderBetween(previous, sample);
previous = sample; // never closed before the next pull
}
This hangs. VideoFrame holds a slot in the decoder's output pool, and that pool is small and fixed. You have to close each frame before pulling the next one, or the decoder fills its output queue and stops producing. Hold one across an iteration and the loop waits for a frame the decoder can't emit because you're holding the buffer it needs.
And you can't close it and keep reading it either — a closed VideoFrame throws on access. There's no version of "keep the reference" that works.
The fix is to stop holding the decoder's frame and hold your own copy:
const held = new OffscreenCanvas(srcW, srcH);
const heldContext = held.getContext('2d');
let previous = null;
for await (const sample of videoSink.samples()) {
const source = sample.toCanvasImageSource();
try {
if (previous) {
const half = (sample.timestamp - previous.timestamp) / 2;
await emit(held, previous.timestamp, half); // the real frame
await engine.renderBetween(held, source); // the invented one
await emit(engine.canvas, previous.timestamp + half, half);
}
heldContext.drawImage(source, 0, 0);
previous = { timestamp: sample.timestamp, duration: sample.duration };
} finally {
if (source instanceof VideoFrame) source.close();
sample.close();
}
}
drawImage into an OffscreenCanvas is a real copy into storage you own, so the decoder's frame closes on schedule.
Two things fall out of this that aren't obvious:
The loop now runs one frame behind, and it has to. The frame between A and B can't be produced until B arrives — and A's duration isn't known to be halved until then either. So each iteration emits the previous frame at half the gap, then the frame invented to fill the other half. After the loop, the last real frame keeps its own full duration, because there's nothing after it to interpolate towards. Forget that tail and your output is one frame short and ends abruptly.
Watch out for detached methods. I refactored engine.renderBetween into a local const renderBetween = engine.renderBetween and got a this-is-undefined error at frame zero — it's a class method, and pulling it off the object drops the receiver. Textbook JavaScript, thoroughly invisible in a file that is otherwise about codecs.
4. When the muxer is cancelled, every error becomes the same error
Video and audio are pumped concurrently:
await Promise.all([pumpVideo(), pumpAudio()]);
This is where I lost the most time, and it isn't a WebCodecs issue at all — it's a Promise.all issue that concurrency in this shape makes lethal.
When the video pump throws (say, the encoder rejects a 4K frame), the error handler cancels the muxer. Cancelling it makes every other pending operation on it reject too, with Output has been canceled. Now two promises are rejected: the real one and the consequence. Promise.all gives you whichever arrived first, and that is routinely the consequence — the cancellation propagates in microtasks while the original rejection is still unwinding.
So for weeks, every 4K failure reported "Output has been canceled" and nothing about why anything was cancelled. I had an error message that described my own cleanup code.
allSettled, then pick the cause over the consequence:
const settled = await Promise.allSettled([pumpVideo(), pumpAudio()]);
const failures = settled
.filter(r => r.status === 'rejected')
.map(r => r.reason);
if (failures.length > 0) {
const message = e => (e instanceof Error ? e.message : String(e));
throw failures.find(e => !/has been canceled/i.test(message(e))) ?? failures[0];
}
Matching on an error string is not something I enjoy writing, and I'd swap it for a typed error the moment the library exposes one. But the generalisable part isn't the regex — it's this: any time a failure in one branch tears down a resource the other branches are waiting on, Promise.all will show you the teardown instead of the fault. Reach for allSettled and rank the reasons.
Bonus: the tab that isn't there
One hazard survives all of the above. A VideoDecoder left too long in a hidden tab can be reclaimed by the browser. It doesn't throw a nice error; the pump just stops.
A watchdog turns that into a sentence someone can act on rather than a spinner that never ends:
const STALL_MS = 15_000;
let lastProgressAt = performance.now();
const watchdog = setInterval(() => {
if (performance.now() - lastProgressAt < STALL_MS) return;
clearInterval(watchdog);
void output.cancel();
}, 1000);
With one caveat I'd underline. When something does throw, only blame the hidden tab when the browser actually says it reclaimed something:
const reclaimed = err instanceof DOMException
&& (err.name === 'QuotaExceededError' || err.name === 'AbortError');
Attributing every failure to a backgrounded tab is how a real bug gets papered over with a plausible sentence. The message would be wrong, and the user would go and act on it.
What I'd tell myself at the start
- Bound one queue, not three. Awaiting the encoder stalls the loop and starves everything upstream for free.
- Never clamp one track's timestamps. Sync is the offset, so any shift applies to both.
-
Own your copies.
VideoFrameandImageBitmapare borrowed from a small pool; anything you keep across an iteration must be yours. - Rank your errors. In a concurrent teardown, the loudest rejection is almost never the interesting one.
These are all measured on my own machine and my own stack (Chrome, WebGPU, mediabunny) — the shapes generalise, the specific numbers may not. If you've built something similar and hit a fifth one, I'd genuinely like to hear it; this is a corner of the platform where there's very little written down.
I ran into all of this building FreeUpscaler, which upscales and cleans up video and images entirely in the browser on WebGPU — no upload, which is the reason the whole pipeline has to be client-side in the first place.
Top comments (0)