DEV Community

Cover image for Recording screen, system audio and mic to MP4 in the browser (WebCodecs + mp4-muxer)
Ryan ji
Ryan ji

Posted on AI-assisted

Recording screen, system audio and mic to MP4 in the browser (WebCodecs + mp4-muxer)

I built a small meeting recorder that runs as a PWA in Edge. It records the screen, the other people's voices (system audio) and my own microphone into one MP4. This post is about how it works, including the parts that took me a while to get right.

What I wanted:

  • one MP4 with screen + system audio + mic
  • hours of recording without holding the file in memory
  • files that seek properly in any player

Setup: Windows 11, Microsoft Edge (Chrome behaves the same). APIs: getDisplayMedia, Web Audio, MediaStreamTrackProcessor, WebCodecs, File System Access API. MP4 muxing is done by mp4-muxer (MIT).

Pipeline

Why not MediaRecorder

MediaRecorder is the easy option, but I wanted control over a few things:

  • Seeking. WebM files from MediaRecorder have no duration or cues, so some players can't seek in them.
  • Frame rate and keyframes. Meeting screens barely move, so I record at 8 fps to keep files small, but still want a keyframe every 5 seconds so seeking is quick and accurate.
  • Pause. Paused time should be cut out, not left as a gap.

WebCodecs gives you all of that. The price is that you manage timestamps yourself.

1. Screen and system audio

const displayStream = await navigator.mediaDevices.getDisplayMedia({
  video: { frameRate: { ideal: 8, max: 8 }, height: { max: 1080 }, displaySurface: 'monitor' },
  audio: { suppressLocalAudioPlayback: false, echoCancellation: false, noiseSuppression: false, autoGainControl: false },
  systemAudio: 'include',
  selfBrowserSurface: 'exclude',
  monitorTypeSurfaces: 'include',
});
displayStream.getVideoTracks()[0].contentHint = 'text';
Enter fullscreen mode Exit fullscreen mode
  • On Windows, you only get system audio if the user picks Entire screen and ticks Also share system audio in the share dialog. Sharing a single window gives you no audio. Users won't guess this, so the app says it in one line under the button.
  • Echo cancellation and friends are for microphones. They do nothing useful on system audio, so they're off.
  • contentHint = 'text' tells the encoder to favor sharp detail over smooth motion. Slide text stays readable.

2. Mixing in the microphone

Two separate audio tracks in one MP4 confuse some players (they play only the first). I mix them into one with Web Audio:

const audioCtx = new AudioContext({ sampleRate: 48000 }); // create inside the click handler
const dest = audioCtx.createMediaStreamDestination();
dest.channelCount = 2;

const sys = displayStream.getAudioTracks();
if (sys.length) audioCtx.createMediaStreamSource(new MediaStream(sys)).connect(dest);

const mic = await navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: true, noiseSuppression: true } });
audioCtx.createMediaStreamSource(mic).connect(dest);

const mixedTrack = dest.stream.getAudioTracks()[0];
Enter fullscreen mode Exit fullscreen mode

Create the AudioContext in the click handler. If you create it on page load it can start suspended, and you get a silent file.

3. Encoding video yourself

MediaStreamTrackProcessor hands you frames one by one, which you feed to a VideoEncoder:

const reader = new MediaStreamTrackProcessor({ track: videoTrack }).readable.getReader();
const first = (await reader.read()).value;
const w = even(first.displayWidth), h = even(first.displayHeight); // H.264 needs even sizes

// try H.264 profiles from best to most compatible
const common = { width: w, height: h, bitrate: 500_000, bitrateMode: 'variable', framerate: 8, latencyMode: 'realtime', avc: { format: 'avc' } };
let cfg = null;
for (const codec of ['avc1.640033', 'avc1.4d0033', 'avc1.42e033', 'avc1.640028']) {
  if ((await VideoEncoder.isConfigSupported({ ...common, codec })).supported) { cfg = { ...common, codec }; break; }
}

const venc = new VideoEncoder({
  output: (chunk, meta) => muxer.addVideoChunk(chunk, meta),
  error: e => console.error(e),
});
venc.configure(cfg);
Enter fullscreen mode Exit fullscreen mode

The loop does three things:

const FRAME_US = 1e6 / 8, KEYFRAME_US = 5_000_000;
const base = first.timestamp;  // recording start
let lastTs = -1, lastKeyTs = -Infinity;

for (let frame = first; frame; ) {
  const ts = Math.max(frame.timestamp - base, lastTs + 1);           // 1. keep timestamps increasing
  if (lastTs < 0 || (ts - lastTs >= FRAME_US * 0.75 && venc.encodeQueueSize < 4)) { // 2. drop extras
    const out = new VideoFrame(frame, { timestamp: ts, duration: FRAME_US });
    const key = ts - lastKeyTs >= KEYFRAME_US;                       // 3. keyframe every 5 s
    if (key) lastKeyTs = ts;
    venc.encode(out, { keyFrame: key });
    out.close();
    lastTs = ts;
  }
  frame.close();
  const r = await reader.read();
  frame = r.done ? null : r.value;
}
Enter fullscreen mode Exit fullscreen mode
  1. Timestamps. When nothing on screen changes, frames stop coming for a while. I take each frame's own timestamp minus the start time, and force it to be strictly increasing.
  2. Dropping frames. max: 8 is a hint, not a promise. Frames that arrive too soon, or while the encoder is backed up (encodeQueueSize), get dropped.
  3. Keyframes. Without regular keyframes, seeking is slow and lands in the wrong place.

And close every frame. Forget frame.close() and you run out of memory within seconds.

4. Audio timestamps from the sample count

For audio, I compute the timestamp from how many samples I've written. Paused time simply never gets counted.

let samples = 0;
// inside the loop (ad is an AudioData)
const n = ad.numberOfFrames, ch = ad.numberOfChannels;
const buf = new Float32Array(n * ch);
for (let c = 0; c < ch; c++) ad.copyTo(buf.subarray(c * n, (c + 1) * n), { planeIndex: c, format: 'f32-planar' });
const out = new AudioData({
  format: 'f32-planar', sampleRate: ad.sampleRate, numberOfFrames: n, numberOfChannels: ch,
  timestamp: Math.round(samples / ad.sampleRate * 1e6), data: buf,
});
samples += n;
aenc.encode(out);
out.close(); ad.close();
Enter fullscreen mode Exit fullscreen mode

While paused, incoming AudioData is just thrown away. On resume, the video side recalculates its base time. The result is a file with the paused section cut out.

The audio encoder tries AAC (mp4a.40.2) first and falls back to Opus.

5. Writing to disk as you go

This made the biggest difference. mp4-muxer has FileSystemWritableFileStreamTarget, which writes straight into a File System Access stream:

// ask for a folder once; keep the handle in IndexedDB for next time
const dir = await showDirectoryPicker({ id: 'rec', mode: 'readwrite', startIn: 'videos' });
const file = await dir.getFileHandle('recording.mp4', { create: true });
const writable = await file.createWritable();

const muxer = new Mp4Muxer.Muxer({
  target: new Mp4Muxer.FileSystemWritableFileStreamTarget(writable),
  fastStart: false,               // streaming, so the moov box goes at the end
  firstTimestampBehavior: 'offset',
  video: { codec: 'avc', width: w, height: h },
  audio: { codec: 'aac', numberOfChannels: 2, sampleRate: 48000 },
});

// on stop
await venc.flush(); await aenc.flush();
muxer.finalize();
await writable.close();
Enter fullscreen mode Exit fullscreen mode

With ArrayBufferTarget, the whole recording sits in memory until you stop. A long meeting in high quality can run out of memory and lose everything. I started that way and switched to streaming. In an automated test that records and saves 100 times in a row, memory doesn't grow (renderer process went from 86 MB to 57 MB).

A FileSystemDirectoryHandle can be stored in IndexedDB as-is. Next time you only check queryPermission / requestPermission and reuse the same folder.

6. Smaller gotchas

  • Static screen = short video. If the last frame came long before you pressed stop, the video track ends early. On stop, I encode the last frame once more with the stop time as its timestamp.
  • Double-clicking Start. Users can click again while the share dialog is open. A simple flag prevents two recordings at once.
  • Folder picker uses up the click. showDirectoryPicker consumes the user activation, so calling getDisplayMedia right after it fails. The app asks the user to click once more.

7. Shipping it to the Microsoft Store

PWABuilder turns the URL into a .msixbundle you can submit. The package is just a shell that opens the site, so code updates are a web deploy with no Store re-review (remember to bump the service worker cache name).

A side effect I didn't plan for: because it installs from the Store like any standard app, with no drivers or virtual audio devices, it also works on locked-down work PCs where you can't install regular desktop software. (Some companies block the Store itself, and please follow your company's policy and tell people when you record.)


The app is called LiteRec, and I'm the developer. 10 recordings are free, then it's $0.99/month.
https://apps.microsoft.com/detail/9ntvppczz2gf

Happy to answer questions about any of the WebCodecs bits in the comments.

Top comments (0)