restyvod

Posted on Oct 25

Spoofing the camera in Google Meet Zoom Teams and more

#showdev #webdev #javascript

I saw a Telegram post about Snap, a Google Meet extension by @deniza-png. Before the call you photograph the empty room; mid-call you snap your fingers and disappear.

I wanted that, plus everything else I was missing on calls: backgrounds, masks, stickers, translated captions, a teleprompter. So I built Telepult.Web, a browser extension that sits on top of the web call and edits the camera before anyone else sees it.

It started on Meet, then grew to Zoom Web, Microsoft Teams, Discord, Slack huddles and about thirty other sites. Same idea everywhere: the call page asks for a camera, the extension hands it a painted stream.

Credit where it is due: I could not find Snap’s author on every network, so I am tagging the GitHub nick. If that is you under another name, say so and I will fix the link.

Intercepting the camera

Meet, Zoom Web and Teams all take the camera through plain getUserMedia and feed the MediaStream into WebRTC. They do not care where the video track came from. So I patch getUserMedia before the page calls it: open the real camera myself, return a track I paint on a canvas.

const origGUM = MediaDevices.prototype.getUserMedia;

MediaDevices.prototype.getUserMedia = async function (constraints) {
  const real = await origGUM.apply(this, arguments);
  try {
    if (constraints?.video) return await wrapCameraStream(real);
    if (constraints?.audio) return await wrapAudioStream(real);
  } catch (e) {
    console.warn("wrap failed", e);
  }
  return real; // if wrapping fails, the call matters more than effects
};

Inside wrapCameraStream the camera track goes into a hidden video, frames are drawn on a canvas, and the outside gets canvas.captureStream. Call sites peek at getSettings and label, so I mirror those from the real camera.

async function wrapCameraStream(real) {
  const camTrack = real.getVideoTracks()[0];
  const video = document.createElement("video");
  video.muted = true;
  video.srcObject = new MediaStream([camTrack]);
  await video.play();

  const pipe = new Pipeline(video);
  pipe.start();

  const outTrack = pipe.canvas.captureStream(30).getVideoTracks()[0];
  outTrack.getSettings = () => ({
    ...camTrack.getSettings(),
    width: pipe.W,
    height: pipe.H,
  });
  Object.defineProperty(outTrack, "label", { value: camTrack.label });
  return new MediaStream([outTrack, ...real.getAudioTracks()]);
}

That is the property of every feature below: whatever I draw on that canvas leaves as normal video. It lands in the recording and on everyone else’s preview. They do not install anything.

Render runs on requestAnimationFrame, capped around 30 fps. If the previous frame is still painting, the tick is skipped; otherwise a weak laptop queues frames and video drifts from audio. The mic is wrapped separately because Meet and Zoom often request it in a second getUserMedia call without video.

MediaPipe

Face, hands and segmentation use Google MediaPipe Tasks Vision. Models run in the browser on WebAssembly, with GPU when it works:

Face Landmarker: 478 face points; masks and face blur sit on them
Hand Landmarker: 21 points per hand; gestures, finger snap, drawing
Image Segmenter with the selfie model: person mask for backgrounds and blur

Models ship inside the extension package (face ~3.6 MB, hands ~7.5 MB, segmenter 244 KB; the whole pack is around 20 MB). Install is a bit slower, but you do not download tens of megabytes from a CDN the moment someone turns on a background on a bad connection. There is also a store rule: Chrome Web Store rejects extensions that load executable code from third-party URLs, and MediaPipe brings WASM with it.

const vision = await FilesetResolver.forVisionTasks(
  chrome.runtime.getURL("vendor/tasks-vision/wasm")
);

const face = await FaceLandmarker.createFromOptions(vision, {
  baseOptions: {
    modelAssetPath: chrome.runtime.getURL("vendor/models/face_landmarker.task"),
    delegate: "GPU", // fall back to "CPU" if GPU init fails
  },
  runningMode: "VIDEO",
  numFaces: 1,
});

Everything loads lazily: a model stays cold until a feature that needs it turns on. Inference lives on a hidden extension page; the call page sends a downscaled ImageBitmap and gets landmarks and a mask back. Width before send is 256–480 px depending on the model set: enough for a face, much cheaper.

The first version sent every frame. When the model fell behind, messages queued, and after a minute the mask lagged the face by seconds. Now a frame that arrives while the model is busy is dropped. Each model combo has its own minimum gap: hands alone can run almost every frame; segmentation with hands is capped around once per 100 ms.

async process(video, needSeg, needFace, needHands) {
  if (this.busy) return; // drop the frame, do not queue
  if (performance.now() - this.lastSend < this.minGap(needSeg, needFace, needHands)) return;
  this.busy = true;
  try {
    const bitmap = await createImageBitmap(this.downscale(video));
    const res = await this.sendToModels({ bitmap, needSeg, needFace, needHands });
    this.apply(res);
  } finally {
    this.busy = false;
  }
}

Face points jitter frame to frame, so new coordinates are blended about half-and-half with the previous ones. Lag is barely visible; the mask stops shaking.

Vanishing

Same scenario as Snap: turn the camera on, leave the frame, press “Capture room”, come back. Then “Vanish” (or a finger snap) fades you out and the empty room stays. “Return” plays the animation backwards.

“Capture room” does not take one frame. It averages 14 frames 40 ms apart. Webcams are noisy in low light; a single frame leaves false spots in the mask. The plate is JPEG-compressed into chrome.storage.local, so the next call does not need a new capture if the camera has not moved.

No segmenter here: with an empty-room plate, the person is whatever differs from it. Live frame and plate shrink to 480 px wide; each pixel gets the sum of absolute RGB diffs. Above a threshold it counts as person.

const DIFF_T = 44;

for (let i = 0; i < n; i++) {
  const j = i * 4;
  const diff =
    Math.abs(live[j]     - bg[j]     - o0) +
    Math.abs(live[j + 1] - bg[j + 1] - o1) +
    Math.abs(live[j + 2] - bg[j + 2] - o2);
  bin[i] = diff > DIFF_T ? 255 : 0;
}

o0, o1, o2 correct for auto-exposure and white balance. Sit down in a dark shirt and the camera lifts the whole picture; without the offset the entire room starts looking different from the plate. The shift is estimated in two passes: mean diff over the frame, then again only on pixels already close to the plate (the background). That is what gets subtracted.

The raw mask is ragged, so it is cleaned:

Blur twice and threshold again: noise goes, edges settle.
Keep only the largest blob; the rest is junk.
Flood-fill background from the frame edges. Wherever fill did not reach is the person. That closes holes in a shirt the same color as the wall.

The Thanos effect assigns each pixel a threshold from its x coordinate plus a bit of randomness. Animation progress goes 0→1; a pixel dies when progress crosses its threshold, so the person crumbles left to right with a ragged edge. Some of the dead pixels become ash particles of the same color; under the person the room plate is drawn.

for (let y = 0; y < HH; y++)
  for (let x = 0; x < WW; x++)
    thr[y * WW + x] = 0.55 * (x / WW) + 0.45 * Math.random();

// each frame
alpha[i] = mask[i] * smooth((thr[i] - p) / BAND + 0.5);

The other transitions (dissolve, glitch, pixels, funnel, smoke, shards, CRT off, portal, tiles, burn, melt, explode) reuse the same frozen mask and only change which pixel dies when.

Finger snap is detected from video; the mic is not involved (unlike Snap). I keep 300 ms of distance between the thumb tip and the nearest of index/middle tips. A snap is fingers together, then quickly apart: look for a minimum followed by distance rising fast enough. 1.2 s cooldown so one snap does not vanish and immediately return.

function detectSnap(history, now) {
  const s = history.filter((p) => now - p.t <= 300);
  const min = minBy(s, "d");
  const max = maxBy(s, "d");
  if (!(min.d < 0.085 && max.d > 0.095 && max.t > min.t + 20)) return false;
  const after = s.filter((p) => p.t >= min.t);
  const first = after[0], last = after[after.length - 1];
  const speed = (last.d - first.d) / Math.max(16, last.t - first.t);
  return speed > 0.00028 && last.d - min.d > 0.04;
}

There is also “Freeze frame”: the call gets the last painted frame until you press again. Later that grew into Live Photo, a short looping burst instead of a dead still, so on long meetings nobody wonders why you stopped blinking.

Background and blur

No room plate here, so the segmenter is required. It returns a confidence mask; I turn that into alpha with a soft ramp: below 0.35 background, above 0.7 person, linear edge in between so hair is not cut out with scissors.

for (let i = 0; i < out.length; i++) {
  let a = person[i];
  if (inverted) a = 1 - a;
  out[i] = a < 0.35 ? 0 : a > 0.7 ? 255 : (((a - 0.35) / 0.35) * 255) | 0;
}

The segmenter mask is tiny. Upscaled to 720p it left a light halo around the head. Before compositing I stretch it with a 1 px blur and threshold again. Then normal compositing: background, camera frame on top via destination-in with the mask.

About twenty built-in backgrounds, plus your own file. Animated ones (matrix, snow, rain, space, fireflies and the rest) are drawn procedurally on the same canvas, so they barely weigh anything in the package.

Masks and face blur

A mask is a few PNG layers (ears, nose, tongue). Each layer stores two source points on the image and two face-landmarker points. From the two pairs I compute a similarity transform and draw so the points match. Ears tilt with the head and shrink when you lean back.

function drawSimilarity(ctx, img, srcA, srcB, dstA, dstB) {
  const scale = dist(dstA, dstB) / dist(srcA, srcB);
  const angle =
    Math.atan2(dstB.y - dstA.y, dstB.x - dstA.x) -
    Math.atan2(srcB.y - srcA.y, srcB.x - srcA.x);
  ctx.save();
  ctx.translate(dstA.x, dstA.y);
  ctx.rotate(angle);
  ctx.scale(scale, scale);
  ctx.drawImage(img, -srcA.x, -srcA.y);
  ctx.restore();
}

Ears bob a little. The dog tongue only shows when the mouth is open: distance between upper and lower lip points.

Anonymity covers only the face, not the whole frame. Face Landmarker gives a 36-point face oval; I build a path, clip, and inside draw either a blurred frame or pixels. Pixels are the classic shrink-by-16 then stretch with imageSmoothingEnabled = false.

ctx.save();
ctx.beginPath();
ctx.moveTo(oval[0].x, oval[0].y);
for (const p of oval) ctx.lineTo(p.x, p.y);
ctx.closePath();
ctx.clip();

if (mode === "blur") {
  ctx.filter = "blur(16px)";
  ctx.drawImage(video, 0, 0, W, H);
} else {
  small.drawImage(video, x, y, w, h, 0, 0, w / 16, h / 16);
  ctx.imageSmoothingEnabled = false;
  ctx.drawImage(small.canvas, 0, 0, w / 16, h / 16, x, y, w, h);
}
ctx.restore();

The oval follows the face, so you can walk around the room while hands and background stay normal.

Gestures and finger drawing

A finger is “out” if its tip is farther from the wrist than its middle joint by about 12%. The set of extended fingers classifies the gesture.

const fingerOut = (lm, tip, pip) =>
  dist(lm[tip], lm[WRIST]) > dist(lm[pip], lm[WRIST]) * 1.12;

function classifyHand(lm) {
  const ix = fingerOut(lm, 8, 6), md = fingerOut(lm, 12, 10);
  const rg = fingerOut(lm, 16, 14), pk = fingerOut(lm, 20, 18);
  const thumb = dist(lm[4], lm[WRIST]) > dist(lm[3], lm[WRIST]) * 1.15;
  const count = [ix, md, rg, pk].filter(Boolean).length;

  if (count >= 4) return "palm";
  if (thumb && count <= 1) return "thumbsup";
  if (ix && md && !rg && !pk) return "victory";
  if (ix && !md && !rg && !pk) return "point";
  return null;
}

A gesture only fires after two consecutive recognitions, with 700 ms between triggers. Otherwise any hand near the face toggles something. Palm and thumbs-up drop ✋ / 👍 into the frame; point toggles the mic; victory shows and hides a corner picture.

Drawing takes index tip (landmark 8) and writes a trail. Coordinates are exponentially smoothed so the line is clean without lagging the finger. Each segment fades in 2.8 s with squared alpha: bright, then gone. No eraser: circle something on the board behind you, three seconds later the line is gone.

Captions with translation

You speak one language; captions in the frame show another. Two stages: speech recognition, then translation. The finished line is painted at the bottom of the same canvas.

Recognition

Transcription is Web Speech API (webkitSpeechRecognition in Chromium). There is no speech model in the package: Chrome listens to the mic, sends audio to its recognition service, returns text. You need a network, and quality depends on language and connection. You also avoid another heavy model next to MediaPipe. In practice this is Chrome, Edge, Brave and Opera. Not Firefox.

const SR = window.SpeechRecognition || window.webkitSpeechRecognition;
const rec = new SR();
rec.lang = "en-US";
rec.continuous = true;     // do not stop after the first phrase
rec.interimResults = true; // drafts while speaking

rec.onresult = (ev) => {
  for (let i = ev.resultIndex; i < ev.results.length; i++) {
    const text = ev.results[i][0].transcript.trim();
    pushCaption(text, ev.results[i].isFinal);
  }
};

Without continuous, recognition dies on the first pause. interimResults gives drafts: while isFinal is false Chrome may still revise the text, but you can show it immediately. The loop starts at ev.resultIndex because results holds the whole session. Restarting from zero on a long call would re-walk half an hour of phrases every time.

Even with continuous: true, Chrome periodically ends the session and fires onend after a pause or a network blip. Ignore that and captions silently die after a few minutes. So onend restarts recognition after a short delay. lang on a running instance does not change, so a language switch recreates the recognizer.

rec.onend = () => {
  if (!needRecognition() || restarting) return;
  restarting = true;
  setTimeout(() => {
    restarting = false;
    try { rec.start(); } catch (_) {}
  }, 280);
};

Translation

Translation uses Chrome’s built-in Translator API. The model for a language pair downloads on first use, then translation is local. No third-party keys. If the API is missing, the frame keeps the source text.

async function translateText(text, source, target) {
  if (!text || source === target) return text;
  const key = `${source}>${target}:${text}`;
  if (cache.has(key)) return cache.get(key);
  try {
    const translator = await Translator.create({
      sourceLanguage: source,
      targetLanguage: target,
    });
    const out = await translator.translate(text);
    if (out) {
      cache.set(key, out); // keep the last ~120 phrases
      return out;
    }
  } catch (_) {}
  return text;
}

Drafts have two problems. There are too many of them, so translation is rate-limited (~450 ms); while it runs, the source shows with an ellipsis. Translation is async, so an old draft can finish after a newer one and rewind the line. A generation counter drops stale results.

async function pushCaption(raw, isFinal) {
  if (!isFinal) {
    if (performance.now() - lastInterimAt < 450) return;
    lastInterimAt = performance.now();
    captions.interim = "… " + raw.slice(-160);
    redraw();
  }

  const gen = ++translateGen;
  const shown = await translateText(raw, captions.sourceLang, captions.targetLang);
  if (gen !== translateGen) return;

  if (isFinal) {
    captions.text = shown.slice(-180);
    captions.interim = "";
  } else {
    captions.interim = shown.slice(-180);
  }
  redraw();
}

Seven languages today. Same “I speak” and “captions in” languages means plain captions with no translation. Downside: others cannot turn captions off on their side. They are part of the video. Sometimes that is the point.

I thought about storing the whole call with timings and sending it to a remote LLM for minutes. Meet and Zoom already sell that. So I did not.

Voice

The mic goes through a Web Audio graph: clone the audio track, MediaStreamSource, effect chain, MediaStreamDestination, and the call gets the destination stream. The track is cloned because Meet/Zoom flip enabled on their own track and without a clone the graph sometimes goes silent.

Effects from stock nodes:

robot: narrow bandpass ~1100 Hz, WaveShaper, lift ~1800 Hz
phone: 300–3400 Hz band like an old line, light crunch
echo: DelayNode ~200 ms with feedback
helium / deep: pitch shift

// phone
const hp = ac.createBiquadFilter();
hp.type = "highpass";
hp.frequency.value = 300;
const lp = ac.createBiquadFilter();
lp.type = "lowpass";
lp.frequency.value = 3400;
const crunch = ac.createWaveShaper();
crunch.curve = makeDistortionCurve(28);
source.connect(hp).connect(lp).connect(crunch).connect(master);

Web Audio has no ready-made pitch shift; an EQ alone just makes a bad mic. I used Chris Wilson’s Audio Input Effects trick: two delay lines whose delay times sweep like a saw. Shrinking delay plays slightly faster (pitch up); growing delay plays slower (pitch down). Good enough for a joke on a standup.

Filler words and swearing counter

Same recognition path as captions. The phrase is lowercased, punctuation stripped, then matched against two lists: swears and fillers (“like”, “basically”, “you know”). Longer phrases first so short ones are not double-counted; words of up to four letters only match whole tokens.

Without headphones the mic hears other people from the speakers and the counter counted their swears. A word only counts when the mic level shows that I am the one talking. The tally can stay private or paint into the frame for everyone.

Stickers and reactions

Stickers are short WebM files with alpha, plus your own upload up to 2.5 MB. A sticker rises for about three seconds and fades; at most eight stickers/reactions on screen at once.

Each sticker is a hidden looping video; its current frame is drawn onto the output canvas. About 15 times a second I also snapshot an ImageBitmap as a fallback: while the video is still loading or the browser stalls it, the last bitmap draws so the sticker does not flash an empty square.

reactions.push({
  kind: "sticker",
  video,   // webm with alpha
  bitmap,  // fallback frame
  life: 3200,
  x0: 0.28 + Math.random() * 0.44,
  y0: 0.55 + Math.random() * 0.22,
});

Custom stickers live in IndexedDB and survive a page reload.

Same canvas, more overlays

Status: a short badge under the chin, pinned to a Face Landmarker point, moves with the face.

AFK for 1 / 3 / 5 minutes: an “away” badge with a countdown.

Ticker at the bottom of the frame. Text is drawn several times with a shift of exactly one copy width so the loop does not hitch.

Corner picture: a diagram or QR in frame without screen share; victory gesture toggles it.

Retro: colored cards on your own preview. Font size on a card is binary-searched so any length fits. For a three-person retro it is faster than dragging everyone onto a separate board.

Where it runs

One extension, many hosts: Google Meet, Zoom Web Client, Microsoft Teams (browser), Discord, Slack, Webex, Whereby and the rest of the catalog on the site. Chromium browsers install from the Chrome Web Store; Firefox has its own build (everything except captions: no Web Speech recognition there).

Try it without installing: senior-frontend-developer.com opens the same remote on your camera. The picture never leaves the machine.

DEV Community