Engineering a browser PNGTuber: canvas alpha, volume frames, local mic privacy

PNGTubers are a product wrapper around three engineering problems: export real alpha, switch sprites from loudness (not ASR), and keep microphone samples off the network. This note walks through the patterns we use on PNGTuber Maker without turning it into a framework tour.

1. Transparent export that survives OBS

Browsers composite to a <canvas>. A common failure mode is toDataURL('image/jpeg') or drawing onto an opaque canvas — both destroy alpha.

Practical rules:

• Create the canvas with an alpha-capable context (getContext('2d', { alpha: true }) — default is already alpha, but do not clear with a solid fillRect of white before export).
• Prefer canvas.toBlob('image/png') over JPEG.
• If you redraw AI output that arrived as RGB without alpha, you must either receive RGBA from the generator or run a background-removal step that writes into the alpha channel — color-keying in OBS is a user-facing apology, not a pipeline design.
• Square output (e.g. 1024²) keeps Browser Source sizing boring. Non-square works; streamers will stretch it wrong.

Edge case: premultiplied alpha vs straight alpha. Soft fur fringes can darken or halo depending on how the encoder and OBS Chromium disagree. If fringes look muddy, inspect whether you are double-multiplying when compositing layers onto the export canvas.

2. Volume-driven frame switching (not lip-sync)

We do not run phoneme detection for the default player. The state machine is intentionally dumb:

states: idle | talking | blink | blinkTalk
input: RMS or time-domain peak from AnalyserNode
params: threshold, attack/release (hysteresis), blink interval

Sketch:

const ctx = new AudioContext();
const src = ctx.createMediaStreamSource(stream);
const analyser = ctx.createAnalyser();
analyser.fftSize = 2048;
src.connect(analyser);

const data = new Uint8Array(analyser.fftSize);
function level() {
  analyser.getByteTimeDomainData(data);
  let sum = 0;
  for (let i = 0; i < data.length; i++) {
    const v = (data[i] - 128) / 128;
    sum += v * v;
  }
  return Math.sqrt(sum / data.length);
}

Hysteresis matters. A single threshold produces chatter at the boundary. Use openThreshold and closeThreshold, or require N consecutive frames above/below before transitioning.

Blink is a timed interrupt: briefly force blink or blinkTalk based on whether you were in talking, then restore. If you only have a closed-eye idle and no blinkTalk art, blinking mid-sentence visually closes the mouth — artists notice, viewers feel “something stuttered.”

3. Wiring into OBS

Two deployment shapes:

1. Hosted player URL → OBS Browser Source. Fast iteration; needs mic permission inside the embedded Chromium; cache-bust when you ship player fixes.
2. Static HTML + local PNGs → file:// Browser Source. Offline-friendly; updates are manual copies.

OBS Mixer mute does not automatically gate getUserMedia inside the Browser Source. Document that in UI empty states. Dual-PC setups fail when the streaming PC cannot reach the hosted player origin — ship a downloadable frame pack for that case.

4. Local privacy for the mic

Requirements we treat as non-negotiable for the analyser path:

• Mic bytes stay in the page. No uploading of PCM/Opus for “quality improvement” without an explicit, separate feature.
• Tear down tracks on page hide (stream.getTracks().forEach(t => t.stop())) so the OS indicator does not linger.
• Autoplay / AudioContext resume: browsers suspend audio contexts until a user gesture — the first “allow mic + click preview” must await ctx.resume().

Generation uploads (reference images, prompts) are a different pipeline from mic volume. Keep those endpoints and retention policies separate in code and in copy so a privacy question has one answer.

5. Performance budget

Browser Source is a real Chromium. Cap:

• No full-page WebGL particle systems behind the avatar for the OBS URL.
• Prefer CSS/drawImage sprite swaps over re-decoding PNGs every frame — decode once into ImageBitmaps.
• rAF loop should read analyser + swap visibility/opacity; do not re-layout the entire DOM.

On mid-tier CPUs, a quiet player matters more than a prettier landing page.

6. Product-facing failure modes (map them to engineering)

User report	Likely cause
White box	JPEG or opaque clear
Mouth dead	permission / AudioContext suspended / threshold
Mouth chatter	no hysteresis / no noise gate guidance
Jitter	frame set misaligned; not a timing bug
Stale bug after deploy	OBS Browser cache

Fix the empty state before adding another ML stage.

Closing

A browser PNGTuber is mostly careful I/O: PNG alpha, AnalyserNode hysteresis, and honest mic scope. The generative front-end is optional; the export and player contracts are not. If you ship those three cleanly, OBS users can ignore the rest of your stack — which is the point.