Stopping the camera from flip-flopping: confidence scoring for AI active-speaker switching

Suman Kumar Khan — Tue, 10 Nov 2026 15:06:07 +0000

The first version of any automatic camera switcher looks great in a demo and terrible in a real conversation. Two people talk, one laughs, someone says "yeah" over the other, and the program output cuts back and forth like a nervous editor. On Prisium, a remote multi-camera live production platform I worked on at Digital Avenues, I built AI camera switching on two platforms. This post is about the one idea that made it watchable: stop asking "who is speaking right now?" and start asking "how sure am I, and for how long?"

The naive version: one threshold, instant cut

The obvious design is a loop. Every frame or audio window, compute a score per camera — "is this person talking?" — and cut to the highest score above a threshold.

// naive-switcher.ts
// The version that flip-flops. Values are illustrative.
function pickCamera(scores: Record<string, number>, threshold = 0.6) {
  const [best, score] = Object.entries(scores).sort((a, b) => b[1] - a[1])[0];
  return score >= threshold ? best : null;
}

Speech is not a clean signal. People pause mid-sentence, breathe, nod and make short backchannel noises. Any of those pushes a score across the threshold for a few frames, and an instant cut turns that noise straight into visible jitter. Lowering the threshold makes it twitchier; raising it makes the switcher miss real turns.

Two signals are better than one

On Prisium the switching decision combined two inputs:

Visual: MediaPipe face landmarks, used to detect lip movement per camera.
Audio: an active-speaker signal from each participant's audio.

Each one fails differently. Audio alone gets fooled by a noisy room or by one participant's microphone picking up another. Lip movement alone gets fooled by chewing, smiling or someone mouthing along. When both agree, you can be much more confident than either alone. Treat them as evidence, not as a verdict.

Note: MediaPipe runs on-device, which matters for live video: you get per-frame landmarks without shipping every participant's video to a server just to decide who is talking.

A rising confidence score instead of an instant threshold

The fix that stopped the flip-flopping was to make confidence something a camera has to earn over time. Each camera keeps a running confidence. When its evidence says "speaking", confidence rises gradually; when it doesn't, confidence decays. The switcher only cuts when a challenger's confidence has climbed past a bar — not the instant one noisy frame looks good.

// confidence.ts
// Illustrative constants — tune against your own footage.
const RISE = 0.08;   // how fast evidence builds confidence per tick
const DECAY = 0.04;  // how fast confidence fades without evidence
const CUT_AT = 0.85; // a challenger must reach this before we cut

type Cam = { id: string; confidence: number };

export function tick(cams: Cam[], evidence: Record<string, number>) {
  for (const cam of cams) {
    const e = evidence[cam.id] ?? 0; // 0..1, combined lip + audio evidence
    cam.confidence = e > 0.5
      ? Math.min(1, cam.confidence + RISE * e)
      : Math.max(0, cam.confidence - DECAY);
  }
}

This is a small change with a big effect. A short "mm-hm" from a listener never builds enough confidence to cause a cut. A real turn — someone who keeps talking — climbs steadily and wins. The switcher stops reacting to frames and starts reacting to turns.

Tip: Rise and decay don't have to be symmetric. Letting confidence fade a little slower than it builds keeps the current speaker on screen through natural pauses.

Hold the last speaker when the signal is unclear

The second rule: when nobody clearly wins, don't move. If no challenger has crossed the bar, the program stays on the last confident speaker. Silence, crosstalk that cancels out, or a dropped frame from a weak connection should never produce a cut on its own.

// decide.ts
export function decide(cams: Cam[], current: string) {
  const challenger = cams
    .filter(c => c.id !== current)
    .sort((a, b) => b.confidence - a.confidence)[0];
  if (challenger && challenger.confidence >= CUT_AT) return challenger.id;
  return current; // unclear → hold
}

This mirrors what a human operator does. When a conversation gets messy, a good director doesn't chase every voice; they wait for the picture to make sense.

When several people speak, go wide

Holding is right for ambiguity, but not for a genuine overlap — two or three people laughing or debating at once. Cutting to any one of them misrepresents the moment. On Prisium the rule was simple: if several participants have strong confidence at the same time, cut to the wide shot.

// decide.ts
const OVERLAP_AT = 0.7;

export function decideWithWide(cams: Cam[], current: string) {
  const strong = cams.filter(c => c.confidence >= OVERLAP_AT);
  if (strong.length > 1) return 'wide';
  return decide(cams, current);
}

The wide shot is a safe, honest frame: everyone is visible, and nothing jumps.

What I'd tell someone building this

Optimise for viewers, not for accuracy metrics. A switcher that is "right" on every frame but cuts constantly is worse than one that is slightly late and calm.
Tune with real recordings. Constants like the rise rate or cut threshold depend on your capture, your audio pipeline and how your users talk. Replay real sessions and watch the output.
Keep the producer in control. Automatic switching should be a mode, not a takeover. The producer studio I worked on kept manual feed switching alongside it.

Warning: Don't add a hard "minimum shot length" timer as your only fix. It hides jitter but also delays real turns. A confidence score solves the cause; a timer just masks the symptom.

The lesson generalises beyond video: whenever a noisy signal drives a visible decision, give the decision memory. Confidence that rises, holds and falls back gracefully turns flicker into judgement.

Originally published at sumankumarkhan.com.