Keeping headlines off people's faces: in-browser face detection with MediaPipe

Our flyer maker has a mode where you skip the AI entirely: you upload your own photo, type a headline, a date and a contact line, and the browser lays it out as a printable flyer. No generation, no credits, and the photo never leaves your device.

The first version placed the headline at the top for most flyer types, because that's where event posters usually put it. Then we tried it with a portrait, which is exactly what a photographer advertising mini-sessions would upload. The headline sat across the subject's forehead and eyes, and the dark gradient we use to keep white text readable turned her face into a shadow.

The fix sounds obvious: find the faces, put the text somewhere else. Doing it in the browser, cheaply and without uploading the photo, took a few decisions worth sharing.

The detector: BlazeFace through MediaPipe Tasks

We use the FaceDetector from @mediapipe/tasks-vision with Google's short-range BlazeFace model. The float16 version is about 230 KB, which is small enough to fetch on demand, and on the CPU it handles a single photo fast enough that nobody notices.

let detector: Promise<FaceDetector> | null = null

function loadDetector(): Promise<FaceDetector> {
  if (!detector) {
    detector = (async () => {
      const mp = await import('@mediapipe/tasks-vision')
      const vision = await mp.FilesetResolver.forVisionTasks('/mediapipe/1.1.0')
      return mp.FaceDetector.createFromOptions(vision, {
        baseOptions: {modelAssetPath: '/models/blaze_face_short_range.tflite', delegate: 'CPU'},
        runningMode: 'IMAGE',
        minDetectionConfidence: 0.5,
      })
    })()
    // If it fails (offline, old browser), forget it so the next try starts fresh.
    detector.catch(() => { detector = null })
  }
  return detector
}

Three details here saved us trouble:

• Dynamic import. The MediaPipe package is only loaded when someone switches to photo mode. Everyone else never downloads it.
• Cache the promise, not the result. Two components asking at the same time share one load. And if the load fails, we drop the cached promise so a later attempt can succeed once the network is back.
• Results cached per image URL. Changing the headline re-runs the layout dozens of times; detection runs once per photo.

Hosting the WebAssembly yourself

FilesetResolver needs the MediaPipe WebAssembly runtime. The examples point it at a CDN, which works until it doesn't, and it adds a third-party request to a page that otherwise only talks to us. The runtime is around 26 MB if you keep both the SIMD and non-SIMD builds, so it doesn't belong in git either.

What we do instead: at build time, a small script copies the four runtime files from node_modules/@mediapipe/tasks-vision into public/mediapipe/<version>/, a folder that is ignored by git. The version is part of the URL, so upgrading the package means a new path and no stale cache. The detector model and the selfie-segmentation model we use elsewhere are served from /models the same way.

Turning boxes into a decision

MediaPipe returns bounding boxes in the photo's pixel space. The flyer draws the photo full-bleed, scaled to cover a 2550 × 3300 canvas (US Letter at 300 DPI) and centered, so part of the photo is cropped. The face position has to go through exactly the same transform, or the decision is made on the wrong part of the picture:

// face: fractions of the photo (x, y, w, h). Returns its vertical span in flyer pixels.
function faceSpanInFlyer(face, image, flyerW = 2550, flyerH = 3300): [number, number] {
  const scale = Math.max(flyerW / image.width, flyerH / image.height)
  const top = (flyerH - image.height * scale) / 2
  return [top + face.y * image.height * scale, top + (face.y + face.h) * image.height * scale]
}

Then, instead of trying to compute the "best" layout directly, we lay out the flyer both ways, text on top and text at the bottom, and score each one by how much face it covers:

function coverScore(position) {
  const {lines, bands} = layoutFlyer({...text, position})
  let total = 0
  for (const [a, b] of bands) {
    const isHeadline = lines.some((l) => l.role === 'headline' && l.y >= a && l.y <= b)
    for (const face of faces) {
      const [y0, y1] = faceSpanInFlyer(face, image)
      const covered = Math.max(0, Math.min(b, y1) - Math.max(a, y0)) / Math.max(1, y1 - y0)
      total += covered * (isHeadline ? 1 : 0.5) // small print hurts less than a big headline
    }
  }
  return total
}

const position = faces.length === 0 ? 'bottom' : coverScore('top') < coverScore('bottom') ? 'top' : 'bottom'

Scoring real layouts, rather than reasoning about the photo alone, means the rule keeps working when the text changes. A one-word headline at the top might clear a face that a three-line one would cover, and the score sees that.

When there are no faces, or the detector can't see them

The short-range model is tuned for faces reasonably close to the camera. In a wide shot of a band on stage, it often finds nothing. That's fine: with no faces we default to the bottom, which suits most photos because the subject tends to fill the center and the top of the frame. The same default applies if anything fails, from a blocked download to an old browser. The flyer still renders; it just doesn't get the smart placement.

The user can always override the choice. "Auto" is only the starting point, with Top and Bottom one tap away.

What it cost

• About 230 KB of model, loaded only in photo mode.
• One detection per photo, on the CPU, with no visible wait.
• No server round trip, and the photo is never uploaded.

If you want to try it with your own photo, it's the "Use your photo" mode of the flyer maker on CreArt. Detection, layout and the final 300 DPI export all happen in the browser.

How do you handle text placement over user photos? I'm curious whether anyone scores layouts like this or goes straight to saliency maps.