DEV Community: Jon Ayo

Keeping headlines off people's faces: in-browser face detection with MediaPipe

Jon Ayo — Fri, 30 Oct 2026 15:00:00 +0000

Our flyer maker has a mode where you skip the AI entirely: you upload your own photo, type a headline, a date and a contact line, and the browser lays it out as a printable flyer. No generation, no credits, and the photo never leaves your device.

The first version placed the headline at the top for most flyer types, because that's where event posters usually put it. Then we tried it with a portrait, which is exactly what a photographer advertising mini-sessions would upload. The headline sat across the subject's forehead and eyes, and the dark gradient we use to keep white text readable turned her face into a shadow.

The fix sounds obvious: find the faces, put the text somewhere else. Doing it in the browser, cheaply and without uploading the photo, took a few decisions worth sharing.

The detector: BlazeFace through MediaPipe Tasks

We use the FaceDetector from @mediapipe/tasks-vision with Google's short-range BlazeFace model. The float16 version is about 230 KB, which is small enough to fetch on demand, and on the CPU it handles a single photo fast enough that nobody notices.

let detector: Promise<FaceDetector> | null = null

function loadDetector(): Promise<FaceDetector> {
  if (!detector) {
    detector = (async () => {
      const mp = await import('@mediapipe/tasks-vision')
      const vision = await mp.FilesetResolver.forVisionTasks('/mediapipe/1.1.0')
      return mp.FaceDetector.createFromOptions(vision, {
        baseOptions: {modelAssetPath: '/models/blaze_face_short_range.tflite', delegate: 'CPU'},
        runningMode: 'IMAGE',
        minDetectionConfidence: 0.5,
      })
    })()
    // If it fails (offline, old browser), forget it so the next try starts fresh.
    detector.catch(() => { detector = null })
  }
  return detector
}

Three details here saved us trouble:

Dynamic import. The MediaPipe package is only loaded when someone switches to photo mode. Everyone else never downloads it.
Cache the promise, not the result. Two components asking at the same time share one load. And if the load fails, we drop the cached promise so a later attempt can succeed once the network is back.
Results cached per image URL. Changing the headline re-runs the layout dozens of times; detection runs once per photo.

Hosting the WebAssembly yourself

FilesetResolver needs the MediaPipe WebAssembly runtime. The examples point it at a CDN, which works until it doesn't, and it adds a third-party request to a page that otherwise only talks to us. The runtime is around 26 MB if you keep both the SIMD and non-SIMD builds, so it doesn't belong in git either.

What we do instead: at build time, a small script copies the four runtime files from node_modules/@mediapipe/tasks-vision into public/mediapipe/<version>/, a folder that is ignored by git. The version is part of the URL, so upgrading the package means a new path and no stale cache. The detector model and the selfie-segmentation model we use elsewhere are served from /models the same way.

Turning boxes into a decision

MediaPipe returns bounding boxes in the photo's pixel space. The flyer draws the photo full-bleed, scaled to cover a 2550 × 3300 canvas (US Letter at 300 DPI) and centered, so part of the photo is cropped. The face position has to go through exactly the same transform, or the decision is made on the wrong part of the picture:

// face: fractions of the photo (x, y, w, h). Returns its vertical span in flyer pixels.
function faceSpanInFlyer(face, image, flyerW = 2550, flyerH = 3300): [number, number] {
  const scale = Math.max(flyerW / image.width, flyerH / image.height)
  const top = (flyerH - image.height * scale) / 2
  return [top + face.y * image.height * scale, top + (face.y + face.h) * image.height * scale]
}

Then, instead of trying to compute the "best" layout directly, we lay out the flyer both ways, text on top and text at the bottom, and score each one by how much face it covers:

function coverScore(position) {
  const {lines, bands} = layoutFlyer({...text, position})
  let total = 0
  for (const [a, b] of bands) {
    const isHeadline = lines.some((l) => l.role === 'headline' && l.y >= a && l.y <= b)
    for (const face of faces) {
      const [y0, y1] = faceSpanInFlyer(face, image)
      const covered = Math.max(0, Math.min(b, y1) - Math.max(a, y0)) / Math.max(1, y1 - y0)
      total += covered * (isHeadline ? 1 : 0.5) // small print hurts less than a big headline
    }
  }
  return total
}

const position = faces.length === 0 ? 'bottom' : coverScore('top') < coverScore('bottom') ? 'top' : 'bottom'

Scoring real layouts, rather than reasoning about the photo alone, means the rule keeps working when the text changes. A one-word headline at the top might clear a face that a three-line one would cover, and the score sees that.

When there are no faces, or the detector can't see them

The short-range model is tuned for faces reasonably close to the camera. In a wide shot of a band on stage, it often finds nothing. That's fine: with no faces we default to the bottom, which suits most photos because the subject tends to fill the center and the top of the frame. The same default applies if anything fails, from a blocked download to an old browser. The flyer still renders; it just doesn't get the smart placement.

The user can always override the choice. "Auto" is only the starting point, with Top and Bottom one tap away.

What it cost

About 230 KB of model, loaded only in photo mode.
One detection per photo, on the CPU, with no visible wait.
No server round trip, and the photo is never uploaded.

If you want to try it with your own photo, it's the "Use your photo" mode of the flyer maker on CreArt. Detection, layout and the final 300 DPI export all happen in the browser.

How do you handle text placement over user photos? I'm curious whether anyone scores layouts like this or goes straight to saliency maps.

Stop asking image models to spell: setting text over AI art with Canvas

Jon Ayo — Fri, 16 Oct 2026 18:50:28 +0000

Image models have gotten very good at pictures and are still unreliable at words. Ask one for an album cover that says "Concrete Halo" and you will get "Concrete Halo" most of the time, "Concrete Hallo" some of the time, and occasionally a beautiful blackletter word that doesn't exist. For a cover, a poster or a book jacket, one wrong letter means the whole image is unusable.

We build design tools on top of image models, and for a while our album cover tool used a premium model specifically because it was better at lettering. It was also several times more expensive per image, and when it did misspell a word, the user paid again to regenerate everything just to fix a typo.

So we split the job. The model paints the artwork only (we tell it explicitly: no text, no letters), and the browser draws the artist name and title on top with the Canvas 2D API. The text is always spelled right, it can be edited without generating again, and the image itself can come from a much cheaper model.

Here is what that took in practice.

1. Load the fonts before you draw

Canvas draws with whatever font is available at the moment you call fillText. If the web font hasn't loaded yet, you silently get the fallback and a cover set in Times New Roman. We load each font explicitly with the FontFace API and cache the promise:

const loaded = new Map<string, Promise<void>>()

function loadFont(family: string, file: string, weight = '400'): Promise<void> {
  const existing = loaded.get(family)
  if (existing) return existing
  const p = new FontFace(family, `url(${file}) format("woff2")`, {weight})
    .load()
    .then((face) => { document.fonts.add(face) })
    .catch((err) => { loaded.delete(family); throw err })
  loaded.set(family, p)
  return p
}

Each of the 16 cover styles maps to a pair of fonts (one for the big line, one for the small one), and we only load the pair the current style needs before drawing.

2. Make the title fit, line by line

Titles vary wildly: "Halo" versus "The Night We Drove to the End of the Coast". We try sizes from the largest down, wrap by words at each size, and stop at the first size where every line fits the safe width and the number of lines stays under the limit:

function fitLines(measure, text, maxWidth, maxLines, max, min) {
  const words = text.split(/\s+/).filter(Boolean)
  for (let size = max; size >= min; size -= 4) {
    const lines = []
    let current = ''
    let ok = true
    for (const word of words) {
      const next = current ? `${current} ${word}` : word
      if (measure(next, size) <= maxWidth) current = next
      else if (!current) { ok = false; break }
      else { lines.push(current); current = word }
    }
    if (current) lines.push(current)
    // Every line has to fit, including a long word pushed to the next line on its own.
    if (ok && lines.length <= maxLines && lines.every((l) => measure(l, size) <= maxWidth)) {
      return {lines, size}
    }
  }
  return {lines: [words.join(' ')], size: min}
}

That last every check came from a bug report in German. A flyer headline wrapped fine, but the long compound word that moved to the second line on its own was wider than the flyer. Checking only the line that triggered the wrap isn't enough.

Letter spacing is done by hand, character by character, with measureText per glyph. ctx.letterSpacing exists now, but drawing glyph by glyph gives the same result in every browser, and wide tracking is part of several styles (vaporwave, electronic).

3. Read the image under the text

White text with a soft shadow works on most generated art, until the model paints a pale pastel sky exactly where the title goes. So before drawing, we sample the band of the image the text will sit on. We downscale it to a 48 × 48 canvas, read the pixels and average their relative luminance:

function bandLuminance(image, iw, ih, y0, y1, size = 3000) {
  const s = 48
  const c = document.createElement('canvas')
  c.width = s
  c.height = s
  const ctx = c.getContext('2d', {willReadFrequently: true})
  const scale = Math.max(s / iw, s / ih)
  ctx.drawImage(image, (s - iw * scale) / 2, (s - ih * scale) / 2, iw * scale, ih * scale)
  const r0 = Math.floor((y0 / size) * s)
  const r1 = Math.ceil((y1 / size) * s)
  const data = ctx.getImageData(0, r0, s, r1 - r0).data
  let sum = 0
  for (let i = 0; i < data.length; i += 4) {
    sum += (0.2126 * data[i] + 0.7152 * data[i + 1] + 0.0722 * data[i + 2]) / 255
  }
  return sum / (data.length / 4)
}

Above a threshold (0.62 worked well in our tests) we switch to dark text and drop the shadow. Below it, we keep light text and paint a gradient "scrim" behind the band: a strip of semi-transparent black that fades out toward the middle of the image. The scrim strength is part of each style: heavy for metal, light for minimal.

4. The model sometimes paints a frame anyway

Even when the prompt asks for full-bleed artwork, a model will now and then return a picture inside a flat white or black border, like a print on a wall. Drawn at full size, that becomes a cover with a thick frame and the text sitting in empty margin.

We detect it on a small 160 × 160 copy. Starting from each edge, we count how many rows (or columns) are flat, meaning low color variance, all close to the color of the first one, and whether they end in a sharp cut where most of the line changes at once. A genuinely plain background, like a moon on a pink sky, doesn't end in a straight cut, so it is left alone. When a side shows a frame, we crop to the inside before drawing.

5. Upscaling honestly

Streaming platforms want covers at 3000 × 3000. The model returns about 1024 pixels, so the browser scales the image up when drawing it. One line made a visible difference:

ctx.imageSmoothingQuality = 'high' // the default is 'low'
ctx.drawImage(image, sx, sy, sw, sh, dx, dy, dw, dh)

It doesn't add detail that isn't there, so we tell users to check the file at 100% zoom before sending it to a distributor. The text, on the other hand, is drawn at full resolution, so the lettering is perfectly sharp even when the art is soft.

What we got, and what we gave up

Exact text, every time, including names with accents and umlauts that models love to mangle.
Editable text. Changing the title redraws the canvas in milliseconds instead of costing another generation.
A cheaper model. The image no longer needs a model that is good at lettering, so the cover runs on our standard model instead of a premium one.
Less typographic magic. Glow is a colored shadow, chrome is a vertical gradient, and neither is as rich as what a model can paint. For most covers that trade is worth it: a slightly plainer title spelled right beats a gorgeous one spelled wrong.

We use the same split, model for the picture and browser for the words, for book covers, posters, flyers, logos and cards. If you want to see the album version working, it's the AI album cover maker on CreArt, where the whole layout above runs client-side when you type your artist name.

If you've solved text on generated images differently, I'd like to hear how in the comments.