One photo, three face shape tools, three answers: how an on-device classifier works

I run The Face Report, a site with a face shape detector that works in the browser. This month I put one portrait through it and through two other free face shape tools. As far as I can tell, all three use the same 478-point MediaPipe face mesh. They gave three answers: Heart at 88%, Round at 72%, and Oval, borderline with Diamond.

The landmark model does not decide a face shape. It gives you points. Someone has to write the rule that turns those points into a label, and that rule is where the tools differ. This post shows how ours works, with code you can run, and what I got wrong.

Get the landmarks

import { FaceLandmarker, FilesetResolver } from "@mediapipe/tasks-vision";

const files = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.32/wasm"
);
const landmarker = await FaceLandmarker.createFromOptions(files, {
  baseOptions: {
    modelAssetPath:
      "https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task",
    delegate: "GPU",
  },
  runningMode: "IMAGE",
  numFaces: 1,
});

const mesh = landmarker.detect(imageElement).faceLandmarks[0];

mesh holds 478 points. Their coordinates are normalized to the image width and height separately, so convert them to pixels before you measure anything. On a portrait photo, distances taken from the raw values are wrong.

Turn eight points into four numbers

Measure	Landmarks
Face height	10 (top of the mesh) to 152 (chin)
Forehead width	54 to 284
Cheekbone width	234 to 454
Jaw width	172 to 397

const px = (i) => ({ x: mesh[i].x * width, y: mesh[i].y * height });
const dist = (a, b) => Math.hypot(a.x - b.x, a.y - b.y);

// Angle at vertex v between points a and b, in degrees.
function angleAt(v, a, b) {
  const [ax, ay, bx, by] = [a.x - v.x, a.y - v.y, b.x - v.x, b.y - v.y];
  const cos = (ax * bx + ay * by) / (Math.hypot(ax, ay) * Math.hypot(bx, by));
  return Math.acos(Math.max(-1, Math.min(1, cos))) * (180 / Math.PI);
}

const [top, chin] = [px(10), px(152)];
const [foreheadL, foreheadR] = [px(54), px(284)];
const [cheekL, cheekR] = [px(234), px(454)];
const [jawL, jawR] = [px(172), px(397)];
const cheekWidth = dist(cheekL, cheekR);

const ratios = {
  heightToWidth: dist(top, chin) / cheekWidth,
  foreheadToFace: dist(foreheadL, foreheadR) / cheekWidth,
  cheekToJaw: cheekWidth / dist(jawL, jawR),
  jawAngle: (angleAt(jawL, chin, cheekL) + angleAt(jawR, chin, cheekR)) / 2,
};

There is no forehead-to-jaw ratio. It equals foreheadToFace * cheekToJaw, so adding it would count the same geometry twice.

Find the nearest profile

Each shape is a set of ideal values for those four numbers. The classifier measures how far a face is from each set, in units of tolerance, and picks the nearest.

const WEIGHT = { heightToWidth: 5 / 17, foreheadToFace: 4 / 17, cheekToJaw: 5 / 17, jawAngle: 3 / 17 };
const TOLERANCE = { heightToWidth: 0.1, foreheadToFace: 0.07, cheekToJaw: 0.1, jawAngle: 10 };
const PROFILES = {
  oval:    { heightToWidth: 1.24, foreheadToFace: 0.88, cheekToJaw: 1.26, jawAngle: 137 },
  round:   { heightToWidth: 1.13, foreheadToFace: 0.88, cheekToJaw: 1.2,  jawAngle: 141 },
  square:  { heightToWidth: 1.16, foreheadToFace: 0.9,  cheekToJaw: 1.2,  jawAngle: 130 },
  heart:   { heightToWidth: 1.24, foreheadToFace: 0.92, cheekToJaw: 1.37, jawAngle: 134 },
  oblong:  { heightToWidth: 1.34, foreheadToFace: 0.88, cheekToJaw: 1.24, jawAngle: 137 },
  diamond: { heightToWidth: 1.24, foreheadToFace: 0.84, cheekToJaw: 1.31, jawAngle: 134 },
};

function classify(ratios) {
  const ranked = Object.entries(PROFILES)
    .map(([shape, ideal]) => {
      let sum = 0;
      for (const key of Object.keys(WEIGHT)) {
        sum += WEIGHT[key] * ((ratios[key] - ideal[key]) / TOLERANCE[key]) ** 2;
      }
      return { shape, distance: Math.sqrt(sum) };
    })
    .sort((a, b) => a.distance - b.distance);

  const [best, next] = ranked;
  return {
    shape: best.shape,
    runnerUp: next.shape,
    matchStrength: Math.round(100 / (1 + best.distance)),
    borderline: next.distance - best.distance < 0.25,
  };
}

classify({ heightToWidth: 1.22, foreheadToFace: 0.88, cheekToJaw: 100 / 78, jawAngle: 135 });
// { shape: 'oval', runnerUp: 'diamond', matchStrength: 85, borderline: true }

That call uses the portrait from the start of this post: cheekbones 100, forehead 88, jaw 78, height 1.22 times the width.

matchStrength is not a probability. It is 100 for an exact match with a profile and falls as the distance grows. I would read a percentage on any tool like this the same way unless the tool says how it was measured.

What I got wrong

I started from textbook ratios. Textbooks measure from the hairline, and the top of the mesh sits below it, so height reads short. The textbook values sat outside the 5th to 95th percentile of real scans on most measures. Square had an ideal jaw angle of 100 degrees when the 5th percentile of real scans was 128. Most faces came out oval by default. I re-centred the profiles on 2,217 production scans and kept each shape's position relative to the others.

That is calibration to a distribution, not an accuracy benchmark. I have no labelled dataset, so I cannot tell you how often the label is right. Oval is still the largest group: 49.2% of 15,429 scored scans on one day in August.

The first score was clipped. Each ratio scored inside a band and stopped changing outside it, so a very long face could lose the information that it was long and land nearer round than oblong. Unclipped distance keeps growing, so the direction survives.

Tolerances have to match across shapes. If one profile has a wider tolerance on a measure than the others, it wins at extreme values by default. Each measure here has one tolerance shared by all six shapes.

A single label hides close calls. When the runner-up is within 0.25 distance units, the result says so and names both shapes. The portrait above is one of those.

The failure that did not throw

On Android, in about 1% of scans, the GPU delegate's texture upload failed silently. The model ran on a blank frame and returned one off-scale mesh, usually with coordinates in the hundreds, and the same pose on every photo. No exception was raised, so the app measured it and reported a result. One person scored 9 out of 100 from a mesh that was not on their face.

The fix is a range check on every point and one retry on the CPU delegate:

const plausible = (points) =>
  points.every((p) => [p.x, p.y].every((v) => Number.isFinite(v) && v >= -2 && v <= 3));

Landmarks can fall a little outside 0 to 1 when a face is cut off at the edge of the frame, so the range is wider than that. Across 48,000 scans, healthy meshes peaked at 1.07 (99.9th percentile) and the bad mesh sat at 5.6 or beyond, which leaves plenty of room between them. If the CPU pass also fails the check, the tool shows no result.

One smaller finding: the landmarker crops the face and runs at 192 px internally, so I cap the input at 2,048 px on the long edge. In tests that moved points by at most 2.9 px RMS and scores by 1, and it spares the GPU a 36-megapixel texture.

What leaves the browser

For this tool the photo and the 478-point mesh stay on the device. The result does not: the shape label, the ratios and the scores are posted to the server, without the mesh, so a person can reopen their result. Free results are deleted after 2 days, and copies of the scores and labels go to usage analytics. You can see the request in the network tab.

Other tools on the same site, such as the face age estimate, use a vision model and do send the photo, after a consent step. "On-device" is a claim about one code path, so it has to be checked per tool.

The open problem

I wrote up the production version, with the result screen, in how our face shape detection works. The part I do not have is labelled data. If you have built something like this and found a way to measure accuracy without a labelled set, I would like to hear how.

This article was written with AI assistance (Claude), working from the production code. The code samples were checked against the production classifier, and I reviewed the article before publishing.