A face recognition model is not rotation- or scale-invariant, and it never needed to be: every image it trained on had the eyes in the same two places. Alignment is what makes that true at inference, and doing it slightly differently from the training pipeline is one of the quietest ways to lose accuracy.
Why recognition needs a canonical crop
A recognition network maps a fixed-size crop to an embedding, and it is trained on crops produced by one specific normalisation. Within that convention the network can spend its capacity on identity, because in-plane rotation, scale and translation have already been removed. Hand it an unaligned crop and it is out of distribution — not catastrophically, but enough that the same person photographed with a 15-degree head tilt lands further from their own gallery vector than from someone else’s.
This is why alignment is not an optional quality step. It is part of the model’s input specification, in the same way that a normalisation mean is. Two pipelines using the same recognition weights but different alignment produce embeddings that are not comparable, and a gallery built under one and queried under the other degrades without any error being raised anywhere.
Five points, and where they come from
The standard detectors emit landmarks alongside the box in one pass. MTCNN — Zhang and colleagues, “Joint Face Detection and Alignment Using Multi-task Cascaded Convolutional Networks” (2016) — is a three-stage cascade whose final stage outputs five points per face. RetinaFace is a single-stage detector predicting a box, the same five points, and a dense correspondence branch. The five are: left eye centre, right eye centre, nose tip, left mouth corner, right mouth corner.
“Left” there means the image’s left, which is the subject’s right. Swapping the two mirrors every aligned crop, and because a mirrored face is still a plausible face, nothing downstream complains — you simply get a systematically worse gallery. It is worth asserting the ordering against a known image once, in code, when integrating a new detector.
Dense models with 68 or more points exist and are used for expression, gaze and 3D fitting. Recognition pipelines almost universally use five, because the transform being estimated has only four degrees of freedom and five points already overdetermine it. More points add noise from landmarks that are harder to localise, not information.
The similarity transform, worked
The target is a canonical template: fixed pixel coordinates for the five points inside a fixed-size crop. The InsightFace/ArcFace template for a 112×112 crop, published in the InsightFace reference implementation, is:
left eye (38.2946, 51.6963)
right eye (73.5318, 51.5014)
nose (56.0252, 71.7366)
left mouth (41.5493, 92.3655)
right mouth (70.7299, 92.2041)
template interocular distance = 73.5318 - 38.2946 = 35.2372 px
eye line sits at y = 51.6, about 46% down the crop
Now take a detected face in a 1920×1080 frame with the left eye at (210.0, 180.0) and the right eye at (290.0, 196.0) — a head tilted so that the right eye sits lower in the image. A similarity transform has four parameters: rotation, uniform scale, and two translations. Two point correspondences determine it exactly.
dx = 290.0 - 210.0 = 80.0
dy = 196.0 - 180.0 = 16.0
measured interocular = sqrt(80^2 + 16^2) = sqrt(6656) = 81.585 px
roll angle = atan2(16.0, 80.0) = 0.19740 rad = 11.31 deg
scale = 35.2372 / 81.585 = 0.4319
a = scale * cos(roll) = 0.4319 * 0.98058 = 0.42351
b = scale * sin(roll) = 0.4319 * 0.19612 = 0.08471
source eye midpoint = (250.0, 188.0)
target eye midpoint = (55.9132, 51.5989)
tx = 55.9132 - (a*250.0 + b*188.0) = 55.9132 - 121.801 = -65.888
ty = 51.5989 - (-b*250.0 + a*188.0) = 51.5989 - 58.440 = -6.841
M = [ [ a, b, tx ],
[ -b, a, ty ] ]
= [ [ 0.42351, 0.08471, -65.888 ],
[-0.08471, 0.42351, -6.841 ] ]
Apply M with an affine warp into a 112×112 output and the eyes land on the template coordinates by construction. Every face processed this way has its eyes in the same two pixels regardless of tilt, distance or position in the frame — which is exactly the invariance the network was excused from learning.
In practice the transform is fitted to all five points rather than to the eyes alone, by least squares. Umeyama’s 1991 solution gives the optimal similarity transform between two point sets in closed form, and it is what the standard implementations use. It is more robust to a single badly-localised landmark than the two-point fit, and the mouth points constrain the vertical placement, which the eyes alone do not. The two-point derivation above is the same computation with three correspondences dropped, shown because you can check it by hand.
Crop, margin and the resampler
- Warp once, do not crop then align. Cropping the box, then rotating, then resizing resamples the image three times and blurs it. Compose the transform and apply a single warp from the original pixels to the 112×112 output.
- Antialias when minifying. The worked scale above is 0.43, a downscale of 2.3×. Bilinear sampling without prefiltering aliases badly at that ratio, and aliasing is position-dependent, so the same face at 300 pixels and at 80 pixels produces measurably different embeddings for no reason connected to identity. Use an area-averaging resampler or a Gaussian prefilter for minification. This is one of the most common silent accuracy losses in a face pipeline.
- Do not invent your own margin. The template fixes how much hair, jaw and background the crop contains. Adding “a bit of context” changes the input distribution the network was trained on.
- Match the colour convention. Many implementations expect RGB scaled as
(x - 127.5) / 128. Feeding OpenCV’s native BGR, or 0–255 where 0–1 was expected, degrades quietly rather than failing — the same class of bug catalogued in the production gap page. - Apply EXIF orientation before detection. Most image libraries do not rotate by the EXIF tag automatically, so portrait-orientation phone photographs arrive on their side and the detector finds nothing at all.
Where alignment fails, and for whom
A 2D similarity transform can undo in-plane rotation and scale. It cannot undo out-of-plane rotation. At large yaw one eye is self-occluded and its landmark is an extrapolation, so the estimated roll and scale are both wrong, and the aligned crop is a distorted face rather than a normalised one. The available responses are 3D-aware alignment or frontalisation, or maintaining pose-bucketed gallery entries so a profile query is compared against profile references.
Below roughly 30 to 40 pixels of interocular distance there is insufficient detail to localise landmarks stably, and upsampling does not create information — it creates a confidently wrong alignment. A minimum face size is a legitimate and necessary rejection criterion. Motion blur and rolling-shutter skew displace landmarks asymmetrically; masks and heavy-framed glasses displace the mouth and eye points specifically, which is worse than a uniform error because it biases the fitted transform rather than adding noise to it.
The differential-performance question runs through this stage too, and it is not neutral plumbing. NIST’s FRVT Part 3: Demographic Effects (NISTIR 8280, December 2019) documented false-positive rate differentials across demographic groups for a large number of submitted algorithms. Landmark detectors have their own differentials, and an alignment failure propagates multiplicatively into the embedding, so measuring only end-to-end recognition hides whether the cause is preprocessing or the recogniser. Evaluate landmark localisation error by subgroup separately; see accuracy and bias in facial recognition.
Face images are biometric data and are regulated as such in many jurisdictions — special category data under the GDPR, and covered by specific state biometric statutes in the United States. This page describes the mechanics of a preprocessing stage. Whether a given system may be built or operated at all is a legal question for counsel, not an engineering one.
Top comments (0)