“99.7% accurate” is not a property of a face recognition system. It is a property of a system, a threshold, and a population, and changing only the third moves it by one to two orders of magnitude. The published audits do not show a mysterious bias; they show a measurable and mechanically explicable difference in error rates between subgroups, and it is that difference, not the headline, that a buyer should be asking for.
The claim
The argument of this page is narrow and, as far as the evidence goes, hard to escape. A face matcher outputs a similarity score. Somebody chooses a threshold on that score. If the distribution of impostor scores differs between demographic groups — and it does, and this has been measured at scale by a national metrology institute — then any single global threshold necessarily produces different false match rates for different groups. The threshold is not a technical detail delegated to the vendor. It is the mechanism by which error is allocated between populations, and it is a policy decision wearing a configuration file’s clothes.
Two clarifications before the evidence, because most public argument about this conflates things that behave differently. First, verification (one-to-one: is this the person on the passport?) and identification (one-to-many: is this person anywhere in a database of a million?) have different error structures, and a differential that is tolerable in the first is not in the second. Second, face recognition and face attribute classification — guessing gender or age from a face — are different tasks with different literatures, and the most-cited study is about the second.
What the audits actually measured
The largest evaluation is NIST’s. In December 2019, Patrick Grother, Mei Ngan and Kayee Hanaoka published NISTIR 8280, Face Recognition Vendor Test Part 3: Demographic Effects, covering 189 algorithms from 99 developers against roughly 18.27 million images of about 8.49 million people, drawn from US government operational datasets. It is not a benchmark somebody assembled from scraped web photos; it is a controlled evaluation with a fixed protocol run by a metrology institute, which is why it is the right citation when someone claims the effect is a myth.
Its findings, in NIST’s own framing: in one-to-one matching, the majority of algorithms showed higher false positive rates for Asian and African American faces relative to images of white subjects, with differentials that in many cases ranged from a factor of 10 to a factor of 100. Among algorithms developed in the United States, the highest false positive rates were often on Native American, American Indian, Alaskan Indian and Pacific Islander faces. In one-to-many matching against a database of 1.6 million FBI mugshots, false positives were elevated for African American women — the case with the most direct consequence for a person, since a false positive there is a wrongly generated investigative lead.
The finding that carries the most mechanistic weight is the exception. NIST reported that algorithms developed in Asian countries did not show the same dramatic Asian-versus-white false positive gap in one-to-one matching. Grother’s stated inference was that the composition of the training data is implicated. That is precisely the shape of evidence needed to move from “there is a disparity” to “here is why”: the disparity tracks the developer’s likely data, not the demographic group, and it reverses when the data does.
The earlier and more famous result is about attribute classification. In 2018, Joy Buolamwini and Timnit Gebru published Gender Shades at the first Conference on Fairness, Accountability and Transparency. They showed that the two facial analysis benchmarks then in common use, IJB-A and Adience, were heavily skewed toward lighter-skinned subjects; built a new benchmark balanced across skin type using the Fitzpatrick scale; and evaluated three commercial gender classifiers on it. Error rates for darker-skinned women reached 34.7%. For lighter-skinned men the maximum error was 0.8%. Both parts matter and the first is often dropped: the paper is as much about the benchmarks being unable to detect the problem as about the classifiers having it.
NISTIR 8280 is a snapshot of 2019 submissions and NIST re-runs FRVT continuously; several vendors have submitted substantially revised algorithms since. Check the current FRVT reports before quoting any specific differential as the state of the art. The mechanisms below do not expire; the magnitudes do.
Mechanism one: training-set composition
A face recognition model learns an embedding in which images of one person are close and images of different people are far apart. It learns which facial variations are identity-bearing and which are nuisance from the examples it is shown. If the training set contains far more identities from one population, the model gets far more gradient signal about which variations distinguish people within that population, and correspondingly less about the others. The result is not that the model “cannot see” under-represented faces; it is that it embeds them into a tighter region, so unrelated people in that region are more similar to each other than the equivalent pair in the well-represented region.
That geometric consequence is exactly what a false match rate measures. Compress a subpopulation into a smaller volume of the embedding space and the impostor score distribution for that subpopulation shifts upward, so more impostor pairs exceed any fixed threshold. This also explains why the effect appears much more strongly in false positives than in false negatives: a genuine pair of images of the same person remains close regardless, while the impostor distribution is what moves.
It also explains why balancing is harder than it sounds. What matters is not the number of images but the number of distinct identities, and the difficulty of the impostor pairs among them — ten thousand images of a hundred people contribute far less discriminative signal than one image each of ten thousand people. The general problem is treated in dataset balancing; the face-specific version is severe because the large public training sets were assembled from web images of public figures, whose demographic composition was never chosen by anybody.
Mechanism two: one threshold, several distributions
Suppose a vendor sets its threshold to achieve a false match rate of 1 in 100,000 on an internal evaluation set. That number is an average over whatever population that set contained. If group A’s impostor scores sit lower and group B’s sit higher, the same threshold might deliver 1 in 300,000 for A and 1 in 10,000 for B, while the headline figure remains honest as an average.
Nothing about this requires malice or even carelessness in the threshold-setting. It is arithmetic: a single cut point on two different distributions gives two different tail masses. The consequence is that a system procured on a single FMR specification has no defined error rate for any actual person, and the specification cannot be audited from the outside without per-group evaluation data.
The available responses are all uncomfortable and should be argued for explicitly rather than defaulted into. Per-group thresholds equalise FMR but require classifying every subject into a group at match time, which many jurisdictions and most ethics reviews will not accept. Raising the global threshold until the worst group meets the target equalises nothing but bounds the harm, at the cost of more false negatives for everybody — and false negatives have their own distribution, which also has to be reported. Improving the training data attacks the cause but is slow and its effect must be measured, not assumed. NIST’s own observation that the most equitable algorithms were also among the most accurate is the strongest available argument that this is a solvable engineering problem rather than a fixed trade-off — but it is an observation about the algorithms submitted, not a theorem.
Mechanism three: base rates in one-to-many search
Identification search multiplies the problem in a way that is easy to miss. Searching a probe against a gallery of N identities performs N comparisons, so the expected number of false matches per search is approximately N × FMR. At an FMR of 1 in 100,000 against a gallery of 1.6 million, that is roughly 16 spurious candidates per search before considering any demographic effect at all.
Now apply the differential. If the effective FMR for one group is ten times higher, that group generates ten times the spurious candidates per search, and every one of those candidates is a person who becomes the subject of attention on the basis of an arithmetic artefact. This is why NIST separating one-to-one from one-to-many matters so much, and why a vendor accuracy figure derived from verification says essentially nothing about behaviour in an identification deployment.
There is a related and separate failure earlier in the pipeline. Face recognition is preceded by detection and alignment, and if the detector misses a face or the landmark model places eyes badly under certain skin tones or lighting, the recognition model receives a degraded crop and its scores degrade with it. That is a distinct source of differential from the embedding geometry above, and it is measurable separately — see what detection and alignment do before recognition runs. Auditing the end-to-end system without separating the two stages leaves you unable to say which one to fix.
What follows for anyone deploying one
The concrete request that follows from all of this is a reporting change, and it is not onerous. Ask for false match rate and false non-match rate reported per demographic group, at the exact operating threshold you will run, on a population resembling yours — not the vendor’s aggregate figure, and not a score on a benchmark whose composition was chosen by somebody else. Where the vendor participates in FRVT, ask which submission and which report date, since the algorithm evaluated is often not the one shipped.
Ask also what happens on a match, because the error rate’s consequence is set by the process around it. A false match that produces a second authentication factor is an inconvenience; a false match that produces an investigative lead is not. And measure the deployed system rather than the model, because camera placement, lighting and image quality shift the operating population away from anything a lab evaluated — a specific instance of the general gap between benchmark and production accuracy.
This page describes measurement and mechanism. Whether a given face recognition deployment is lawful or permissible is a legal and regulatory question that varies by jurisdiction and is not addressed here; several jurisdictions restrict or prohibit specific uses outright. Nothing above should be read as a view on whether a particular system should be deployed.
Top comments (0)