Every comparison of AI headshot tools follows the same shape. Someone uploads photos to four services, posts a grid of outputs, and declares a winner based on which grid looks nicest.
That tells you almost nothing, and the reason is worth spelling out, because the same flaw shows up in a lot of informal model evaluation.
The sampling problem
These tools return 40 to 120 images. The reviewer picks 4 to show you.
So what's actually being compared is not the models. It's the reviewer's taste applied to each model's output distribution, filtered through however long they were willing to scroll. Run the same test twice with the same person and you can flip the ranking by changing which four you pick.
If a comparison doesn't tell you the selection rule before it shows you images, the images are decoration.
Four things a real test would control
Same inputs, and enough of them. Face models are extremely sensitive to input variety. Fifteen photos from one afternoon in one room produce worse results everywhere than fifteen from across a year. If the reviewer used their camera roll for one tool and a fresh selfie session for another, the test is over before it starts.
More than one face. This is the big one and almost nobody does it. These models perform very differently across skin tones, ages, facial hair, and glasses. A single-subject test tells you how a tool handles that subject. Published comparisons routinely generalise from n=1 to "the best AI headshot generator".
A stated acceptance criterion, decided before looking. "Would I put this on LinkedIn" is fine as a bar. It just has to be written down first, and then applied to every image, so the output is a count. Tool A: 14 of 60 usable. Tool B: 31 of 60. That's a comparison. A grid of four is an advert.
Identity similarity, measured. You can put a number on "does it still look like me". Take a face recognition model, embed the input photos and each output, and compute cosine similarity. Above roughly 0.65 reads as clearly the same person to most observers, below 0.5 gets the "that's not quite you" reaction.
That last one converts the vaguest complaint about these tools into something you can plot. And it separates two failure modes that grids conflate: outputs that are bad photographs, and outputs that are good photographs of a stranger.
Why the usable-rate matters more than the ceiling
If you're picking a tool, the metric you care about is not how good the best image is. Every one of these produces some good images.
You care about the hit rate, because that's what determines whether you spend five minutes choosing or forty minutes hunting. A tool that returns 60 images with 30 usable beats one that returns 120 with 12, even if the single best image comes from the second.
Reviews almost never report this, because reporting it requires looking at every output, which is boring. Showing four is not boring.
Reading the comparisons that already exist
Given all of that, here's what I actually do with vendor comparison pages, which is where most of this content lives.
Read them for the claims, not the images. Turnaround time, number of styles, resolution, licence terms, whether edits cost extra. Those are checkable facts and a vendor page that gets them wrong is easy to catch.
Ignore the sample galleries entirely, including on pages I've worked on. Sample galleries are cherry-picked by construction. That isn't dishonesty, it's what a gallery is.
Comparison pages like BetterPic against HeadshotPro and the executive-focused breakdown against Aragon are useful for exactly that: the spec columns. Where they claim a difference in output quality, apply the sampling scepticism above, same as anywhere else.
There's also a longer-form review of HeadshotPro and BetterPic together that gets closer to a real methodology than most, though it's still a single subject, which is the limitation I'd want fixed before trusting any of them including ours.
The test you can actually run
The honest version costs you nothing:
- Use the free tier of two or three tools with the same 15 photos.
- Before looking, decide your bar. "Would I use this on LinkedIn."
- Count. Not rank, count.
- Divide the price by the count. That's your cost per usable image, which is the only number that matters.
Ten minutes, and it's specific to your face, which is the thing every published comparison cannot be.
More from this series
- 5 LinkedIn mistakes that filter engineers out before anyone reads the CV
- Background removal without a green screen is an underdetermined problem. Here's what the maths is doing.
- How AI headshot generators actually work: LoRA, 15 selfies, and the overfitting problem
- Bewerbungsfoto in Deutschland: seit 2006 freiwillig, trotzdem überall
On BetterPic
Top comments (0)