Every comparison of portrait generators reaches a different conclusion. Some of that is commission. Most of it is that the evaluation was designed badly, and the failure modes are the same ones you get in any ML evaluation.
Here is a harness that avoids them.
The class imbalance nobody controls for
These models fail unevenly across the input distribution. Strong-prescription glasses, tightly curled hair, deeper skin tones, pronounced asymmetry. Those are the tail, and the tail is where products differ.
A test set of three cooperative faces samples only the head of the distribution. You will measure a tie and report a winner.
Stratify deliberately:
STRATA = ["glasses_strong", "hair_high_volume", "skin_tone_deep",
"facial_hair_distinct", "asymmetry_marked", "baseline_easy"]
# at least two subjects per stratum, and report per-stratum rather than pooled
Pooled accuracy on an unbalanced set is the single most common way these comparisons go wrong.
The two axes people collapse into one
Photographic quality and likeness are separate and products trade one for the other. Heavier stylisation raises the first and lowers the second.
Score them independently or the ranking is meaningless. Likeness is the harder one to automate, and the cheapest reliable instrument is a human blind test: mix five generated with five real, hand them to someone who knows the subject, and measure discrimination rather than preference.
Face-embedding cosine similarity is tempting and misleads. Two images can sit close in embedding space and still look like a sibling rather than the person, because the embedding was trained for identification under variation, which is a different objective from "reads as the same individual".
Input hygiene, which dominates everything
The most common self-inflicted error: fifteen photos from one session. The model cannot separate the subject from the shirt, so it learns both.
assert len({p.date for p in photos}) >= 4, "vary the days"
assert len({p.garment for p in photos}) >= 3, "vary the clothing"
Without those two assertions you are benchmarking how gracefully each product degrades on bad input, which is a real property and not the one you are claiming to measure.
Latency: measure the interval users feel
Vendors quote start-to-first-output, which is compute. Log four timestamps and the useful interval falls out:
t0 upload complete t1 job queued t2 first output t3 first output downloaded
t3 - t0 is what people experience. A run returning 100 images of which four are usable costs more human time than one returning 30 of which twelve are, and no published figure captures that.
Where to see the method rather than the verdict
We publish comparisons and we are a participant, so read them for the criteria: the generator roundup, the app-first comparison, and what happens when you try this in ChatGPT, which is a different architecture with no fine-tuning step and therefore much weaker likeness.
Whether a genuinely free option exists is the fair version of the pricing question, and how we got here as a company is context for the obvious conflict of interest.
The twenty-minute version
One hard face, fifteen photos across four days, two products the same afternoon, blind discrimination test by someone who knows them. That beats every published comparison including ours, because it samples the only part of the distribution you care about.
More on this: Yield is the only metric that matters, and no vendor publishes it
Disclosure: the harness above was built against the AI headshot generator I work on.
Top comments (0)