The experiment, the blind judging, and the measurements in this post are the author's own, published first in Japanese. AI rewrote that write-up into this English, protocol-only version, added the flowchart, checklist and FAQ, wrote the illustrative Python, and cross-checked every number against the Japanese source. #ABotWroteThis
Short answer: strip one design element at a time, judge each variant blind with two independent AI judges plus an unrelated control image, and check any numeric metric against a second lens — in my run, the only change that broke recognition came with a 16% drop in hair-color saturation.
If you maintain an AI-illustrated character — a persona, a VTuber-style avatar, a recurring mascot — across enough generated art, you eventually run into a question that sounds trivial and isn't: change the outfit, and is it still the same character, or someone else wearing a similar face?
- The problem: "still looks right" is a gut call you can't defend when someone asks which part of the design was doing the work.
- The answer: strip one design element at a time, judge each variant blind with two independent judges plus an unrelated control image, and check any numeric metric against a second lens. In my run, costume, hardware, and hairstyle changes survived; the variant with a 16% saturation drop failed, its color-locked rebuild passed, and a plain color-distance metric ranked that failure backwards.
- Do this next: if you want to run the same test, the 7-step sequence near the end is the one I would reuse on a second character; the sections in between show why each step exists.
For a long time my answer was a gut check: does this still look right? That works fine right up until someone asks you to defend the call, and you notice you can't actually say which part of "looks right" was doing the work.
Quick answer
Strip the character's design down one visual element at a time — hardware, hairstyle, costume, color — and run each variant through a blind, two-judge same/different check, with one unrelated-character control image mixed in to catch judges that default to "same" out of habit. In the run below, hardware, hairstyle, and costume changes — including a hairstyle-plus-costume change stacked together — didn't break recognition. The variant that failed carried a 16% drop in hair-color saturation, and rebuilding that same variant with the color locked to reference passed — while a plain color-distance-to-reference metric missed all of this, ranking the failing variant as more faithful than the one that passed. The full protocol, the numbers, and the three design mistakes I hit along the way are below.
So I stopped deciding by feel and ran an experiment instead: strip the character's design down one element at a time, and measure — with a blind, two-judge protocol — how far it goes before an outside viewer stops calling it the same character. This is that protocol, the result that surprised me, a metric that would have quietly handed me the wrong answer if I'd trusted it alone, and three ways the experiment itself was broken before it produced anything trustworthy. The character itself isn't the point here. It appears on the cover in three of the looks from the run, but the text below never spells out its design specification — what's reusable is the method, and that's the whole subject of this post.
TL;DR
- I stripped one visual element at a time from an AI-illustrated character and ran each variant through a blind, two-judge same/different check.
- Hardware, hairstyle, and costume changes — hardware on its own, hairstyle on its own and stacked with a costume change — didn't break recognition, including a combination I expected to fail.
- The failing variant carried a drop in hair-color saturation, and rebuilding it with the color locked to reference passed — the color change is the one consistent with both runs. A naive "distance to reference" color metric missed it, and actually ranked the failing variant as closer to official than the version that passed.
- I hit three design mistakes along the way — a filename that leaked the answer, a confounded variable, and (in a separate body-proportion check) a measurement diagram that didn't represent what it claimed to. The first two forced redos of the blind verdicts above; the diagram was rebuilt on its own track.
The protocol: strip one element, ask one blind question
The setup has four parts, each one closing a specific hole in "I looked at it and it seemed fine."
One variable at a time. Take the character's design spec and list what could plausibly change: hardware, hairstyle, costume, color. Generate one variant per element, everything else held fixed. Change two things in the same image and a "different" verdict tells you nothing about which change caused it — a mistake I made once, below.
Blind judges. Neither judge sees a filename, caption, or any other hint about what was changed, or whether anything was changed at all — just an image and one question. This rule exists because of a real failure the first time I skipped it; that's the first item in the failures section below.
One question, asked the same way every time. "Is this the same character as the reference, yes or no?" Nothing about art quality, nothing about pose. Folding in a second axis lets a judge dodge the identity question by scoring the drawing instead.
Two independent judges, plus a control. A single judge can't tell you whether a "same" verdict is about the image or about that judge's own bias toward agreement. That is the whole reason for the second judge: when two independent judges reach the same verdict, I treat it as a property of the image rather than of one judge's habits, and when they disagree, that's not a bug to resolve by picking a side, it's data: record it as a boundary case instead. To check the judges weren't just defaulting to "same" out of habit, I mixed in one image of a completely unrelated character as a control. Both judges called it "different." One control image is a small check, not proof — I read it as "the judges were not answering 'same' by reflex," and nothing stronger.
"Two independent judges" here means two blind calls to a multimodal model, not two people — the run record logs them as two independent AI judges, each shown the image with the change concealed. They share the underlying model, so I read their agreement modestly: as two blind votes, not as two unrelated observers.
What survived getting stripped
| Element changed | Verdict |
|---|---|
| All visible hardware removed | Same — both judges |
| Hairstyle changed (three variants: cropped short, grown long, braided) — alone, and each paired with a different costume | Same — both judges |
| Costume changed together with a hairstyle variant (denim overalls, a heavy knit sweater, a raincoat; the work jacket appears only in the hardest combined case below) | Same — both judges |
| Unrelated character used as a control | Different — both judges, correctly caught |
I expected some of these to fail. (The costumes were not tested as a lone variable: per the run record, each was worn with one of the hairstyle variants, and the work jacket only in the hardest combined case.) Hardware especially — I removed every piece I could identify: the front-placket buttons, shoulder tabs, a waist belt, cuff buckles, boot trim, an ankle strap, and a cross-body strap — seven pieces in total. None of it moved the verdict. Whatever was carrying this character's identity, it wasn't the surface detail I'd assumed was doing the work.
The change that tracked the failure
What separated the failing run from the passing rebuild wasn't a shape at all. It was color — specifically, how saturated the hair color read.
To my eye it wasn't a shift to a different color family — the hair read as the same color, just duller. I didn't measure hue separately; what I logged was a channel-spread metric on the cropped hair region — roughly, how far apart the red and blue channel means sit, used here as a stand-in for how vivid the color reads. I logged it across four categories: the official reference art, a variant that had passed blind judgment, a variant that had failed it, and the unrelated control character.
| Category | Spread value |
|---|---|
| Official reference | 57.6 |
| Passing variant | 56.2 |
| Failing variant | 48.3 |
| Unrelated control character | 10.7 |
Worth being honest about what this table is: a logged record, not a number I can hand you a script to reproduce today. The specific crops that produced it were discarded in the course of the work, so I can't re-run this exact measurement and get 48.3 back out. I'm citing it as what was recorded at the time, not as something you should expect to reproduce byte for byte.
What it does support is arithmetic anyone can check from the table above: the failing variant sat 16% below the official reference on this metric — (57.6 − 48.3) / 57.6 ≈ 16% — while the passing variant sat close to the official number. That's what separated the failing run from the passing rebuild, at least for this one character in this one test: a double-digit percentage drop in how saturated the hair color looked.
The metric that would have quietly lied to me
Saturation isn't the only lens you'd reach for. The obvious first move is simpler: how far is this variant's color, in plain RGB space, from the official reference? Closer should mean more faithful.
I checked that too, with a plain Euclidean distance on the hair region — this check was run later, while fact-checking the write-up, on freshly drawn crops of the surviving images (the original crops were already gone). Mean RGB per image, near-white background pixels excluded so the crop's edges don't pull the average toward white. A minimal version of the script I ran looks like this:
import numpy as np
from PIL import Image
def mean_hair_color(path: str) -> np.ndarray:
"""Mean RGB of a manually cropped hair region, near-white background excluded."""
pixels = np.array(Image.open(path).convert("RGB")).reshape(-1, 3).astype(float)
is_background = (pixels[:, 0] > 235) & (pixels[:, 1] > 235) & (pixels[:, 2] > 235)
return pixels[~is_background].mean(axis=0)
def distance_to_reference(sample_rgb: np.ndarray, reference_rgb: np.ndarray) -> float:
return float(np.linalg.norm(sample_rgb - reference_rgb))
In that re-measurement, the failing variant measured 13.5 from the official reference; the passing variant measured 16.7. The failing variant looked closer to official than the passing one did. If I'd trusted this metric alone, I would have shipped the broken version and rejected the good one — backwards, with a straight face and a clean number to back it up.
This wasn't a fluke of exactly where I drew the crop box, either. I re-ran it with the crop boundary varied three ways — the base crop, one shrunk inward from it, one grown outward from it — and the failing variant came out closer to official than the passing variant in all three. The ranking held; only the exact distances moved around with the boundary.
The reason a raw distance metric misses this is worth sitting with: a color can drift toward gray — lose saturation — while its position in RGB space stays numerically close to where it started, especially if the hue itself barely moves. Distance-to-reference and saturation-of-reference are measuring genuinely different things, and only one of them tracked the failing variant here. The lesson I take isn't "use saturation specifically" — it's that here, a single distance-style metric ranked the failing variant as more faithful than the passing one, and the only thing that surfaced it in this run was comparing it against a second lens (the spread metric).
Three ways I got the experiment itself wrong
The results above are the ones I trust. Two of the mistakes below forced redos of those verdicts — the leaked filename and the confounded variables; the third surfaced in a separate body-proportion check and cost a rebuild of that diagram, not of the verdicts. I only have this one pipeline to show, but none of the underlying mistakes look specific to character art to me: a label that leaks the answer, a variable that moves uninvited, scaffolding nobody checked. I'd watch for the same three in any measurement setup I build next.
1. The filename told the judge the answer. The first round used the real, descriptive filenames — the surviving files still carry names like "notag". As the Japanese write-up records it, one judge's stated reasoning referenced that filename rather than the picture; I no longer have the raw judgment log to quote, so take that as the write-up's account, not a transcript. The fix is the part the later records do show: every image got a neutral, sequential name before a judge saw it. I only observed this with an AI judge, but the neutral-name rule stays regardless of who judges — a labeled filename hands any rater the same hint.
2. Changing three things at once confounded the result. For the hardest test case — hairstyle, costume, and the hair accessories all changed together — the hair color drifted slightly too, unintentionally, as a side effect of regenerating the image. When that version came back "different," I couldn't tell whether removing the hair accessories or the color drift had caused it — the hairstyle variants had already passed on their own, and the costumes had passed in combination with them. Given the saturation finding above, the color drift is the more likely culprit, but the test as designed couldn't separate the two. Fix: rebuild the image with the hair color locked precisely to the reference, so only the intended variables move, then re-run. The rebuilt version passed blind judgment.
3. The measurement scaffolding didn't represent what it claimed to. Separately, I built a skeleton overlay — a diagram connecting joint positions with lines — to sanity-check that costume changes weren't distorting body proportions. The first version treated each joint as an isolated point to plot, with no anatomical hierarchy connecting them: it never established crown-to-jaw-to-neck before jumping to the shoulders. Once a coat or hood got layered on top, the lines drifted off the actual body outline, because the diagram had never encoded where the neck connects to the head in the first place. The fix was to rebuild it in real skeletal order — crown, jaw, neck, shoulder, arm, leg — anchored to a pose estimate instead of hand-plotted points.
Different surfaces, same root cause: in all three cases, something was standing in for ground truth — a filename standing in for image content, one image standing in for "nothing else changed," a set of points standing in for body structure — without anyone checking that the stand-in actually represented what it claimed to. That check is worth doing before you trust any measurement, not just this one.
Passing blind judgment isn't the same as feeling confident
One more result is worth reporting because it doesn't resolve cleanly. The rebuilt hardest-case image — hairstyle, costume, and hair accessories all changed, hair color locked to reference — passed blind judgment: both judges called it the same character. My own reaction looking at it wasn't nearly that clean. I'd believe it if someone told me it was the character; I'm honestly not sure I'd have called it correctly if nobody told me anything first.
I don't read that as the measurement failing. My working hypothesis — drawn from exactly this one gap between the judges' pass and my own hesitation, not from comparing viewer groups — is context: the same image can land differently depending on whether the viewer already knows the character. On that hypothesis, a first-time viewer working from the image alone needs it to carry more identity signal, while a viewer who already knows the character fills in the rest — the way you'd still recognize a coworker who changed their haircut. The analogy is how I picture it, not something this experiment tested.
Practically — and this is a policy I adopt on the safe side, not a measured result: a blind pass is a floor, not a green light by itself. For anything going to an audience that's never seen the character before, I'd want a wider safety margin than blind judgment alone certifies; for an established audience, the same variant may have more room than the raw pass/fail suggests.
The 7-step sequence I would reuse on a second character
I have only run this on one character, in one session, so the results above are all I can vouch for. What I can add about the procedure is only what is visible in the steps themselves: none of them name this character's assets — they read "pick one element, generate one variant, hide the labels, add a control, ask two blind judges." If I test a second character, this is the sequence I plan to reuse. Treat it as my run sheet from this one experiment, not as a validated general method:
List the variable candidates from your character's design spec — anything you might plausibly want to change later.
Generate one variant per element, holding everything else fixed. Resist testing two changes in one image; you saw above what that costs you.
Strip filenames and any metadata that could hint at what changed before a judge sees the image.
Use two or more independent judges. When they disagree, record it as a boundary case instead of forcing a single answer.
Mix in a control image — a character that has nothing to do with yours — to catch judges (AI or human) that default to "same" out of habit.
If you're using a numeric metric, check it against a second one before trusting a ranking. A metric that agrees with itself across two lenses is worth more than one that just scores well by itself.
Keep two separate records: what measured as not required for recognition, and what you're keeping anyway as a deliberate call. They come from different kinds of evidence — one's an experimental result, the other's a decision — and collapsing them into a single "rules to follow" list will cost you the reasoning the next time someone asks why a rule exists.
Step 7 is the one I underrated going in. Measuring that an element is droppable and deciding to actually drop it are two different jobs. The first is what the blind test answered for me in this run. The second is a call about what your character should look like, and a passing grade on the first doesn't make that call for you — it just gives you a real answer to make it with, instead of a guess.
What this write-up gives you
- A one-variable strip test you can run with blind, operationally separated judges plus one control image.
- A way to read a same/different verdict as evidence, while keeping what you choose to drop as a separate design record.
- The habit of checking any single numeric metric against a second lens before trusting its ranking.
- A stand-in check: before trusting a measurement, verify that filenames, single images, and plotted points actually represent what they claim to.
FAQ
Do I need a research team to run this protocol?
I didn't, for this run. My "two independent judges" were two independent calls to a multimodal model — no lab involved. What I actually relied on is what the run record shows: two independent, blind AI judgments per image, with the change concealed. They share the underlying model, so I do not read their agreement as two unrelated observers — and any correlated bias two runs of the same model might share is beyond what this run can rule out.
How many judges do I actually need, and what if they disagree?
I used two independent calls, plus one control image (a character unrelated to mine). The second judge is there so a verdict isn't one judge's habit; the control is there to catch reflex "same" answers — that division of labor is why I wouldn't run it with fewer, though I haven't tested other counts. If the judges disagree, I record it as a boundary case instead of forcing a tiebreak — disagreement is data, not a bug to resolve by picking a side. (In this run, the recorded verdicts were agreements.)
Does passing blind judgment mean a redesign is safe to publish?
Not by itself — that's the policy I run, not a measured result. A blind pass is a floor, not a green light. My untested working assumption is that a first-time viewer needs more identity signal carried in the image alone while a returning viewer fills in from context, so for a redesign aimed at a new audience I take a wider safety margin than the raw pass/fail suggests.
Why not just measure color distance to the reference and call it done?
Because a single distance-style metric can rank the failing variant as more faithful than the passing one. A color can drift toward gray — lose saturation — while staying numerically close in RGB space, especially when hue barely moves. Check any one metric against a second, differently-shaped metric before trusting its ranking.
What's the difference between "not required for recognition" and "safe to drop"?
They're different kinds of evidence. In this run, the blind test recorded which elements could go without breaking recognition for this one character — that's an experimental result. What you actually choose to drop is a design decision layered on top of that result. Keep both as separate records; collapsing them into one list loses the reasoning behind the choice.
This started as a Japanese write-up about testing my own AI-illustrated character's identity boundaries. The English version here is an AI-assisted rewrite focused only on the measurement protocol — the character's own design details, and a couple of asides specific to that original piece, aren't reproduced here on purpose. The distance figures were re-run and confirmed against the surviving images while fact-checking the Japanese source; the spread-value table can no longer be reproduced because its crops are gone, and I've said so rather than presenting it as something you can re-run. Everything above reports one character in one production run; the Japanese source's fact-check ledger is the record it was checked against.
Top comments (0)