Image models are nondeterministic functions. Same input, different output, and the variance is controlled by a seed you can pin. Anyone who has written tests against a flaky service already knows what that implies: a single call proves nothing. You need a second call with one variable changed, or you are reading noise as behavior.
The comparisons I read mostly skip this. One image per product, results side by side, call it a benchmark. So I ran a proper version on one anime image platform, with a control run for every claim.
The fixture
I built one test character with assertable properties, the way you would design a fixture rather than a mood board:
- dark brown bob cut straight at the jawline
- pale grey green eyes
- a mustard base layer whose left sleeve turns charcoal below the elbow
- exactly four tool loops on the right thigh
- a charcoal patch on the left knee
Two of those are side-specific, one is countable, two are pure color. Side and count are the ones worth including, because a symmetrical character with no countable elements passes every test by accident.
All runs used the same model, mode and settings, with only the stated variable changed between a run and its control.
Check 1, prompt following with a batch instead of a single call
Ten checkable instructions in one prompt, generated four times with nothing changed but the seed.
Six instructions appeared in all four outputs: the camera framing, which hand held which object, the two-color sleeve on the correct arms, the loops on the correct thigh, the patch on the correct knee, and the eye color. Those are model behavior and you can build on them.
One failed in all four: the requested low camera angle never arrived. The count landed once, as four straps on a tool roll the model invented, and came back as six, seven and six in the others.
I later ran the same prompt four more times with a single color word changed. Across those eight generations, the correct count of four appeared twice. A single sample would have supported either conclusion, which is the argument for control runs in one number.
One more thing surfaced only because there were four outputs. The prompt asked for the object at chest height and for the character to look up. Exactly one output satisfied the first, and it is the same output that failed the second. The model coupled two instructions I had written as independent.
Check 2, state across calls
Same character, two new scenes, fresh prompts.
High-level identity held everywhere. Hair, eyes and the main colors survived without effort. The fine-grained state did not. In both new scenes the camera sat behind the character, and the entire left and right assignment flipped: the mustard sleeve and the loops moved to the opposite side of the body. They stayed on the side nearest the camera, which is where they had been in the original.
If you are generating a comic or a character sheet where handedness matters, that is a real constraint, and you will never see it with a symmetrical design.
Check 3, reference input versus full description
Same scene, same seed, two runs. First with the full written description. Second with the description deleted and the earlier image attached as a reference at default strength.
The reference carried hair, eyes and the main colors with zero description, which is a genuine reduction in prompt length. It dropped the color break at the elbow, so both sleeves came back plain. The loops multiplied.
Large flat regions transfer. Small structural information does not. That split repeated in every check where the character moved between generations.
Check 4, edit versus regenerate
One instruction: change the mustard layer to deep teal, keep everything else, with the preserved elements listed by name including the charcoal sleeve.
The edit held the composition almost exactly and then turned both sleeves teal, erasing the element I had named in the preservation clause, and repainted the knee patch. The output also came back at a smaller resolution than the source.
The control, the original prompt with one word changed and generated fresh, kept every property correct across all four runs and produced a completely different composition.
Regenerating behaved as the method predicts. The edit is the interesting half, because it decides whether you can fix an artifact you already like without losing the parts that made it worth keeping.
What this means if you are evaluating a platform
Treat the evaluation like testing an unreliable dependency. Assert on properties, not on vibes. Run every assertion twice. Pin the seed when you want to isolate a variable and release it when you want to sample the distribution. Count things instead of eyeballing them, and pick a fixture where counting is possible.
I ran all of this on Tsubaki.3 on PixAI, which is anime focused and exposes models, LoRAs, reference input and instruction editing in one panel. The platform's SDXL model guide covers why the older model family behaves differently on prompt syntax, which matters if you plan to mix families.
The numbers above are that platform's numbers. The method is portable, and it takes an afternoon.
If you want to run it, start here and design your fixture with one asymmetric property. That single detail will tell you more than any gallery of cherry-picked samples.
Top comments (0)