A Reproducible Benchmark for Visual Consistency in AI Characters

AI character systems often look convincing in a single image and inconsistent across a sequence. Hair length changes, clothing details disappear, apparent age drifts, or a video no longer resembles the portrait that introduced the character. The usual response is to keep rewriting prompts. That helps occasionally, but it does not explain which constraint failed or whether a model update improved the system.

This article describes a small, reproducible benchmark for treating character appearance as a testable contract. It is based on evaluation patterns used by the Ponys.ai team, generalized so the method can be reused with any image or video pipeline.

1. Separate identity from scene

A prompt becomes hard to debug when identity, camera, environment, action, and rendering style are mixed into one paragraph. Store them as separate fields:

character_id: aurora-017
identity:
  apparent_age: adult
  face_shape: oval
  eye_color: green
  hair: long black hair with straight fringe
  stable_marks:
    - small mole below left eye
wardrobe:
  primary: navy field jacket
  required_details:
    - silver zipper
    - two chest pockets
scene:
  location: railway platform
  time: blue hour
camera:
  framing: medium shot
  lens_equivalent_mm: 50
style:
  rendering: cinematic realism
  color_rule: cool ambient light, neutral skin
negative_constraints:
  - no logo
  - no school uniform
  - no childlike proportions
~~~

The `identity` block should change rarely. Scene and camera fields are allowed to vary. This separation makes it possible to determine whether a failure belongs to identity preservation, prompt composition, or rendering.

A character can be defined in a structured [character creation flow](https://ponys.ai/create), while published examples can be sampled from a [character discovery surface](https://ponys.ai/discover) for the same evaluation set.

## 2. Build a minimum test matrix

One attractive result is not evidence of consistency. For every character version, render at least the following matrix:

| Case | Variable changed | Variables held constant | Purpose |
|---|---|---|---|
| A1 | seed | identity, scene, camera | measure seed sensitivity |
| A2 | camera angle | identity, wardrobe, scene | detect face drift |
| A3 | expression | identity, camera | detect age and feature drift |
| A4 | lighting | identity, wardrobe | detect color instability |
| A5 | environment | identity, camera | detect prompt competition |
| A6 | image to video | all identity fields | detect temporal identity loss |

Use at least three seeds per still-image case. The video case should include the first frame, a midpoint frame, and the last frame. That gives enough observations to distinguish a one-off generation failure from a systematic issue.

## 3. Score attributes, not impressions

A reviewer saying “it feels different” is useful feedback but poor regression data. Split the score into observable attributes.

### Identity score (45%)

- face geometry: 15%
- hair shape and color: 10%
- eye color: 5%
- stable marks: 10%
- apparent age band: 5%

### Wardrobe score (20%)

- garment category: 8%
- dominant color: 5%
- required details: 7%

### Scene compliance (15%)

- location: 5%
- time and lighting: 5%
- camera framing: 5%

### Safety and negative constraints (20%)

- prohibited logos or text: 5%
- age representation: 10%
- excluded clothing or body traits: 5%

Calculate the weighted score, but keep hard failures separate. An age-representation or safety violation must fail the release even if every visual detail is otherwise correct.

~~~text
visual_score = 0.45 * identity
             + 0.20 * wardrobe
             + 0.15 * scene
             + 0.20 * constraints

release = visual_score >= 0.90
          AND identity >= 0.92
          AND hard_failures == 0
~~~

## 4. Keep a manifest for every render

Store enough information to reproduce an output later:

~~~json
{
  "run_id": "aurora-017-v4-A3-s42",
  "character_spec_version": "4.2.0",
  "model_version": "image-model-2026-07",
  "prompt_template_version": "visual-contract-3",
  "seed": 42,
  "input_image_sha256": null,
  "output_sha256": "...",
  "reviewers": ["r01", "r02"],
  "scores": {
    "identity": 0.94,
    "wardrobe": 0.88,
    "scene": 0.97,
    "constraints": 1.0
  },
  "hard_failures": []
}
~~~

Without model, template, seed, and character-spec versions, a failed result cannot be reproduced and a successful one cannot be trusted as a baseline.

## 5. Diagnose by failure family

### Face drift across camera angles

Check whether the identity description depends on vague adjectives such as “beautiful” or “distinctive.” Replace them with observable geometry and stable marks. Compare the same seed before changing the entire prompt.

### Clothing details disappear

Required wardrobe details may be losing attention to a long scene description. Move mandatory details into a compact constraint block and test against a neutral background first.

### Color changes under different lighting

Separate object color from illumination. “Navy jacket under warm sunset light” is more testable than “warm orange jacket scene.” Review both the rendered pixel color and the semantic garment label.

### Still image passes, video fails

Video introduces temporal drift. Generate a baseline with minimal motion, then increase camera and character motion separately. Compare the first, middle, and final frames rather than judging only the thumbnail. A dedicated [AI video generation flow](https://ponys.ai/ai-video-generator) can be tested with the same manifest used for the still-image baseline.

### Every seed fails differently

This usually indicates an underspecified identity contract, not a bad seed. Return to the stable identity fields and remove scene complexity. A controlled [AI image generation run](https://ponys.ai/ai-image-generator) should pass before the character is tested in motion.

## 6. Add release gates

A practical release policy can be small:

1. All hard constraints pass in every case.
2. Median identity score is at least 0.92.
3. No individual case has identity below 0.85.
4. The score difference across three seeds is below 0.10.
5. First, midpoint, and final video frames retain all stable marks.
6. Every failure stores a run ID and failure family.

Run this suite whenever the model, prompt template, image conditioning, video pipeline, or character definition changes. Do not replace the baseline after a single better-looking output; require the full matrix to improve or remain within tolerance.

## 7. Review with paired comparisons

Absolute scores drift between reviewers. Pair the old and new output, hide which version is newer, randomize left and right placement, and ask one focused question at a time:

- Which result preserves identity better?
- Which result follows wardrobe constraints better?
- Which result preserves identity from still image to video?

Two reviewers should independently score the hard attributes. Resolve disagreements by pointing to a named attribute, not by averaging vague preference.

## Final checklist

Before calling an AI character visually stable, confirm that:

- identity and scene are stored separately;
- every output has a reproducible manifest;
- multiple seeds and camera angles are tested;
- scores are split by identity, wardrobe, scene, and constraints;
- safety-critical constraints cannot be averaged away;
- video is reviewed at multiple timestamps;
- failures are assigned to a concrete family;
- release gates run again after every material pipeline change.

The useful shift is simple: stop treating consistency as a prompt-writing talent and start treating it as a versioned test contract. That turns visual drift from an argument about taste into an engineering problem that can be reproduced, diagnosed, and improved.