A Reproducible QA Harness for AI Image Editing Features

An AI image editor can pass a demo and still fail in production.

The happy-path image looks clean, the progress state feels fast, and the output is impressive at first glance. Then real users upload low-contrast photos, click the action twice, refresh during processing, or ask the model to remove an object that overlaps a hand.

The hard part is not generating one good result. The hard part is making the feature testable when outputs are nondeterministic.

I use a small QA harness built around fixed fixtures, explicit invariants, repeatable run metadata, and human-readable review sheets. It does not pretend that image quality can be reduced to one perfect score. It makes failures easier to reproduce and discuss.

1. Define the operation and its invariants

Start with a task contract.

For an object-removal feature, the operation may be:

Remove the selected object and reconstruct the occluded background.

The invariants describe what should not change:

- Preserve image dimensions.
- Preserve unmasked faces and hands.
- Preserve global color and exposure.
- Do not introduce text, logos, or new objects.
- Return one final image for one accepted job.

These statements are more useful than “the output should look good.” They tell QA what to compare and give engineering a concrete failure category.

2. Build a fixture matrix

One perfect input is not a test suite.

I begin with a small matrix that covers different failure surfaces:

Fixture	Main challenge	Expected risk
clean_bg_01	Object on a flat wall	Visible fill boundary
texture_02	Object over brick or fabric	Repeated texture artifacts
person_03	Object overlaps an arm	Anatomy damage
shadow_04	Object casts a shadow	Shadow remains after removal
edge_05	Object touches frame edge	Smearing or incomplete fill
small_06	Tiny object in a large scene	Selection precision

Keep the original files immutable. Store the mask or selection data beside each fixture so the same region can be tested after a model or UI change.

For a real public interface, I may first inspect an object-removal workflow to understand its user-visible contract: what users upload, how they identify the edit, what progress is shown, and what output they receive. The harness should test the promise the interface makes, not an imagined internal implementation.

3. Log every run as data

Generated files alone are not enough. Record the conditions that produced them.

type ImageEditRun = {
  runId: string;
  fixtureId: string;
  operation: "object_remove" | "image_generate" | "image_edit";
  startedAt: string;
  completedAt?: string;
  inputSha256: string;
  maskSha256?: string;
  prompt?: string;
  modelLabel: string;
  attempt: number;
  status: "queued" | "running" | "succeeded" | "failed" | "timed_out";
  outputPath?: string;
  providerRequestId?: string;
  errorCode?: string;
};

The hash matters because filenames can lie. A file named clean_bg_01.png may have been recompressed or replaced. A checksum tells you whether two runs actually used the same bytes.

Also record the user-facing model label separately from any internal provider identifier. Product copy can change even when the backend does not, and the harness should preserve both layers when they are available.

4. Test the state machine, not just the final image

AI features are asynchronous products. Their UI states deserve independent tests.

A minimal state machine looks like this:

idle -> validating -> queued -> running -> succeeded
                                  |-> failed
                                  |-> timed_out

For each transition, verify:

• The primary action cannot create accidental duplicate jobs.
• A refresh does not lose a job that still exists.
• A terminal failure displays a useful message.
• A retry creates a traceable new attempt.
• Credits or quotas change only at the intended stage.
• The completed output belongs to the current user and job.

This catches a class of bugs that a visual quality review will never see.

5. Use visual metrics as signals, not verdicts

Pixel comparison is useful for deterministic UI screenshots. It is usually too strict for generative output.

Instead, calculate focused signals:

type ReviewSignals = {
  widthMatches: boolean;
  heightMatches: boolean;
  protectedRegionDelta: number;
  editedRegionDelta: number;
  outputFileReadable: boolean;
  containsUnexpectedAlpha: boolean;
};

The protected region is everything outside the intended edit. A large change there is suspicious. The edited region should change, so a near-zero delta can indicate that the requested operation did not occur.

These signals route outputs for review. They do not decide whether reconstructed hair, fabric, or architecture looks believable.

6. Add a compact human review taxonomy

Human review becomes more consistent when reviewers label failure types instead of writing a new paragraph every time.

Useful labels include:

• edit_not_applied
• protected_region_changed
• identity_drift
• anatomy_damage
• texture_repetition
• edge_smear
• lighting_mismatch
• unexpected_object
• unsafe_or_disallowed_output

Allow multiple labels. An output can damage a hand and create a repeated wall texture in the same run.

Add one short note only when the label is not enough. The goal is to create data that can be grouped across releases, not a folder full of unsearchable opinions.

7. Run the same harness against a broader generation surface

An object remover is a constrained edit. A general image-generation interface has a wider output space, but the test structure can remain similar.

You can run the checklist against a general image-generation interface by switching the operation contract and fixtures. Preserve the same run IDs, timestamps, input hashes, retry policy, and artifact layout. Replace object-removal invariants with prompt adherence, reference preservation, aspect ratio, and prohibited-change checks.

The goal is not to rank two different tools with one misleading number. It is to reuse a disciplined testing system across multiple AI image operations.

8. Make retries explicit

Nondeterminism creates pressure to rerun a failed example until it looks good. That hides reliability problems.

Choose a retry policy before testing:

attempt 1 = primary result
attempt 2 = allowed only for transport or provider error
attempt 3+ = exploratory, excluded from release score

If the product intentionally offers creative regeneration, record that as a user action rather than silently replacing the first result.

Your report should show both first-attempt quality and eventual best output. They answer different questions.

9. Produce a release sheet anyone can read

The final artifact should not require opening a test runner.

For each fixture, include:

1. Input thumbnail
2. Mask or requested edit
3. First output
4. Retry output, if allowed
5. State-machine result
6. Automated signals
7. Human failure labels
8. Link to raw run metadata

Then summarize:

fixtures: 6
first-attempt successes: 4
provider failures: 1
quality failures: 1
duplicate jobs: 0
lost-after-refresh: 0
manual review required: 6

Do not publish invented numbers. Generate this summary from the recorded runs. If a check was not performed, mark it not_tested instead of treating it as a pass.

The Bottom Line

Reliable AI image QA is not a screenshot of the best output.

Define the operation. Freeze the fixtures. Record every run. Test the asynchronous state machine. Use visual metrics as routing signals. Label human-review failures. Keep retries explicit. Then produce a release sheet that connects every conclusion to an artifact.

You will still make judgment calls. The difference is that the next engineer, reviewer, or product owner can see exactly what you judged—and reproduce the conditions that led there.