Character consistency isn't a seed trick: a 2-stage image pipeline that actually locks the face

If you're building an app that generates the same character across many scenes, you've probably hit the wall already: seeds drift, LoRA training is heavy and slow, and "same character, new pose" prompting quietly changes the face. The approach that actually holds up in production is a 2-stage pipeline — generate one canonical base image, then edit from that base as the reference for every new scene. Consistency comes from the reference, not the seed.

Below: why the common approaches drift, how the 2-stage pipeline works, the async job queue that makes it deployable, and the serverless-GPU setup that keeps it affordable. There's a free, runnable slice of the whole transport layer at the end.

Why the obvious approaches drift

Seeds. A seed pins the noise, not the identity. Re-use a seed with the same prompt and you get the same image — but that's reproduction, not consistency. The moment you change the prompt ("now she's in a café"), the denoising path changes and the face re-rolls with it. Seeds give you determinism for identical inputs; they give you nothing for new scenes.

Prompt-only ("the same woman as before"). The model has no memory. Every generation is a fresh sample from the distribution your words describe. "Same face as last time" isn't in the prompt vocabulary — there is no last time.

LoRA per character. This one actually works — that's why everyone suggests it — but look at what it costs in an app context: curate 15–40 images per character, run a training job per character, store and load adapter weights per character, and repeat all of it whenever a user creates someone new. For a personal project, fine. For an app where users create characters on demand, you just signed up to run a training farm.

The 2-stage pipeline

The fix is embarrassingly direct once you see it:

Stage 1 — CAST          Stage 2 — RE-SCENE (repeat forever)
text-to-image           image-edit model
"describe character" →  base image + "put them in a café" → scene 1
     = base image       base image + "walking in the rain" → scene 2
                        base image + "reading by a window"  → scene 3

Stage 1 runs a text-to-image model once (I use Z-Image-Turbo) to cast the character — one clean, canonical image. This is the identity. Store it.

Stage 2 runs an image-edit model (Qwen-Image-Edit-2511) for every scene after that. The crucial part: the input isn't a text description of the character — it's the base image itself, plus an instruction describing only what should change (pose, setting, lighting). The model's job is no longer "imagine this person" but "keep this person, change the scene."

Consistency stops being a property you hope the sampler preserves and becomes a property of the conditioning. The reference image is in the input every single time, so the face holds — across twenty scenes, fifty scenes, whatever.

Two details that matter in practice:

• Describe only the change in Stage 2. If your scene prompt re-describes the character ("a woman with long dark hair…"), you're inviting the edit model to re-interpret identity. Prompt the delta: location, pose, lighting. Identity comes from the pixels, not the words.
• This isn't people-only. The reference-edit mechanic doesn't care what the subject is — the same pipeline holds a consistent dog, robot, or anime character. Anything you can cast in Stage 1, you can re-scene in Stage 2.

The part nobody tells you: you can't call a GPU from a web request

Here's where most "it works in my notebook" projects die in deployment. A generation takes 10–40 seconds. An HTTP request does not want to live that long — serverless platforms cap request duration, browsers give up, and users retry and double-charge themselves.

The pattern that survives production:

client → POST /api/generate
             └── create Job row (status=queued) → return job id immediately
worker   picks up job → calls the GPU endpoint → writes image to storage
             └── Job row: status=completed, resultUrl
client → GET /api/status/:id   (poll)
             └── completed? render the image

The request that starts the job returns in milliseconds. The GPU work happens out-of-band. The client polls — a webhook or SSE is the usual alternative, but polling needs zero extra infrastructure (no public URL, identical local and deployed), so that's what I use. A few hard-won rules:

• The job row is the source of truth, not the GPU vendor's job id. Vendors lose webhooks; your DB shouldn't care.
• Make completion idempotent. Two racing polls — or a retry after a timeout — will happily complete the same job twice unless the finalize path claims the row atomically.
• Store results in object storage and serve URLs, not bytes through your app server.

Hosted API or self-host? (honest version)

Let me be straight before the cost section, because "self-hosting is cheaper" as a blanket claim is false and you'd be right to call it out.

For most low-volume projects, a hosted image API is the correct choice. You skip GPU ops entirely, you pay per image, and at a few hundred generations a month the math favors the API — my setup would lose that comparison. If that's you, wrap the API and ship.

Self-hosting wins in three specific situations:

• Volume. A hosted API's per-image price has the vendor's margin baked in — that margin is your cost floor, forever, on every image. Self-hosted, your unit cost is raw GPU-seconds; it doesn't inflate as you grow. Somewhere in the thousands-of-images-a-month range the lines cross, and past it the gap only widens.
• Control. The vendor picks your model, your price, your rate limits, and what content gets filtered — and can change any of them under you. Self-hosted, an upstream deprecation or price hike is someone else's news.
• Idle cost. Serverless GPU that scales to zero means a quiet app costs ~nothing to keep alive, which is what makes a side project survivable.

The trade is real: you take on packaging, cold starts, and ops. The rest of this section is how to make that trade cheap. (A follow-up post works through the actual break-even math with public prices — API per-image rates vs GPU-seconds — so you can find your own crossover point.)

Keeping it cheap: serverless GPU, scale-to-zero, bake the model

Self-hosting the pipeline on serverless GPU (I run RunPod) means you pay GPU-seconds only while a job runs, and workers scale to zero when idle.

The catch is cold start. When a worker wakes up, it has to get the model into VRAM before it can do anything. Three things cut that down hard:

1. Bake the model into the container image. Downloading tens of GB of weights at boot is where cold starts go to die. Weights in the image = pull once, cached on the host.
2. Slim the runtime. Build toolchains and dev dependencies don't belong in an inference image. Every GB you remove is faster pulls and faster boots.
3. One model per endpoint. A fat "does everything" image pays everyone's cold start. Per-endpoint images only load what that endpoint runs.

None of this is exotic — it's just deliberate packaging. The difference between a hobby deployment and one you can put real users on is mostly these three decisions.

Try the transport layer (free, zero infra)

I extracted the whole async transport — queue, worker, polling, serving — into a free slice you can run on a laptop in about two minutes: SQLite instead of Postgres, a stub worker instead of the GPU, and the real RunPod dispatch code included read-only so you can see exactly how the live version differs.

git clone https://github.com/ownstackhq/ai-image-job-queue
npm install && npx prisma db push && npm run dev

Type a prompt, watch the job go queued → processing → completed while the original request has long since returned. The 2-stage engine itself (the Stage 1/Stage 2 logic above, running on real GPU) lives in the full kit at ownstackhq.com, along with the deploy scripts and the cost-tuning chapter.

I'd genuinely like to hear where this approach breaks for your use case — especially if you've pushed reference-edit consistency further than twenty scenes, or found a case where per-character LoRA is still worth the training farm.

---

FAQ

Why not just reuse a seed?
A seed fixes the sampling noise, so the same prompt reproduces the same image. Change the prompt to get a new scene and the whole denoising path changes with it — including the face. Seeds give reproducibility for identical inputs, not identity across different ones.

Why not train a LoRA per character?
LoRA holds identity well, but it's a training job per character: dataset curation, GPU training time, and adapter storage, repeated for every character your users create. Reference-based editing needs one generated image and no training, which is what makes it viable inside an app rather than a workflow.

How do you keep GPU costs down?
Serverless GPU workers that scale to zero, model weights baked into the container image (no boot-time downloads), a slim runtime image, and one model per endpoint. Idle costs approach zero and cold starts drop dramatically.

Does this work for non-human characters?
Yes. The pipeline conditions on a reference image, not on a human-specific representation — the same cast-then-edit loop holds a consistent animal, robot, or anime character across scenes.