If you've built anything on top of a text-to-video model, you've hit this: the first clip is great, the second has a slightly different nose, and by the fifth it's a different person. A prompt like "a woman with a copper-red bob and an olive jacket" fits millions of people, and the model picks a new one on every request.
I run RefVid, a small tool built around one idea: you save a character or a product once, as a few photos, and every clip after that is generated from exactly those photos. This post covers what that takes in code. We use Kling's reference-to-video endpoints through fal's queue API, and everything below comes from our production code and our own test clips.
Words alone drift; photos hold
We started by measuring the problem. We took one invented character, Mia, and generated the same three scenes twice on Kling 3.0 Omni, 5 seconds each:
- Prompt only, with Mia described in as much detail as we could fit (hair, freckles, earrings, jacket).
- The same scene prompts with no appearance words at all, plus four reference photos of Mia attached.
Prompt only. A copper-red bob and an olive jacket, but not quite the same woman as in the other scenes.
Four reference photos, no appearance words in the prompt. The same face in every scene.
The prompt-only set cost $1.26 for three clips, and each one gives you a different person who matches the description. The reference set keeps her. The full side-by-side is on our consistent character guide.
What we took from it: once you attach references, stop describing the person in the prompt. When the words and the photos disagree, the model blends them, and the blend is exactly the drift you were trying to fix.
The request shape: elements, not just images
Kling's reference-to-video takes subjects as elements. An element is one frontal image plus extra views of the same subject. The prompt points at it as @Element1, @Element2, and so on. Here is the core of how we turn a saved subject into an element (simplified from our provider):
const MAX_ELEMENT_IMAGES = 4; // one frontal + up to three extra views per element
function toElement(imageUrls: string[]) {
const [frontal, ...rest] = imageUrls;
if (!frontal) throw new Error('at least one reference image is required');
const views = rest.slice(0, MAX_ELEMENT_IMAGES - 1);
return {
// reference_image_urls needs at least one image, so a one-photo subject repeats its frontal view
element: { frontal_image_url: frontal, reference_image_urls: views.length ? views : [frontal] },
// photos beyond four go in as general references instead of being dropped
extra: rest.slice(MAX_ELEMENT_IMAGES - 1),
};
}
Three things we learned the hard way:
- Order matters. The first image is treated as the frontal view, so make it your sharpest, most evenly lit, straight-on shot. A three-quarter view in that slot gives you a slightly turned face in every clip.
- One photo still works, but it has to be sent twice. The endpoint wants at least one extra view, so a single-image subject sends its frontal again.
-
Overflow goes to
image_urls. If someone uploads five photos, the fifth still helps as a plain reference instead of being silently ignored.
The prompt must name every element
If the prompt never mentions @Element1, the model is free to ignore the photos. People don't write @Element1, though. They write "Mia pours coffee". So before a prompt goes out, we swap each subject's name for its token:
// "Mia holds NOIR 07 and says \"Hi, I'm Mia\""
// → "@Element1 holds @Element2 and says \"Hi, I'm Mia\""
bindSubjects(prompt, ['Mia', 'NOIR 07']);
The details are where it breaks:
-
Whole words only. "Mia" must not match inside "Miami". We use Unicode-aware boundaries,
(?<![\p{L}\p{N}])before and(?![\p{L}\p{N}])after. They're skipped for Chinese, Japanese and Thai, where words have no spaces around them. -
Not inside quoted dialogue. Kling 4 clips have sound, and if the swap reaches the spoken line, the voice literally says "at Element one". Text inside
"…",“…”,„…“,«…»and「…」is left alone. - Unnamed subjects get prefixed. If a subject is attached but never named, its token is put at the start of the prompt so the model still uses it.
-
Respect authors who already use tokens. If a prompt already contains
@Element1, we leave it untouched.
Long sequences: split, then chain
Faces hold up best in short clips. Drift creeps in as the camera moves and the face turns to angles the references don't cover. So instead of one long generation, we do two things.
Multi-shot requests. Kling accepts a multi_prompt list, one prompt per shot, and splits the duration between them. On Kling 4, each shot's prompt is capped at 512 characters. Names and translations can push a shot over that cap, so in that case we send one prompt that lists the shots in order rather than let the request fail:
const shotsFit = shots.every((s) => s.prompt.length <= 512);
const body = shotsFit
? { multi_prompt: shots, elements }
: { prompt: shots.map((s, i) => `Shot ${i + 1} (${s.duration} s): ${s.prompt}`).join('\n'), elements };
Chained clips. For a series, we start each clip from the last frame of the previous one (start_image_url) and attach the same element every time. We made a four-shot running-shoe sequence this way: four 5-second clips at $0.42 each, plus $0.03 to pull the last frames out.
Shot 1. Each of the next shots starts from the previous shot's last frame, with the same product element attached.
Shot 4. Same shoe, same path, same light direction.
The start frame carries the scene and the light. The element carries the identity. You need both: with only the start frame, the identity slips once the camera moves away from what that frame showed.
Products are harder than faces
People forgive a slightly different jaw. Nobody forgives a misspelled logo. For products:
- Shoot three photos on a plain backdrop: the hero angle, a three-quarter view, and a close-up where the label fills the frame.
- Put the label in the prompt as a fact about the scene, not as a description: "the LUMEN label faces the camera and stays readable".
- Keep the label large in frame in at least one shot. Small text in a wide shot has too few pixels to come out right.
LUMEN is a serum we invented for testing. Three reference photos, Kling 3.0 Omni; the label carries over into each scene.
Plumbing notes
-
Use the queue and a webhook, not a blocking call. A 3-second 720p Kling 4 clip with sound took about 80 seconds to generate for us. We submit to
queue.fal.run/<model>?fal_webhook=<url>, verify the webhook signature, and keep a status poll as a fallback. - Treat a 5xx on the result fetch as retryable. A failed fetch of the result doesn't mean the job failed. We only mark a generation failed when fal reports an error for the job itself.
- Make the webhook idempotent. Record the request id before you act on it. Webhooks get delivered twice, and you don't want to refund or charge twice.
What I'd tell anyone building this
- Version the reference set. Most "the face changed" bug reports come down to clip 1 and clip 4 having been made from different photos.
- Strip appearance words from prompts once references are attached.
- Keep clips short and cut between them. Chaining 5-second clips beats one 15-second clip for identity.
- Test with an invented character, so you know what "the same person" means.
If you want to try it without building any of this, RefVid does all of the above behind one upload. Ask in the comments about the request shapes or the regexes. I'm happy to go into more detail.
Top comments (0)