Building a Face-Consistent Two-Person Video Generator: Architecture, Models, and a Minimal Prototype

Marita P — Wed, 07 Oct 2026 06:10:42 +0000

The "two photos in, one rap performance out" generators are everywhere right now. If you're a developer, the product is a curiosity — but the engineering underneath it is a genuinely interesting problem, and it maps onto a skill you'll reuse for a lot more than memes.

This post is a build guide, not an explainer. I'll walk through the architecture, the model choices at each stage, and a minimal prototype that gets you to "a face stays the same across a moving clip" without a PhD in diffusion.

The Hard Problem, Stated Plainly

Forget the orange stage. The core challenge in any personalized video generator is identity preservation: keep a specific face believable across dozens of frames while the body moves, the mouth animates, and the camera shifts.

A generic face can drift and nobody notices. Your user's face has to stay recognizably them for the whole clip, or the joke dies and the product looks broken. This is the axis everything else hangs on.

The Pipeline, as Something You'd Actually Build

A working prototype is five stages. Each one is a replaceable component, which matters because you'll swap models at every layer.

upload ──► face detect/align ──► identity embed ──► motion+scene ──► audio ──► composite/export

1. Face detection and alignment

You can't feed a raw selfie to a diffusion model. First you localize the face, crop it, and normalize it to a canonical orientation.

Detection: RetinaFace or SCRFD (both available via insightface) are the standard picks — fast on CPU, accurate enough.
Alignment: fit five facial landmarks (eyes, nose, mouth corners) and apply a similarity transform to warp the face to a fixed 112×112 or 256×256 template.

from insightface.app import FaceAnalysis

app = FaceAnalysis(name="buffalo_l", providers=["CPUExecutionProvider"])
app.prepare(ctx_id=0, det_size=(640, 640))

img = load_image("person_a.jpg")
faces = app.get(img)          # each face has .bbox, .kps, .normed_embedding

The normed_embedding on each detected face is the prize — it's the identity vector you'll use downstream.

2. Identity embedding

This is the step most tutorials skip. You need a compact vector that represents the person, independent of pose, lighting, and expression. Face recognition models trained on large datasets (ArcFace-family backbones) give you exactly that — a 512-dim embedding where "same person" = "high cosine similarity."

Keep the embedding for the source face (the user's photo). In generation, you'll pull the generated face back toward this vector.

3. Motion and scene

Here you have two real architectural choices, and they're not equivalent:

Approach	How it works	Trade-off
Face-swap over a template	Generate or reuse a fixed performance clip, then swap in the target face frame-by-frame	Fast, cheap, identity locked by construction; less control over scene
Identity-conditioned video diffusion	Drive a video model with the identity embedding (IP-Adapter-style injection or LoRA)	More flexible scenes; harder to keep identity stable, higher compute

For a two-person rap clip, the first approach is why these tools feel instant: the expensive part (a believable performance with choreography and camera) is solved once, at build time, and reused for every user. You're not generating a scene from scratch per request — you're compositing faces into a pre-rendered one. That's a cost and latency decision, not just a convenience.

If you want bespoke scenes, look at identity-injection via IP-Adapter + AnimateDiff, or a fine-tuned LoRA per recurring character. Budget 10–100× the compute.

4. Audio

Don't synthesize the copyrighted reference track. Generate an original beat plus lyrics and vocals — typically a music/lyrics LLM for the bars, then a TTS or singing-vocals model (Bark, or a commercial text-to-song API) for the performance. Keep audio as a separate pass and mux it at the end; coupling it to the video model only adds failure modes.

5. Composite and export

Stitch frames with ffmpeg, mux the audio, encode for social (vertical, H.264). Nothing exotic here, but get the frame rate and aspect ratio right before you scale.

A Minimal Face-Swap Prototype

If you want something working in a weekend, the face-swap route is the pragmatic choice. The modern standard is insightface's inswapper model:

import cv2, numpy as np
from insightface.app import FaceAnalysis
import onnxruntime as ort

# Load models once
swapper = insightface.model_zoo.get_model("inswapper_128.onnx",
                                          providers=["CPUExecutionProvider"])
source = app.get(load_image("person_a.jpg"))[0]      # identity to inject

video = cv2.VideoCapture("template_performance.mp4")
writer = setup_writer("out.mp4", fps=video.get(cv2.CAP_PROP_FPS))

while True:
    ok, frame = video.read()
    if not ok:
        break
    targets = app.get(frame)
    for t in targets:
        frame = swapper.get(frame, t, source, paste_back=True)
    writer.write(frame)

writer.release()

This gives you a template performance with a stable face in under a hundred lines. Two people? Run the swap for each of two source identities, or use a template that already has two performers in fixed left/right positions and assign each face to its slot.

Where the Real Complexity Lives

The prototype gets you 80% there. The last 20% is where products are won or lost:

Occlusion and motion blur. Hands, microphones, head turns — a raw swap breaks down. The good tools add a face-consistency loss or an identity-regularized diffusion pass on top of the swap.
Multi-face assignment. Keeping face A on the left and face B on the right across 12 seconds, without drift or swap, is a tracking problem as much as a generation problem.
Latency and cost. A reusable template keeps per-request cost near constant. Generating per user scales your GPU bill with your traffic — the difference between a hobby project and a dead one.

Build It or Use It?

If this is your product, build the pipeline — the identity-preservation problem is a genuine moat worth owning.

If you just need the output — say, a shareable two-person clip for a campaign or a quick test of the format — don't re-solve a solved problem. Something like AI Rap Duo already bundles the template, the identity preservation, and the original beat into a two-photo upload with no prompt. That's the right call when the goal is the video, not the architecture.

Either way, the interesting engineering question is the same one that makes any personalized AI product hard: keeping a real, specific face believable while everything around it moves. Nail that, and the rest is just plumbing.

How AI Rap Duo Video Generators Work Under the Hood (and How to Pick One)

Marita P — Mon, 05 Oct 2026 04:26:56 +0000

The "Hotel Lobby AI" trend is everywhere right now: two photos in, a twelve-second rap performance out, with two familiar faces sharing an orange stage and a hanging microphone. If you're a developer or a curious builder, the interesting question is not what these tools do — it's how they do it, and which one actually fits your stack.

This post breaks the pipeline down, then gives you a short framework for choosing between the main approaches.

The Pipeline, Step by Step

Most two-photo rap duo generators run roughly the same sequence under the hood:

Face detection and alignment. The tool finds the faces in your one or two uploads, crops them, and aligns them so the downstream model can work with a consistent input. If you upload two separate portraits, the left/right placement is usually fixed at this stage.
Identity mapping. This is the hard part. The model has to keep your person's identity stable across dozens of frames while the body moves, mouths rap, and the camera shifts. This is typically a face-swap or identity-embedding step layered on top of a motion prior — not a raw text-to-video call.
Motion and scene generation. The orange stage, the performer choreography, and the camera are baked in as a template, which is why these tools need no prompt. You're not describing a scene; you're selecting from a pre-built one.
Audio generation. Rather than reusing the copyrighted "HOTEL LOBBY" recording, many tools synthesize an original beat plus lyrics and vocals. The lyrics are often generated from an optional "topic" field you can fill or leave blank.
Render and export. Everything is composited into a short clip you can download and post.

The practical takeaway: the engineering moat is in identity preservation and audio, not in the scene. That's why the good tools feel instant — the expensive part was solved at build time, not at generation time.

Prompt or No Prompt? The Real Trade-off

The tools split into two camps, and the choice matters more than the orange stage.

Approach	Input	Control	Typical use
Two-photo rap duo generator (e.g., AI Rap Duo)	One or two portraits, optional topic	Low — scene is fixed	Fast, no-prompt clips for social
Prompt-driven video model	Text prompt + reference images	High — every frame is yours	Iterative, bespoke scenes
Editor template	Original clip + manual edits	Medium	Remixing the reference audio

If your goal is a shareable clip in minutes, the two-photo route wins because it removes the fiddly part without removing the one decision that actually matters: who is in the video. A tool like AI Rap Duo takes two portraits or a single duo photo and returns a built-in Hotel Lobby–style performance with an original AI-written verse and beat — no prompt assembly required.

If you need maximum control over lighting, camera, and choreography and are willing to iterate, a prompt-driven model is the fit — but budget for the extra time and the separate audio step.

Why Identity Preservation Is the Hard Part

The single hardest problem in this space is keeping a specific face believable across a moving, lip-syncing body. A generic face can wobble and nobody notices; your face has to stay recognizably you for the joke to land. That's why two-photo generators lean on face-identity embeddings and face-swap priors rather than a raw diffusion pass — and why the audio is usually generated separately from the visuals. It's also why the best tools feel instant: they pre-solved the expensive part at build time, so the per-generation cost stays low.

A Quick Selection Checklist

Want original audio, not the reference track? Look for generators that synthesize lyrics + vocals + beat rather than replaying a copyrighted recording.
Care about identity stability? Prefer tools built specifically for face-to-video, not general text-to-video, since the former optimize for keeping a face consistent across frames.
Need it fast? Two-photo generators are the shortest path from idea to shareable clip.

The trend is fun, but the interesting engineering problem is the same one that makes any AI video product hard: keeping a real, specific face believable while everything around it moves. The tools that nail that are the ones worth watching.

DEV Community: Marita P