Most people don't want to write prompts. They want to drop their own face, or their friends', into the clip everyone else is already posting. That's the bet my site is built on.
So for the last few weeks, most of my work on aiMakeVideo has been one loop: spot a TikTok trend, rebuild its reference clip as a template, and ship a page where you upload a photo and get the trend back with you in it. Two recent ones: a Thriller dance generator (one photo, you lead a zombie crowd for 15 seconds) and an Outstanding Tiny Desk generator (one to five photos, your band in a 19-second office concert).
The pipeline is simple to describe. Almost every step broke at least once. Here's the list.
The pipeline in one paragraph
Tear the reference clip down to a shot table with timecodes. Turn the clip into a depth video with an open-source depth model, so the motion, the camera path and the cuts come from the original while the people don't. Generate one "empty set" plate with every person removed. Then send the model the depth video, the plate, a face crop plus the full photo of each person, and a prompt that walks through the shot table beat by beat. Lay the trend's audio track under the result, charge a flat price, and refund automatically if the render fails.
With that setup the cuts land where the original's cuts are. On the Tiny Desk template they come out within three frames of the reference. The problems are everywhere else.
1. The reference clip leaks into your output
The first paid Thriller render looked great: the face held from the close-up all the way to the wide crowd shot, and the moves matched the track. Then I noticed the lead was wearing a cross necklace. The person in the photo wasn't. The dancer in the reference clip was.
A depth map is supposed to carry shape, not identity, but a necklace is shape. The model saw something on the chest and filled it in with the most plausible thing it knew.
Fix: the prompt now forbids hats, caps, necklaces and chains unless the uploaded photo shows one. I'm still watching the next real renders to confirm it holds. Lesson: list what's on the reference performer's body and say explicitly what not to carry over.
2. Drawing on the depth map makes black mouths
Singing trends need open mouths, and a plain depth map carries no mouth information. So I tried being clever and drew a dark hole on the depth video wherever the original singer's mouth was open.
The model took it literally. About 30% of open mouths came back as flat black patches. A softer, feathered grey hole plus "teeth and tongue, never a black patch" in the prompt brought that down to about 15%. The plain, untouched depth map: 0%.
Fix: stop drawing on the depth map. I wrote a small script that counts open mouths with a mean brightness that's too low and too flat, and ran it on every test render, which is how I got those numbers instead of a gut feeling.
3. Lip sync is the first thing viewers notice
I once put a render up as a page example after checking the cast, the scene, the cuts and the audio offset. Everything except the mouths. They didn't follow the song, and it was spotted within seconds of someone else watching it.
The cause is structural: the video model renders silently, the depth map has no mouth data, and the singing is only described in words. It can't follow lyrics it never heard.
Fix: nothing goes up as an example until I've watched it with the track and compared mouth open/close against the original. I also measure it now: correlating mouth-open frames with the source gives about 0.7 on a good run, where chance is about 0.13. Wide shots are where it slips: on one run the background singers had their mouths open 78% of the time, against 34% in the source.
4. Users upload their dog
My first paying customer uploaded dog photos to a dance template. That template used a pipeline built for replacing people, and it did what it was built for: it swapped the dancer for a random woman. No dog in sight, full price charged.
Fix: a face check in the browser before upload. A lightweight face detector runs twice, on the whole photo and on the upper-middle crop, and its confidence decides the route. Real people scored 0.79 and up; the dogs scored 0.53 and 0.55; a cartoon gnome scored 0.66. A clear cut-off between those sends people to the person pipeline, and pets and characters to a different one. The user sees the choice preselected and can override it.
5. One photo of two people becomes a paper cutout
On a two-person template, I let people upload a single photo of both of them. The model kept the photo's waist-up crop, so the result looked like two people pasted in front of a counter: hard edge, no camera move.
Fix: the browser cuts the photo down the middle and sends each half as its own person, with the normal two-photo prompt. The result had natural bodies, a real push-in, and no ghost third person made from the strips at the cut. Rule since then: one person per reference image, always.
6. Empty seats need a policy, not an error
The Tiny Desk clip has five parts: lead singer, keyboard, saxophone and two backup singers. Very few people have five good photos ready, so only the lead is required. Empty seats are played by AI musicians described in the prompt, and they stay the same people from shot to shot.
One detail from the shot table turned into a product decision: the lead singer and the keyboard player are never on screen at the same time. So if you upload only one photo, you play both parts and appear in every shot. A competitor fills that seat with a stranger instead. I tested the one-photo version and kept it, because people make these videos to see themselves.
7. The boring web bugs
Two non-AI problems took longer to find than they should have.
Autoplay ignores preload. Each template page opened with a muted autoplay preview. On a slow connection it pulled 3–5 MB on first render and pushed my config API call from about 0.5 s to 6.1 s. Now the page shows the poster first and attaches a light, silent preview copy only after the data fetches and the session check settle and the browser is idle. The Tiny Desk preview went from 5.0 MB to 1.9 MB.
Re-cut samples stay invisible. My sample videos are cached for a day at the browser and at the edge. When I re-encoded an example under the same filename, the server had the new file and my own browser kept showing the old one. Now a replaced sample always gets a new name (-v2), plus a test that checks every page points at a real preview file.
What I'd tell myself three weeks ago
- Write the shot table before the prompt. Every fix above starts from knowing exactly what happens at each timecode.
- Measure the failure you care about (black mouths, lip sync, face match) with a script, even a crude one. "Looks fine" is how the necklace shipped.
- Decide what happens with the weird inputs (pets, group photos, missing bandmates) before launch. Users find them on day one.
- Refund failed renders automatically. It costs less than the support emails.
The prompts and reference workflows from my earlier rebuild are in a GitHub cookbook, and the full process for that one is in my previous post.
If you've built anything on top of video models: which of these bit you too, and what broke that isn't on this list?
Top comments (0)