DEV Community: Milos Tu

I turn TikTok trends into one-photo AI video templates. Here's what keeps breaking

Milos Tu — Sat, 31 Oct 2026 11:55:43 +0000

Most people don't want to write prompts. They want to drop their own face, or their friends', into the clip everyone else is already posting. That's the bet my site is built on.

So for the last few weeks, most of my work on aiMakeVideo has been one loop: spot a TikTok trend, rebuild its reference clip as a template, and ship a page where you upload a photo and get the trend back with you in it. Two recent ones: a Thriller dance generator (one photo, you lead a zombie crowd for 15 seconds) and an Outstanding Tiny Desk generator (one to five photos, your band in a 19-second office concert).

The pipeline is simple to describe. Almost every step broke at least once. Here's the list.

The pipeline in one paragraph

Tear the reference clip down to a shot table with timecodes. Turn the clip into a depth video with an open-source depth model, so the motion, the camera path and the cuts come from the original while the people don't. Generate one "empty set" plate with every person removed. Then send the model the depth video, the plate, a face crop plus the full photo of each person, and a prompt that walks through the shot table beat by beat. Lay the trend's audio track under the result, charge a flat price, and refund automatically if the render fails.

With that setup the cuts land where the original's cuts are. On the Tiny Desk template they come out within three frames of the reference. The problems are everywhere else.

1. The reference clip leaks into your output

The first paid Thriller render looked great: the face held from the close-up all the way to the wide crowd shot, and the moves matched the track. Then I noticed the lead was wearing a cross necklace. The person in the photo wasn't. The dancer in the reference clip was.

A depth map is supposed to carry shape, not identity, but a necklace is shape. The model saw something on the chest and filled it in with the most plausible thing it knew.

Fix: the prompt now forbids hats, caps, necklaces and chains unless the uploaded photo shows one. I'm still watching the next real renders to confirm it holds. Lesson: list what's on the reference performer's body and say explicitly what not to carry over.

2. Drawing on the depth map makes black mouths

Singing trends need open mouths, and a plain depth map carries no mouth information. So I tried being clever and drew a dark hole on the depth video wherever the original singer's mouth was open.

The model took it literally. About 30% of open mouths came back as flat black patches. A softer, feathered grey hole plus "teeth and tongue, never a black patch" in the prompt brought that down to about 15%. The plain, untouched depth map: 0%.

Fix: stop drawing on the depth map. I wrote a small script that counts open mouths with a mean brightness that's too low and too flat, and ran it on every test render, which is how I got those numbers instead of a gut feeling.

3. Lip sync is the first thing viewers notice

I once put a render up as a page example after checking the cast, the scene, the cuts and the audio offset. Everything except the mouths. They didn't follow the song, and it was spotted within seconds of someone else watching it.

The cause is structural: the video model renders silently, the depth map has no mouth data, and the singing is only described in words. It can't follow lyrics it never heard.

Fix: nothing goes up as an example until I've watched it with the track and compared mouth open/close against the original. I also measure it now: correlating mouth-open frames with the source gives about 0.7 on a good run, where chance is about 0.13. Wide shots are where it slips: on one run the background singers had their mouths open 78% of the time, against 34% in the source.

4. Users upload their dog

My first paying customer uploaded dog photos to a dance template. That template used a pipeline built for replacing people, and it did what it was built for: it swapped the dancer for a random woman. No dog in sight, full price charged.

Fix: a face check in the browser before upload. A lightweight face detector runs twice, on the whole photo and on the upper-middle crop, and its confidence decides the route. Real people scored 0.79 and up; the dogs scored 0.53 and 0.55; a cartoon gnome scored 0.66. A clear cut-off between those sends people to the person pipeline, and pets and characters to a different one. The user sees the choice preselected and can override it.

5. One photo of two people becomes a paper cutout

On a two-person template, I let people upload a single photo of both of them. The model kept the photo's waist-up crop, so the result looked like two people pasted in front of a counter: hard edge, no camera move.

Fix: the browser cuts the photo down the middle and sends each half as its own person, with the normal two-photo prompt. The result had natural bodies, a real push-in, and no ghost third person made from the strips at the cut. Rule since then: one person per reference image, always.

6. Empty seats need a policy, not an error

The Tiny Desk clip has five parts: lead singer, keyboard, saxophone and two backup singers. Very few people have five good photos ready, so only the lead is required. Empty seats are played by AI musicians described in the prompt, and they stay the same people from shot to shot.

One detail from the shot table turned into a product decision: the lead singer and the keyboard player are never on screen at the same time. So if you upload only one photo, you play both parts and appear in every shot. A competitor fills that seat with a stranger instead. I tested the one-photo version and kept it, because people make these videos to see themselves.

7. The boring web bugs

Two non-AI problems took longer to find than they should have.

Autoplay ignores preload. Each template page opened with a muted autoplay preview. On a slow connection it pulled 3–5 MB on first render and pushed my config API call from about 0.5 s to 6.1 s. Now the page shows the poster first and attaches a light, silent preview copy only after the data fetches and the session check settle and the browser is idle. The Tiny Desk preview went from 5.0 MB to 1.9 MB.

Re-cut samples stay invisible. My sample videos are cached for a day at the browser and at the edge. When I re-encoded an example under the same filename, the server had the new file and my own browser kept showing the old one. Now a replaced sample always gets a new name (-v2), plus a test that checks every page points at a real preview file.

What I'd tell myself three weeks ago

Write the shot table before the prompt. Every fix above starts from knowing exactly what happens at each timecode.
Measure the failure you care about (black mouths, lip sync, face match) with a script, even a crude one. "Looks fine" is how the necklace shipped.
Decide what happens with the weird inputs (pets, group photos, missing bandmates) before launch. Users find them on day one.
Refund failed renders automatically. It costs less than the support emails.

The prompts and reference workflows from my earlier rebuild are in a GitHub cookbook, and the full process for that one is in my previous post.

If you've built anything on top of video models: which of these bit you too, and what broke that isn't on this list?

I rebuilt a viral Youtube Short with AI — it got 522K views

Milos Tu — Mon, 19 Oct 2026 10:28:06 +0000

Complex action is no longer the hard part. AI can reproduce it now. What it still can't do for you is direct — and that is the actual skill: shot sizes, camera moves, where the cut lands, and what you deliberately keep out of frame.

I rebuilt a 321-million-view Korean Short to test that. First attempt: 522,000 views. The result is 42 seconds: three 15-second generations cut together.

Most AI video falls apart for a boring reason — it cuts too much. Every second and a half the model hands you a new angle, and the result reads as a slideshow of unrelated frames. The clip I was rebuilding does the opposite: four camera setups inside the first four seconds, then 27 seconds without a cut. That contrast is the whole trick.

Here is the full process, including the three prompt sets that produced the result.

TL;DR — four cuts inside four seconds, then 27 seconds of continuous slow motion. One reference image, reused by all three clips. Three 15-second prompts, every beat timecoded to a camera and a single event. Expect several takes per prompt: the density is why one pass rarely lands.

The two clips

The original — 32 seconds, 321 million views. A Korean Short where a couple keeps a falling egg in the air. The uploader blocks off-site playback, so here is the cover — click through to watch:

▶ Watch the original on YouTube

My rebuild — 42 seconds, 522,000 views. Three 15-second Seedance 2.5 clips, same kitchen, same beat structure, plus a dog:

One case, not a formula: the same prompts will not reproduce it, and they did not reproduce the original's numbers either. What is reproducible is the order of work:

Tear the reference down frame by frame — write a shot list before you write a prompt.
Build exactly one reference image — the whole Short runs on it.
Write timed prompts — three 15-second clips, each beat given a timecode, a camera, an action.
Generate several takes per prompt — the action is too dense for one clean pass.
Cut in the timeline — sound effects are requested in the prompt; music and captions are not.

That order is the point. Everything below is detail.

1. The shot list comes first

Before any prompt, the reference is broken into beats down to the frame. Nearly every instruction later in this post comes from this table.

Timecode	What happens	Camera
0s00f–1s10f	She cuts a snack; an egg rolls toward the table edge	Eye-level medium
1s10f–2s00f	The egg leaves the table	Side view
2s00f–2s26f	Egg falling; her astonished face	Low angle from the floor, looking up
2s26f–4s08f	The egg keeps falling	Top-down
4s08f–8s11f	He throws a whole tray of eggs into the air and dives after the falling one	Wide, tracking the dive
8s11f–11s06f	He lands flat; his right hand catches the egg and bounces it skyward	Low, on the floor
11s06f–12s08f	The egg rising	Locked on the egg
12s08f–13s26f	She reacts — mouth open — and dives after it	Medium
13s26f–16s05f	She moves under the egg and jumps, both arms up	Wide
16s05f–17s19f	Second save: she pushes it back up	Medium
17s19f–19s22f	Two levels: her jumping, him climbing from chair to table	Wide + near
19s22f–20s22f	Her face, close, eyes wide	Close-up
20s22f–22s22f	He hits the egg forward	Medium
22s22f–23s26f	She grabs a frying pan	Medium
23s26f–25s03f	Side view of the same hit	Side
25s03f–26s03f	She slides on her knees, pan held up	Follow
26s03f–28s03f	Top-down on the pan, push in to a close-up	Top-down → macro
28s03f–31s17f	The egg lands in the pan; thumbs-up	Medium

Two things fall out of this table immediately:

The first four seconds are four different setups. That's what makes the egg's fall feel chaotic.
The next twenty-seven seconds never cut away from the action. Slow motion is what keeps a single continuous shot readable when that much happens inside it.

If you take one thing from this post: write the shot list first. A prompt written without a shot list produces exactly the slideshow you were trying to avoid.

2. One reference image, not three

This is where the workflow differs from the usual prompt list: three prompt sets, one image.

Left: the reference image, built once and reused by all three clips. Right: a frame from the output. The composition, the characters and the palette all come from that one still.

The kitchen is built once — 9:16, the woman with purple braids on the left holding a tray, the man at the table on the right drinking water, one egg at the table edge between them — and every clip references that same image.

Two decisions worth copying:

Both characters are in frame from the start. Every clip needs her, him and the egg, so the reference has to establish all three at once. The cost is that each prompt then has to work to keep the wrong character out of individual shots (more on that below).

The iterations went into the egg and the poses, not the room. A reference image is cheap to fix and expensive to re-shoot. Fix it before you spend a single credit on video.

If you build your own, keep the shape: one room, two people, one small object that is going to fly.

3. The three prompt sets

Each set is a 15-second clip. The first line of every set names who is who, because one image carries both characters. Every beat has a timecode.

Set 1 — the egg rolls off the table

She is the purple-braid woman, he is the black-haired man.

[0-2s] Reference.

The black-haired man sits on the chair drinking water; the
woman carries a tray of eggs and is slowly walking toward the
counter top in front of her; the camera focuses on the single
egg on the dining table, which is rolling toward the table
edge and has just dropped off the table.

[3-5s] The instant the egg rolls off the table and drops, the
picture enters slow motion: a low-angle close-up from the
floor shows the egg falling through mid-air in slow motion.
While the low-angle close-up holds on the egg, the
black-haired man must watch the egg's slow-motion fall in
extreme panic.

[6-9s] The purple-braid woman looks toward the front of the
camera in extreme anxiety and panic; she throws the plate in
her hands and the eggs on it hard into the air behind her. The
picture enters slow motion; the eggs scatter in mid-air behind
the woman; the purple-braid woman dives forward toward the
front of the camera in slow motion, anxious and terrified.

[10-12s] Slow motion, shot from a ground-level point of view.
A slow-motion shot of the purple-braid woman: she lies on the
floor and slides toward the camera; in the background some
out-of-focus eggs are falling in slow motion and smashing on
the floor behind her. She looks up in utter terror at one egg
in the upper foreground that is falling toward the ground, and
she reaches one hand out along the floor, palm up, trying to
catch the egg falling from the air.

[13-15s] Slow motion. The egg drops into the woman's hand; she
pushes upward hard but does not hold it — the egg is pushed
high and flies up out of frame above the camera. The woman
must keep watching the egg in utter terror, and only this one
egg may appear in the picture.

Rich sound effects, no background music, no subtitles.

Set 2 — the man's run and the first volley

This is where the negative constraints do the heaviest work: three beats in a row end with the woman must not appear in the picture, because the single reference image contains her.

She is the purple-braid woman, he is the black-haired man.

[0-1s] Reference.

Close-up of the man's chest and head. In extreme panic his
gaze moves slowly in slow motion from the upper right to the
upper left, and as he looks he slowly spits the water in his
mouth out in slow motion. The woman must not appear in the
picture.

[4-7s] Slow-motion shot of the man's upper body. Side-on to
the camera, he takes two quick steps toward the edge of the
frame, turning his head as he runs to look up above the camera
in extreme panic, then he jumps hard in slow motion with both
hands raised in a lifting motion. The woman must not appear in
the picture.

[8-11s] Slow motion, close-up of the man as he jumps in
mid-air in slow motion with both hands raised in a lifting
motion. An egg is flying from the air toward the man's hands;
he pushes up hard but does not catch the egg and sends it high
up, out of frame above the picture. The woman must not appear
in the picture.

[12-15s] Slow motion. In the background the man makes the
lifting motion, watching an egg above him in anxious panic as
it slowly flies up into the air. Side shot of the woman: in
the foreground she stands on the chair, looking toward the egg
in the sky with her back to the camera, knees bent and loading
to jump. There is no egg on the table at this moment.

Rich sound effects, no background music, no subtitles.

Set 3 — the spike, the bowl and the face

The busiest set. She spikes the egg downward like a volleyball, he slides in on his knees with a bowl, and the clip ends on a gag instead of a clean catch.

She is the purple-braid woman, he is the black-haired man.

[0-2s] Reference [].

Slow motion, low-angle shot. In extreme anxiety and panic the
woman stands on the dining chair that the man sits on in the
reference image, looking up above the camera; she loads and
jumps high, reaching out one hand in the pose of a volleyball
spike. The black-haired man must not be in this shot.

[3-5s] Slow motion, shot of the black-haired man. In the
corner of the reference image, in utter anxious panic, he
looks up above the camera and runs toward the camera, pulling
a big bowl out from behind him at the same time, running in
slow motion toward the camera and out at the edge of the
picture, anxious.

[6-8s] Slow motion, side shot of the woman in mid-air in a
spike pose, her face full of anxious panic. An egg is flying
slowly from above her shoulder toward the front of her hand;
with one slow-motion spike she hits the egg and sends it
flying down and out of the picture at a diagonal. The egg must
not be broken by her hit.

[9-10s] Slow motion, high-angle follow shot of the
black-haired man sliding toward the camera on his knees, his
weight leaning back, holding a big bowl in both hands with the
bowl facing up in front of him, ready to catch something.

[11-12s] Over-the-shoulder view of the man, slow motion. His
weight leans back as he slides toward the woman on his knees,
holding the big bowl in both hands in front of his chest ready
to catch the egg; the woman is falling from mid-air in utter
terror watching the egg; an egg is flying slowly through the
air toward the man.

[13-15s] Slow motion, slightly high-angle follow shot of the
man sliding on the floor toward the camera, his weight leaning
back, his expression going slowly from anxious panic to happy,
looking up above the camera, holding the big bowl in both
hands in front of his chest as if he wants to catch something.
The slow motion ends: a raw egg smashes hard onto the
black-haired man's face, the egg breaks and the yolk and white
cover his face; the man freezes for a moment, then a close-up
of his face as he breaks down and screams.

Rich sound effects, no background music, no subtitles.

4. What I learned about prompting complex action

These six rules are the transferable part. They apply to any multi-shot AI video, not just this kitchen:

1. One reference image for all three clips. Every set opens by naming who is who, and reuses the same image instead of rebuilding one per shot.

2. Every beat names its camera. Low angle, side, top-down, over-the-shoulder, follow — plus shot size and movement. The prompts read like a shot list, not a mood description.

3. Slow motion is stated in almost every beat. With this much action in 15 seconds, slow motion is what makes it legible. It is the one stylistic instruction that never gets left implicit.

4. Negative constraints replace missing references. Because one image holds both characters, shots that need only one of them say so explicitly: the woman must not appear in the picture, only this one egg may appear in the picture. If you've ever written JSON Schema or regex, this is the same idea: constrain what must not occur, don't hope the model infers it.

5. Beats are 2–4 seconds and exactly one event. Camera, action, expression, and what the object does. Nothing else.

6. The same closing line every time. Rich sound effects, no background music, no subtitles — leaving the edit to the edit.

And the unwritten rule that matters most: plan for several takes. The action is too dense for one clean pass. Generate multiple, keep the best beat from each, cut them together. Expecting a single prompt to land is the most common way this fails.

5. Things that will trip you up

The source was a working note, not a tutorial, and a few details don't line up:

The prompts don't end like the reference. The teardown ends with the egg landing in a frying pan and a thumbs-up; set 3 ends with a raw egg on his face and a scream. It's a comedy remix, not a frame-accurate copy — decide your ending and write it into the last beat.
The pan becomes a bowl. Use one word in both places or the prop will drift between clips.
Set 3 opens with an empty reference slot (Reference []). All three clips start from the same kitchen image — fill it.
Set 1 fights itself. [6-9s] throws a whole tray of eggs into the air, while [13-15s] says only one egg may appear. Keep the scattered eggs out of focus in the background and apply the rule to the foreground.
Timecodes restart in every set. [0-2s] in set 1 and in set 3 are different clips. Generate each as its own 15-second clip.

6. Adding the dog

The version that took off added a dog to make the scene more complex. My working document records the change and the view count but not the rewritten prompts, so that pass was written by hand.

The rule that made it work is the same one the two characters already follow: put the dog into the reference image first, then name it in every beat and give it one job in the rally — going for the egg. A dog that only sits in the background does not survive a 15-second shot.

Closing

The three prompt sets were generated in aiMakeVideo with Seedance 2.5, image-to-video, 9:16, 15 seconds per clip. The shot list is my own teardown of the reference Short.

If you want the parameter side of this — what each clip costs in credits, how 480p and 720p compare — that's collected in this open-source companion repo.

The transferable part isn't the egg. It's the order: shot list → one reference image → timed beats → multiple takes → cut in the timeline.

Credits

The reference Short — 계란을 구하고 싶어요 by 김프로KIMPRO. The shot list above is a teardown of their video.
The version with the dog, and the working document the prompts come from — mine.
Prompts were generated in aiMakeVideo with Seedance 2.5.

Originally published at aimakevideo.com.