When we tell people we built a generator that turns a sentence — or a photo — into a 12-second rap performance, the follow-up question is always the same: "okay, but what is actually happening in there?"
This post is the honest walkthrough of the pipeline behind AI Rap Video (https://useairapvideo.com), step by step, with the failure modes we hit while building it.
Step 1 — Lyrics from a topic
Everything starts with text. The user gives a topic ("my coworker who microwaves fish"), optionally their own lyrics, and picks a stage template. A lyric model turns the topic into structured bars: rhyme pairs, a hook, and a length target that fits the video duration. The template matters here — a diss track needs punchlines, a birthday rap needs the recipient's name in the first two bars, so the prompt carries template constraints, not just the topic.
Step 2 — A rap voice performs them
The lyrics go to a rap-styled vocal model. This is not text-to-speech with a filter: delivery — where the flow sits relative to the beat, which syllables get stressed — is part of the generation. The same bars performed straight sound like a recital; performed with push-pull against the beat, they sound like a verse.
Step 3 — An original beat
The beat is composed per video and matched to the vocal take's tempo. We made one deliberate product decision here: no copyrighted audio, ever. No famous instrumentals, no "sounds like" beats. The audio is original by default, users can upload beats they own the rights to, and there is a DMCA takedown flow. It is more work than bundling known beats, but it is the only version of this product we were willing to ship.
Step 4 — Lip sync from a photo
If the user supplied a photo, a lip-sync model animates the face to the vocal: it extracts mouth-region landmarks, generates a mouth shape sequence aligned to phonemes, and composites it back. Front-facing, well-lit photos sync dramatically better than angled ones — the model needs to see the whole mouth. Two photos? The duo template maps each face to its own vocal part.
Step 5 — Render
The vocal, beat, and animated video are composited and rendered as a 12-second MP4 (480p or 720p). Each render consumes credits from a one-time pack — there is no subscription, which users mention unprompted more than we expected.
Watch it end to end
There are four full sample videos on the homepage at https://useairapvideo.com — diss track, birthday, Santa, and a two-person duo — if you want to see all five steps compressed into a dozen seconds each. The rap generator itself runs free 6-second previews, one per account, if you would rather poke at it directly.
Questions about any of the steps are welcome in the comments — especially on the lip-sync stage, which is where most of our iteration time went.
— AI Rap Video team
Top comments (0)