DEV Community

theheaton hsu
theheaton hsu

Posted on Fully Autonomous

How I built an AI rap video generator as a solo maker

A few weeks ago my feed was full of the "Hotel Lobby" rap edits: two people in an orange booth, one hanging mic, rapping a famous song with swapped-in celebrity faces. I wanted a version where you use your own photo and get an original rap instead. So I built one. Here's how the pipeline works and what I learned.

What it does

You upload one photo (or two for a duo), pick a scene and a beat, and type a topic like "my golden retriever who steals socks". A few minutes later you get a 5–15 second MP4 where the face raps original lyrics in sync with a new beat.

The pipeline

The job is split into small steps so each one can fail and retry on its own:

  1. Moderation first. The topic and any custom lyrics are checked before anything expensive runs. Public figures and people who haven't given permission are not allowed.
  2. Lyrics. An LLM (Anthropic's Claude) writes bars from the topic. The prompt fixes the number of lines to the video length, so a 5-second clip doesn't get 16 bars. Users can also paste their own lines, one per bar.
  3. Beat, vocals and video. The lyrics, the beat style (Trap, Boom Bap, Drill, Old School) and the scene go to ByteDance's Seedance 2.0 family, which generates the music, the voice and the lip-synced video in one pass.
  4. Duo mode. Two photos become two performers in one frame, trading lines. Getting the model to keep both faces and not invent extra people took more prompt work than anything else.
  5. Delivery. The finished file goes to object storage and the user downloads an MP4 in 9:16, 1:1 or 16:9.

Renders take 2–4 minutes, so the frontend polls a task record instead of holding a request open.

Credits and failures

Video generation costs real money per second, so I use credits instead of a flat subscription. Credits are deducted when a task starts and refunded automatically if the render fails, so nobody has to email me to get credits back for a broken video.

Stack

  • TanStack Start (React 19, Vite) on Cloudflare Workers
  • Drizzle ORM, better-auth
  • A small task table that tracks each AI job, its cost and its status

What I'd do differently

  • Ship a free tool early. A free AI rap lyrics generator (no account needed) is an easy way for people to try the idea before they pay for a video.
  • Test with pets early. People love making their dog rap. Faces that aren't human break assumptions in prompts, so test them from day one.
  • Keep scenes few and good. Seven scenes with tuned prompts beat thirty generic ones.

If you want to try it, it's at rapvideo-ai.io. New accounts get free credits for a first short video. I'd love feedback on the pipeline, especially from anyone who has fought with lip-sync on non-human faces.

Top comments (0)