Sketching a photo-to-rap-video pipeline

A "photo to rap video" feature looks like a single button, but it is four loosely coupled stages: understanding the photo, writing something worth hearing, performing it, and cutting the result to the beat. Each stage has a different failure mode, and keeping them separate is the difference between a demo and something you can actually debug.

1. Condition on the photo, not just the prompt

The first stage turns an uploaded image into a structured brief: the subject, the setting, an art direction in a few words, and a safety decision. A small vision model is enough:

const brief = await vision.describe({
  image,
  schema: {
    subject: "string",
    setting: "string",
    tone: "string",
    safe: "boolean",
  },
});

Keeping this output structured matters more than its eloquence: the next stage should be able to run without the image at all, which also makes it cacheable and cheap to retry.

2. Write lyrics that fit a fixed window

Rap is dense. A twenty-second clip holds roughly sixty to eighty words, so the topic has to be compressed before generation, not after. Ask for a fixed bar count, then count syllables on the server and regenerate once if the model overshoots:

const bars = await lyrics.write({ brief, bars: 8 });
if (syllables(bars.text) > 90) bars = await lyrics.write({ brief, bars: 8, tighten: true });

One regeneration is usually enough. Looping until the count fits tends to flatten the writing.

3. Separate the vocal from the beat

Rendering vocals and music together usually sounds muddy, and it makes retries expensive. Generate or select the beat as a fixed-length loop, synthesize the vocal line separately, then align both on the bar grid. A bad take then only costs the vocal pass, and you can keep a library of beats that are known to be license-clean.

4. Cut the video to the bars

The last stage is ordinary video work: take the still, add subtle motion so it does not feel frozen, and cut on bar boundaries. Run beat detection on the rendered mix rather than trusting the BPM you asked for; models drift, and the edit is what viewers actually notice.

const beats = detectBeats(renderedMix);
const timeline = beats.filter((_, i) => i % 4 === 0).map((t) => ({ at: t, duration: 0.4 }));

Why the pipeline shape matters

The whole loop is AI Rap Video in practice: choose a scene, give it a topic, and it runs these stages end to end. Keeping them separate is what lets you swap the lyric model without re-rendering video, or re-cut a clip without regenerating the vocals.

If you are building something similar, start with the brief. When the structured description of the photo is wrong, no amount of prompt tuning downstream will fix the result.