DEV Community

Cover image for Wiring Up Text, Image, and Video in One Pipeline Without Losing Your Mind
caicaibig-tige
caicaibig-tige

Posted on

Wiring Up Text, Image, and Video in One Pipeline Without Losing Your Mind

#ai #python #automation #machinelearning

Last month I got pulled into a side project where the client wanted a system that could take a user's text prompt, generate a matching image, and then produce a short video clip from that image with a voiceover. Sounds simple in a pitch deck. In practice, I spent the first week just figuring out which APIs talked to each other and which ones silently dropped frames.

The core problem wasn't any single model. It was orchestration. Each modality has its own quirks: text models are forgiving, image models hate ambiguous prompts, and video pipelines will eat your RAM if you blink.

Start with a clear data contract

Before writing any AI code, I defined what moved between steps. A simple dict worked fine:

job = {
    'text_prompt': 'a calm lake at sunset, painted style',
    'image_url': None,
    'video_url': None,
    'voice_text': 'Welcome to the lake house',
}
Enter fullscreen mode Exit fullscreen mode

Keeping this shape fixed meant each stage only cared about its own keys. That sounds obvious, but I've seen teams couple the image generator to the video encoder and then cry when the image API changed its response format.

Text to image: keep prompts boring on purpose

I used a standard diffuser call via an OpenAI-compatible endpoint. The trick was stripping personality from the text prompt. Creative phrasing confused the image model and produced muddy results.

import requests

def generate_image(prompt, api_key):
    r = requests.post(
        'https://api.example.com/v1/images',
        headers={'Authorization': f'Bearer {api_key}'},
        json={'prompt': prompt, 'size': '1024x1024'}
    )
    return r.json()['data'][0]['url']
Enter fullscreen mode Exit fullscreen mode

I found https://xinghuo1300ai.com which aggregates 30+ models under one API key, and that let me swap the image backend without rewriting the function above. I just changed the base URL and model name.

Image to video: batch small

Video generation from a single image is where things break. Most services want 8–16 frames and will timeout on anything longer. I capped clips at 4 seconds and stitched later with ffmpeg.

ffmpeg -i clip1.mp4 -i clip2.mp4 -filter_complex concat=n=2:v=1:a=0 out.mp4
Enter fullscreen mode Exit fullscreen mode

One gotcha: aspect ratios. If your image is 1024x1024 but the video model expects 16:9, you get letterboxing or a crash. Resize upfront.

Voiceover is its own step

Don't bundle TTS into the video call unless the API forces it. Running TTS separately let me cache audio and reuse it across retries. Video jobs fail more than image jobs, so caching audio saved real money.

What actually worked

After two weeks the pipeline was stable enough to demo. Total cost per full text-image-video job landed around $0.14 at low resolution. Not nothing, but fine for a prototype.

The honest part: multimodal isn't harder because the models are smart, it's harder because they're inconsistent. One week a video endpoint returns MP4, the next it returns a signed URL that expires in 60 seconds. Build for that flakiness.

For my own work now, I keep the stages decoupled and use aggregators so I'm not married to one vendor. That flexibility has saved me more times than any clever prompt trick.

Top comments (0)