I spent a weekend wiring Google's Gemini and Veo APIs into a single app just to feel where the edges of multimodal AI actually are. It turned into a small studio I now use daily, and along the way I learned more about these models from plumbing them than from any paper. Here's the honest technical debrief.
Three pipelines, three completely different problems
I wanted one prompt box that could do video, image editing, and document Q&A. Naively I assumed they'd share most of the stack. They don't.
1. Image-to-video: the enemy is time, not pixels
Generating one good frame is solved. Video is about temporal coherence — frame 13 must agree with frame 12 or you get flicker and identity drift. Modern video models treat the clip as one object in space and time (latent diffusion over a width x height x time volume, with spatiotemporal attention) rather than 120 independent images. Conditioning on a reference image as the first frame is what makes image-to-video feel controlled: you've handed the model a strong anchor and asked it to extrapolate motion, not invent a world.
The surprise: native audio sync (Veo 3.1 generating clip + soundtrack jointly) does more for perceived realism than another notch of resolution. A door slam landing on the exact frame the door shuts is uncanny in a good way.
2. Instruction-based image editing: preservation is the hard part
Generating is unconstrained; editing must change one thing and preserve everything else. Condition the diffusion model on both the instruction and the source image's latents, cross-attend the instruction to steer only the referenced region, and bias hard toward preserving unedited latents. Push that preservation too soft and the subject's face quietly morphs across edits — the classic 'character consistency' failure that makes or breaks storytelling use-cases.
3. PDF chat: it's retrieval, not a long context
The naive 'paste the whole PDF' approach dies on long files (models get lost in the middle) and costs you the full document every turn. The version that works is a tiny RAG pipeline: chunk with overlap that respects structure, embed chunks into a vector index, retrieve the few nearest passages per question, and ground the answer in only those passages with a citation. Half the real work is just parsing hostile PDFs (multi-column, scanned, tables) into clean ordered text before any model sees it.
What was genuinely hard solo
- Cost control. Every modality has a different price curve. I collapsed everything to one credit balance and route to the cheapest model that clears a quality bar per task. Hard-coding model names at call sites is a trap; put them behind one config.
- Latency UX. Video takes seconds-to-minutes. The product is mostly about making waiting feel intentional — optimistic UI, job queues, auto-refunding failed jobs so a timeout never costs a user a credit.
- Glue > models. The models are an API call. The studio is chunkers, parsers, queues, a credit ledger, and a lot of error handling. That's the actual product.
The takeaway
If you want to understand these models, stop reading and wire three of them into one app. The cheapest experiment is still the same one I ran: feed a model a single image and watch what it does with time. The result of mine, if you want to poke at it, lives at geminiomni-ai.com — but the real value was the debugging, not the demo.
Happy to compare notes if you're building in this space.
Top comments (0)