What I Learned Wiring an AI Video Model Into a Content Pipeline

rucai zhou — Tue, 18 Aug 2026 02:12:53 +0000

We needed short product videos on a schedule our two-person team could not hit by hand. The tool that ended up doing the heavy lifting was Runway Aleph, an in-context video model — and for the headshots that go alongside them we use BeardlessAI, a free tool to remove beards from photos. This is a writeup of the integration, including the parts that did not work.

The problem with generate-first tools

Our first attempt used text-to-video. Type a description, get a clip. It demoed beautifully and was useless in production for one reason: no continuity. Every generation was independent. Two clips of "the same" product were two different products. You cannot cut that together.

In-context models work differently. You pass real footage as context and describe a bounded change against it:

input:  product_shot_v3.mp4
prompt: "Remove the price tag on the right side of the box.
         Keep the box, the surface, and the lighting unchanged."

The job is not invention. It is a scoped edit to something that already exists. That is what made it fit an actual pipeline.

What the pipeline looks like

Roughly four stages:

raw footage ──▶ shot manifest ──▶ edit passes ──▶ QC gate ──▶ publish
                  (JSON)          (one op each)   (human)

The shot manifest was the part I underestimated. It is just a JSON file describing each shot and the operations queued against it:

{
  "shot_id": "hero_02",
  "source": "raw/hero_02.mp4",
  "ops": [
    { "type": "remove",  "target": "price tag, right of box", "preserve": "box, surface, lighting" },
    { "type": "relight", "target": "overcast afternoon",      "preserve": "subject position, framing" }
  ]
}

Two things this bought us. First, ops are replayable — when we changed our mind about the look, we edited the manifest and re-ran rather than reconstructing what we did from memory. Second, it forced one operation per entry, which turned out to matter a lot.

Lesson 1: one operation per pass, always

I tried batching. "Remove the price tag, relight it as an overcast afternoon, and add a plant on the left" produced output where all three happened, all three badly.

Split into three passes, each one was clean. More calls, more credits, better results — and crucially, when pass two looked wrong I knew it was pass two. Batched prompts are not debuggable. You get a bad frame and no idea which clause caused it.

Lesson 2: `preserve` is not optional

Early prompts only said what to change. The model kept over-reaching — remove one object, and the background subtly re-renders, so the shot no longer cuts against its neighbours.

Adding an explicit preserve clause fixed most of it:

- "Remove the car"
+ "Remove the silver car at the left edge of frame.
+  Keep the pedestrians, storefront and lighting unchanged."

That diff is responsible for more quality improvement than every other change we made.

Lesson 3: the QC gate has to be human, and at full res

We tried to automate QC with a similarity check against the source. It caught gross failures and missed everything that mattered.

The failures that actually hurt are semantic: a hand with the wrong number of fingers, signage that became plausible gibberish, a reflection that no longer matches the object casting it. Structural similarity scores these as fine.

Failure clusters, reliably:

Region	Failure rate	Notes
Hands / fingers	High	Still the worst case
Text & signage	High	Degrades to convincing nonsense
Reflections, glass, chrome	Medium	Physically inconsistent
Rigid geometry	Medium	Small warps are very visible
Frame edges	Medium	Least surrounding context

We now gate on a human reviewing those five regions at full resolution. Preview-window review passed shots that fell apart on a large screen, which we discovered in a client review rather than in QC. Once.

Lesson 4: cache aggressively, prompts are deterministic-ish enough

Same source plus same prompt gets you close enough to the same output that re-running is usually waste. We key a content-addressed cache on hash(source_file + prompt + params). During a week of iteration that cut spend by more than half, because most re-runs are re-runs of a shot you already settled.

What I would tell someone starting

Do not rebuild your pipeline around this. Add it at one stage — object removal is the highest-value, lowest-risk entry point — and prove it survives QC before extending.

And benchmark honestly. Compare against your existing process including review time, not just generation time. Generation is fast. Review is not, and review is where the real cost sits.

The gain here is not that impossible things became possible. It is that expensive things got cheap enough to attempt, which is quieter and more useful.

DEV Community: rucai zhou