DEV Community: Deep Variance

Where the GPU time goes when you serve a video model

Deep Variance — Fri, 23 Oct 2026 03:28:51 +0000

Our post on denoising steps explained what one clip costs: the size of each pass through the network, times the passes in each step, times the steps. That is the cost of drawing.

A model in production pays for more than drawing. The bill counts every second the GPU is reserved, and many of those seconds are spent on something else: waiting for work, moving weights in and out of memory, sharing badly, or making a clip that is thrown away. This note goes through the four, and the levers for each. Every lever trades something, so every one needs measuring.

Waiting: the GPU with nothing to do

A GPU that is switched on and idle costs the same as one that is busy. Traffic for a video product comes in bursts: a launch, a time zone waking up, a batch job. Between the bursts, a server sized for the peak waits.

The obvious answer is to switch GPUs off when nobody needs them. The price is the cold start. Before a new worker can draw anything, it has to load the weights, which for a large video model fill a good part of the GPU's memory, and often warm up compiled kernels that are built on the first call. A request that lands on a cold worker waits for all of that.

So the lever is a balance, not a switch: how many workers stay warm, how fast new ones start, and how long a customer can wait. Each workload has a different answer, and the answer moves as the traffic does.

Moving: when the model does not fit

A video pipeline is more than the denoising network. It also has a text encoder that reads the prompt and a decoder that turns the result into frames. Together they may not fit in one GPU's memory, especially at long lengths and high resolutions.

The common fixes move work in and out of memory. Hugging Face's guide to reducing memory in diffusers describes the options plainly:

CPU offloading moves parts of the model to the CPU and back when they are needed. It reduces memory dramatically, and the guide calls it extremely slow.
Model offloading moves whole components instead, keeping the one that runs many times, the denoising network, on the GPU while it works. It costs less time.
Decoder tiling decodes the frames in overlapping tiles to lower the peak, with a warning that tiles can differ slightly in tone.

Each fix turns a memory problem into a time problem, or a quality risk. The cheapest choice depends on the GPU, the model and the clip size, which is why it should be measured on the hardware that will run it rather than chosen once.

Sharing: one GPU, one clip

Language model servers get much of their efficiency from batching: many requests share each pass through the network, and systems such as Orca schedule them token by token so the batch stays full. Video is harder. One long, sharp clip can fill a GPU's memory by itself, and requests differ in length, resolution and format, so they do not batch neatly.

There are two ways to change the unit of work. You can split a large GPU into smaller isolated pieces, as NVIDIA's Multi-Instance GPU does on some data center GPUs, so small jobs stop wasting a whole card. Or you can split one clip across several GPUs: xDiT combines methods such as sequence parallelism and PipeFusion to run one diffusion transformer request on many GPUs at once, which cuts the wait for a long clip.

Neither is free. Small slices limit the clip size. Splitting a clip adds communication between GPUs, so the speed-up depends on the model, the clip and how the GPUs are connected.

Redoing: work that is thrown away

Some GPU time makes a clip that nobody sees. A production pipeline that checks its output, as a careful one should, makes a shot again when it fails: wrong words, stray text, a frame that breaks a rule. Every retake costs a full shot of GPU time. Fewer failures at the source is a cost lever, not only a quality one.

There is also repeated work inside a single clip. Neighbouring denoising steps often compute similar things. Training-free caching methods reuse part of that work instead of computing it again: DeepCache reuses high-level features between steps, and TeaCache estimates, for video models, which steps can reuse earlier outputs.

Reusing work changes the numbers inside the model, so it can change the picture. A cache setting that is invisible on one prompt can soften motion on another. It has to be compared with the original model's output, like every other speed change. Our post on verifying every speed-up explains how we do that.

Measure all four together

The four interact. Keeping more workers warm cuts waiting and raises idle cost. Offloading fits a bigger model and slows each clip. Splitting a clip across GPUs shortens the wait and adds communication. Caching saves passes and can raise the retake rate if it hurts quality.

That is why we treat serving as a loop rather than a setting. Deep Variance builds the auto-tuning inference stack for production diffusion workloads: it profiles each model on the GPU that runs it, tunes it to that GPU, verifies each change against the original model's output, and tunes again when the model, the GPU or the traffic changes.

If you run video models in production, a few questions show where your GPU time goes:

How busy are your GPUs across a day, and how long does a cold start take?
Does the whole pipeline fit in memory at your longest clip, or does it offload?
How do requests share GPUs, and what does splitting a clip buy you?
How many takes are thrown away, and has any caching you use been checked against the original output?

If you want to compare notes on yours, write to us.

Originally published on the Deep Variance blog.

Motion graphics from a prompt, written as code

Deep Variance — Fri, 23 Oct 2026 03:28:50 +0000

Some videos should not be filmed. A sale that starts on Friday, a product that launches next week, a bakery that opens on Saturday: the message is a few words, a date and maybe a picture. What makes it work is how the words move. That is motion graphics: designed type, shapes and pictures, animated to music.

DV4 makes motion graphics from a prompt. This post explains how, because the method is unusual. There is no template behind it.

Where motion graphics beat footage

Filmed shots are good at people, places and things in light. Motion graphics are good at messages: a launch date, three reasons to buy, a before and after, an opening announcement. They read clearly on a phone with the sound off, and every word is exact, because every word is set as type.

In DV4 a motion graphics video is designed motion with your pictures and music. No footage is generated. It is on the paid plans, in landscape or vertical.

No template: a film written for this request

A common way to make an animated video from a prompt is a template: a set of layouts and moves with your words dropped in. It is fast, and every customer gets the same moves.

DV4 has no templates. A model writes each film as code: the layout of every scene, the type, the colours, the way each element enters and leaves, the cuts. It works the way a motion designer who writes code would, with a file of its own that it edits, renders and edits again.

So that two requests do not get the same film, there is no house style. No default palette, no default easing, no end card that every video shares. Each job draws its own style direction and its own references, and the model commits to them unless your prompt or your pictures say otherwise.

How the rest of DV4 works, the planning, the checks and the assembly, is in From one line to a finished video.

The model looks at its own drafts

Writing code for a film without seeing it is like setting type with your eyes closed. So the model renders a draft, looks at frames from it and fixes what it sees: a title that is too small, a scene that crowds the edge, a transition that hides the next line before you can read it.

It cannot skip that step. The model can only finish once the version it is finishing has rendered and been looked at.

What the checks catch

The model is not the only one looking. Code checks every render and hands the findings back before the film can be finished:

Frozen stretches. If nothing on screen moves for longer than half a second outside the final moment, the model is told where, so it can keep something moving.
Words at the edge. Text recognition reads the frames. A word that touches the edge of the frame, or crowds it, is reported with its time.
Black frames. A flash of black between scenes is caught and named.

Before you see the finished film, its words and its frames go through DV4's safety checks, as every video does. The code itself is scanned before it ever runs, may not call the network, and runs only in a headless browser on our own workers.

It will not invent your numbers

Animated statistics are a motion graphics favourite, and a liability. A model that is asked for "three big numbers" will happily make some up.

DV4's plan is checked for exactly that. A statistic appears only when the figure came from you. A quote appears only when you wrote the words in quotation marks. If you want "40% off" on screen, put "40% off" in your prompt.

Your pictures, your music

You can add your own pictures, of the product, the shop or the team, and the film is designed around them rather than around stock shapes. For sound, DV4 composes music for the film, and the scenes are cut to its beats. If you have a track you want to use, upload it as your soundtrack instead.

Motion graphics play over music, not over speech. A motion graphics video does not take a voice recording.

How to write the prompt

Say what the film is for, what it must say and how it should feel:

A 15-second vertical announcement for Bloom Bakery's grand opening this Saturday at 9 am. Warm and lively. Show the date big. End with the address: 12 Elm Street.

A few habits help:

Put every exact word in the prompt. Names, dates, prices and addresses appear as you wrote them.
Give the length and the format. Vertical for reels and stories, landscape for the web.
Name a feeling, not a font. "Calm and premium" or "loud and fast" gives the model room to design.
Add your pictures when the film is about something real.

What it does not do yet

Motion graphics in DV4 are not narrated: there is no voice-over. The film does not use a brand kit yet, so give your colours in the prompt if they matter. It does not edit a video you already have. And when a film fails, which can happen with a model writing code, the job is refunded and you can run it again.

Try it

Write the announcement you have been putting off, in one paragraph, and watch DV4 design it, look at its own drafts and finish it. You can make motion graphics in DV4 on any paid plan.

Every draft is rendered and every frame is checked, and all of it runs on GPUs and CPUs that cost money by the second. Making that work faster and cheaper is the other half of what we do at Deep Variance: we build the auto-tuning inference stack for open-source diffusion models, and DV4 runs on the stack.

Originally published on the Deep Variance blog.

AI video with dialogue: lip sync from the words in your prompt

Deep Variance — Fri, 23 Oct 2026 03:12:03 +0000

A video of a barista saying "Your usual?" to a regular is a different thing from a video of a barista pouring milk. The first one has a person in it. It is also much harder to make well. The words must be the ones you wrote, the lips must move with them, and the voice must sound like it belongs in a busy café rather than in a recording booth.

This post explains how DV4 makes video with dialogue from a text prompt, how to write the lines, and how DV4 checks that the people on screen said what you asked. It also lists what dialogue in DV4 does not do today, because the limits matter when you plan a video.

Dialogue that comes out of the shot itself

You write the lines in your prompt, and the people on screen say them, with their lips in sync with every word, in the scene's own sound. There is no separate voice track laid over a silent picture.

That is the important part. DV4 generates the speech, the lip movement and the sound of the place in the same pass as the picture. The barista's voice carries the room: the espresso machine behind her, the cups, the street through the door. Nothing has to be mixed to make it sound like it was recorded there, because it was made there.

Dialogue is on the paid plans. On the free plan, videos have the scene's sound, but nobody speaks.

How to write the lines in a prompt

Write the scene the way you would brief a director, and put the spoken words in quotation marks. For example:

A small café in the morning. A barista in a green apron looks up as a regular walks in and says "Your usual?" He smiles and says "Make it a large today."

A few habits help:

Say who speaks. "The barista says" is clearer than a line floating in the prompt.
Keep lines short. One or two sentences per person per shot reads naturally and is easier to keep in sync.
Describe the place. The sound comes from the scene, so a busy kitchen sounds different from an empty office.
Leave text out of the picture. Ask for words on screen as on-screen text, not as a sign in the shot. DV4 adds titles as exact type afterwards.

DV4's planner turns your prompt into a shot list with every spoken line written down in the plan, shot by shot. That script matters later, when DV4 checks the result.

Why the lips stay in sync

Lip sync goes wrong when the voice and the face are made separately and then lined up. The timing drifts, a word lands a frame late, and the mouth makes a shape that no sound matches.

In DV4, the words go into the shot's prompt in quotation marks, and the video model produces the voice and the moving mouth together. Because there is one source for both, there is nothing to line up. The same is true for the rest of the sound in the shot.

How the full pipeline runs, from your line to the finished file, is in From one line to a finished video.

DV4 listens to every shot

A video model can still get the words wrong. It may drop a word, swap one, or mumble a line into something else. So DV4 does not take the shot on trust.

When a shot with dialogue comes back, speech recognition listens to its sound and writes down what it hears. DV4 then compares those words with the lines in the plan. If too many words differ, the shot fails the check.

Shots with no lines get the opposite check. A shot that should be silent must not have someone talking in it, so DV4 listens for unexpected speech there too.

When a shot fails the check

A shot that fails is not used. DV4 makes it again with a new seed, a new random start that gives a different take, and the new take goes through the same checks.

A silent shot that keeps producing speech gets one more try. If that try also fails, DV4 drops that shot's scene sound rather than ship a stranger muttering in the background, and it tells you. With each finished video you see its flags, so you know what changed before you post it.

Captions from the words that were spoken

Captions in DV4 come from the same check. The words come from your script, so they are spelled the way you wrote them, names included. Only the timing comes from the speech recognition, so each caption appears when its words are spoken.

Captions are burned into the video, and you also get a WebVTT captions file to download. When faces fill the bottom of the frame, the captions move to the top so they do not cover anyone's mouth. Composed music sits under the dialogue rather than on top of it, on the same paid plans.

What dialogue in DV4 does not do yet

Plan around these:

No voice choice. People speak with the video model's own voice. You cannot pick a preset voice or clone your own.
No narration or voice-over. Speech comes from people on screen, not from a narrator over the footage.
No uploaded recordings. You cannot upload an actor's line and have DV4 build the picture around it.
No real public figures. DV4 refuses to make them, with or without dialogue.

If your video depends on a specific voice, a brand voice or a recorded performance, DV4 is not the right tool for that part today.

Try it

Write a scene with two lines of dialogue in quotation marks, choose landscape for the web or vertical for a reel, and watch the shots get planned, filmed and checked. You can make a video where people speak in DV4 on any paid plan, and start on the free plan to test the scene and its sound first.

Every check in this post runs a model, and every model call is GPU time. Making those calls faster and cheaper is the other half of what we do at Deep Variance: we build the auto-tuning inference stack for open-source diffusion models, and DV4 runs on the stack.

Originally published on the Deep Variance blog.

AI product video from one prompt: what comes out finished

Deep Variance — Fri, 23 Oct 2026 03:06:51 +0000

Search for "AI product video" and most of what you find is a clip. A video model takes a prompt and returns eight seconds of a bottle turning in soft light. It looks good. It is not an ad. An ad has a product name spelled correctly, a cut that lands on the music, a line of text that says what to do next, and a length that fits where it will run. Somebody still has to do all of that, and that somebody is usually you, in an editor, at eleven at night.

This post is about the gap between the clip and the ad, and which parts of that gap DV4 closes from one line. We say where it does not, too.

What a product ad needs that a clip does not have

Take a plain brief: a 15-second ad for a ceramic pour-over kettle, calm morning mood, for the shop's home page. To ship it you need:

more than one shot, so the video has a start, a product moment and a close
the product name on screen, spelled right, in a readable size
music that fits 15 seconds and sits under the picture rather than on top of it
an end card with the one thing you want people to do
a check that nothing in the shots is wrong before it goes out

A video model gives you the shots, one at a time, from one prompt each. The rest is production work. DV4 does the production work around the model. How the pipeline runs, step by step, is in From one line to a finished video. Here we stay with the ad.

The prompt: the product, the mood and the length

You write what you want to see. One line is enough: the product, the mood, the length and where it runs. DV4 plans every shot from that line, films each shot and hands you a finished video, with the sound, the music and the text in place.

A longer prompt helps when you know what you want. Name the setting, the colour of the product, the one feature the ad is about and the words you want on screen. DV4 keeps your words; it does not invent a slogan for you unless you leave the text to it.

Your own pictures of the product

A model that has never seen your kettle will draw a kettle. For a product ad that is not good enough, so on paid plans you can upload pictures of the product, up to four per video. A shot can start from one of them as its first frame or use it as a reference, so the kettle in the ad is your kettle: the same spout, the same handle, the same glaze.

If you have no pictures yet, the free plan still works from the prompt alone. That is the right place to test a concept before the product photos exist.

Shots that follow a plan

DV4 does not send your line straight to the video model. A planning model first writes a shot list: framing, camera movement and length for each shot, and the words that go on screen. Then the shots go to the video model one by one. The free plan makes up to two shots; paid plans make more, up to sixteen on the highest plan, with a video of up to two minutes.

For a product ad the plan matters more than the model. A wide establishing shot, a close product shot and a final shot with the end card is a structure that works for a kettle, a candle or a pair of shoes. DV4 writes that structure for you.

The product name as type, not as pixels

Video models draw letters badly. Ask one for a kettle with the brand on the box and you get a brand that is almost yours, with one letter wrong. For an ad that is the worst possible failure, because the one thing the viewer must read is the one thing that is broken.

So DV4 never asks the video model to draw text. Every shot prompt asks for no readable text at all. The title, the callouts and the end card are added afterwards as exact type, on paid plans, from the words in the plan. Your product name is set in a font, at a size that reads on a phone, in a place that does not cover the product. If you turn captions on, they come from the words actually spoken, burned in, with a captions file to download beside the video.

Music that fits the cut

An ad needs music that is the right length. A 15-second ad with a 30-second track fades out in the middle of a phrase, and everyone hears it. On paid plans DV4 composes an original instrumental for each video, made to its length and mixed under any dialogue. If you already have a track, upload your own soundtrack and it plays in place of the composed music.

Every shot checked before you see it

Video models make mistakes: a stray word drawn on a wall, a mouth moving in a shot that should be silent. On paid plans DV4 checks each shot before it goes into the video. Text recognition looks for letters the model drew by mistake. Speech recognition listens to the shot and compares what it hears with the script. The frames are checked for safety, in every job, on every plan. A shot that fails is made again.

When something in the final video is not as planned, DV4 tells you. With each result you see any flags, for example that the music was left out after a failure, so you know what you are looking at before you post it.

Landscape for the web, vertical for reels

Every plan makes landscape and vertical video. The home page gets 16:9; the reel gets 9:16. Resolution depends on the plan, from 720p up to 4K.

A 30-second 1080p video is usually ready in about five minutes. On paid plans you can start several videos of the same request at once and follow each shot as it is planned, filmed and checked, then keep the one you like.

What you still do yourself

DV4 does not know your product. You bring the pictures, the name and the one line that says why it matters. You decide where the ad runs, and you watch the result before it goes out. DV4 does not edit a video you already have, and it does not read your product from a web page: the input is a prompt, and on paid plans your pictures and your soundtrack. Brand kits and team seats are not in the product yet.

One more limit, on purpose. DV4 never makes sexual or adult content, and it never makes real public figures. Every request goes through the same safety checks, on every plan.

Start with a 15-second clip

The free plan needs no card. Write one line about your product and you can make a clip of up to 15 seconds at 720p: enough to see whether the concept works before you spend anything. When it does, a paid plan adds your pictures, the type, the music and the checks. You can make a product ad with DV4 today.

At Deep Variance the video model is one step among many, and each step is GPU time. Running those steps faster and cheaper is the other half of our work: we build the auto-tuning inference stack for open-source diffusion models. DV4 runs on the stack. The ad you make is the proof that it works.

Originally published on the Deep Variance blog.

From one line to a finished video: inside DV4

Deep Variance — Fri, 23 Oct 2026 03:06:38 +0000

DV4 is an agentic AI platform for creators. That means it plans the work, does the work and checks the result. You write one line, and DV4 makes a finished video, with the shots, the music and the on-screen text in place. It runs on open-source models and on our auto-tuning inference stack. It is live today at dv4.ai.

A video model makes one clip from one prompt. It does not plan a story, write music, check its own work or set type. DV4 does those jobs around the model. This note goes through those jobs in order. Our example line is: "A short ad for a matte white kettle, calm morning mood."

DV4 does each job in order. When a shot fails its checks, DV4 makes it again.

1. Safety, before DV4 makes anything

The first step is not creative. DV4 checks the request against its content rules before it spends anything. If the request breaks a rule, the job stops and DV4 makes nothing.

The safety checks run again later: on the plan, including every spoken line, and on the frames of each shot. If the safety check cannot run, the job stops and does not count against your plan. We prefer a refused job to an unchecked video.

2. A plan, written like a director's

Next, a planning model reads your line and writes a plan. The plan is not a longer prompt. It holds:

a shot list, with framing, camera movement and length for each shot
the spoken lines, if anyone talks
a music brief: style, mood and tempo
the words that go on screen, such as a title or an end card

The planner works from a reference on cinematography that our founders wrote. Then code checks the plan. Does each shot fit the length limits? Do the shots add up to the length you asked for? Does a shot ask the video model to draw text? If a check fails, the planner gets one more try with the failures listed.

You can also choose a motion graphics video instead of filmed shots. For that, a model writes the whole film as code. It renders drafts, looks at frames from them and fixes what it sees before it finishes.

3. Music comes before the pictures

DV4 makes the music beside the plan, not after the video. The track gets its beats measured. The timeline then places the cuts on those beats.

Speech works another way. When a person talks on screen, the video model makes the voice and the lip movement in the same pass as the picture. That keeps the lips in sync with the words.

4. Shots, made one by one

Now the timeline fixes each shot's exact length in frames, and the shots go to the video model. A shot that continues the one before it starts from that shot's last frame, so the action carries on across the cut.

Shot prompts never ask for readable text. Video models often misspell words, and a misspelled product name ruins an ad. The words come later, as exact type.

5. Every shot gets checked

When a shot comes back, DV4 runs these checks on it:

The file itself. Are the size, frame rate and frame count what the timeline asked for?
Stray text. Text recognition looks for letters the model drew by mistake.
Speech. Speech recognition listens to the shot and compares the words with the script.
Unexpected speech. A shot that should be silent must not mumble.
Safety. DV4 checks sampled frames for unsafe content.

The first take has letters the model drew by mistake, so it fails. A new seed gives a new take, and the new take passes.

When a shot fails, DV4 does not ship it. It makes the shot again with a new seed, a new random start that gives a different take. If the second take also fails, DV4 picks a fallback that fits the failure:

It keeps the better of the two takes and flags it.
It voices the lines a different way.

An unsafe shot gets one more try with stricter safety rules. If that take also fails, DV4 stops the job, and the job does not count against your plan.

6. Words, set as type

The title, callouts and captions are not drawn by the video model. A design model writes a layout: font, size, place on screen and timing. Code then measures each block against the real frames. Does it cover a face? Is the contrast high enough to read? Then the model looks at stills of the result and fixes what reads badly.

Captions use the words from the script and the times from the speech check. So each caption appears when its words are spoken.

7. Assembly

Last, DV4 joins the shots and smooths the joins between takes that continue each other. It lays the music under the dialogue, renders the text on top and runs a final probe on the file. You get an MP4, plus a captions file when you ask for one.

What this means

You write one line. DV4 does the planning, the checks and the retakes. You do not need to know which model made which shot, or how to time a cut to a beat.

Many of these steps run a model, and models cost GPU time. Making that work faster and cheaper is the other half of what we do at Deep Variance. We build the auto-tuning inference stack for open-source diffusion models.

DV4 is live. Start creating at dv4.ai.

Originally published on the Deep Variance blog.

Why denoising steps decide what a generated video costs

Deep Variance — Fri, 23 Oct 2026 03:00:43 +0000

Ask a text model for a sentence and it writes the sentence once, word by word. Ask a video model for five seconds of a cat on a windowsill, and it draws the whole clip many times. Each draft is a little less noisy than the one before.

That one fact drives much of the GPU cost of video. This note explains it without math: how the work adds up, where the levers are, and why each lever needs a check. It is for anyone who pays for, plans or runs open-source video and image models.

Drawing out of noise

Most open-source image and video models today are diffusion models, or close relatives trained with a method called flow matching. For this post, both work the same way. They start from pure noise, random static in the shape of the output. Then they remove a little noise at a time, guided by your prompt, until only the picture remains. Each removal is one denoising step. The idea dates from a 2015 paper. DDPM showed in 2020 that it can make high-quality images.

Each step is a full pass through a large neural network. The network does not touch only the part of the frame that changes. It reads and rewrites the whole clip, every time.

From noise to picture. Each arrow is one step, and each step is a full pass through the network.

So the cost of one clip is close to this:

the cost of one pass × the passes in each step × the number of steps

Most steps need one pass. Guidance, which this post explains below, can make it two. Everything else in this post is about one of those three terms.

Multiplier 1: the number of steps

Fewer steps means fewer passes. There are two common ways to get there. A better sampler takes larger steps with the same model, as DPM-Solver shows. Distillation goes further. Engineers train a second model to reach the same result in fewer steps than the original. Progressive distillation is one early, public example of the method.

Distillation has a price. A distilled model is a different model. It can lose detail, light or variety. You have to compare its output with the original before you trust it.

Multiplier 2: the size of each pass

A clip is too large to work on as raw pixels. So modern models work in a smaller latent form. During training, an encoder learns to squeeze clips into that form. To make a new clip, the model denoises in the small space, and a decoder turns the result into frames at the end. Latent diffusion made this approach standard.

Inside the network, the latent clip is cut into small patches across space and time, and each patch becomes a token. Many current video models are transformers that work on these tokens. The DiT paper introduced this design for images.

Here is the important part. Attention is the part of a transformer that lets each token look at every other token. Its compute grows with the square of the number of tokens, while the rest of the pass grows linearly (Attention Is All You Need). If a clip has twice the tokens, attention does about four times the work. So as clips get longer or sharper, attention takes a larger share of each pass. That is why a longer or sharper clip costs much more than a short, small one.

The levers for the size of a pass are:

Attention kernels. A kernel is a small program that does one calculation on the GPU. An attention kernel that is written for how the GPU's memory works wastes less time moving data. FlashAttention is a public example.
Lower precision. Weights and activations can use fewer bits per number. Less data moves, and newer GPUs do more low-precision math per second. The FP8 formats paper describes one common choice.
Fused operations. Two small steps that each read and write memory can often run as one step that reads once.

Each of these changes the numbers inside the model a little. So each one needs a quality check too.

Multiplier 3: guidance can double each step

Many models use classifier-free guidance to follow the prompt more closely (Ho and Salimans). In its common form, each step runs the network twice. One run uses your prompt and one run does not, and the model moves away from the second toward the first.

That can double the work in every step. Some distilled models need no guidance, which removes the second run. Again, the output can change.

The parts outside the steps

The steps are usually most of the work, but not all of it:

The decoder. After the last step, the decoder turns the latent clip back into frames. For long, sharp video, the decoder can run out of GPU memory. Decoding the clip in tiles, one part at a time, keeps it inside the memory budget.
The text encoder. The text encoder turns your prompt into numbers once per clip. It runs once, so it takes little time next to the steps. But its weights can take a lot of GPU memory.
Loading the model. Weights for a large video model can fill a large part of a GPU. Loading a large model for each request would waste much of the GPU time, so a serving stack keeps models loaded and warm.

What one clip costs, not to scale. With guidance, each step takes two passes. The model loads once.

What to take away

If you pay for diffusion inference, ask three questions about any setup:

How many steps does each output take, and was the model distilled to take fewer?
How large is each pass, in frames, resolution and precision, and are the attention kernels matched to the GPU?
Is guidance on, and does the product need it?

Each answer points to a lever. Each lever also changes the output. That is why we compare each speed change with the original model's output. That is our work at Deep Variance. We build the auto-tuning inference stack for production diffusion workloads. It tunes each model for the GPU that runs it and checks each change against the original model.

If you run diffusion models in production and want to compare notes, write to us.

Originally published on the Deep Variance blog.

Why every speed-up needs a quality check

Deep Variance — Fri, 23 Oct 2026 03:00:38 +0000

Here is an easy way to make a video model faster: make it worse. Cut the steps in half. Drop the precision. Turn off guidance, the second pass that keeps the picture close to your prompt. The model will render faster. It can also draw blurrier faces, flatter light and hands with the wrong number of fingers.

Every speed-up in inference is a trade. Some trades cost nothing you can see. Some cost a lot. The only way to know which is which is to measure. This post is about why we treat that measurement as part of the work, not an extra step at the end.

A faster model can be a different model

The changes that make diffusion inference cheaper all touch the numbers inside the model:

Lower precision stores and multiplies numbers with fewer bits. Small rounding errors appear in every step.
Faster attention kernels compute the same thing in a different order, or at lower precision. The results can differ in the last digits.
Fewer steps, through a distilled model, give a model that learned to imitate the original, not the original itself.

A diffusion model runs its network many times per output, and each step feeds the next. A tiny difference in step one can grow into a different face by the last step. So "it ran faster" tells you nothing about whether it still makes the same video.

Wrong answers can look right

The wider field now measures this. When AI systems write GPU kernels, some of those kernels pass weak tests and fail strong ones. KernelBench-Verified tested AI-written kernels against a stronger baseline and with hidden test inputs. Some kernels skipped work they were meant to do, and the average gain turned into a loss. The Correctness Illusion built a small set of kernels with known bugs. A common test on one input shape passed them. Tests with many random inputs caught every one.

Even correct code can give a user different outputs for the same request. Thinking Machines showed this for language model servers (Defeating nondeterminism in LLM inference). Some kernels give slightly different results at different batch sizes, and the batch size changes with the server's load. The same is true of every kernel we change, so a test must cover more than one case.

For images and video, verification is harder. Two different videos can both look fine, so "it looks fine" is not a test. With the same prompt and the same random start, the original model gives one reference video. We compare against that.

Same prompt, same random start. The faster version drifts away from the original, and its score falls below the limit.

How we verify a change

Our rule: we compare each speed change with the original model on the same type of GPU. In practice:

Render a reference. We run the original model, as its makers ship it, on the same GPU and the same prompts.
Render the candidate. We run the changed version with the same prompts and the same seeds, the numbers that set the random start.
Compare. We score how close the candidate's frames are to the reference. We score how well the output still matches the prompt. When people speak, we score whether the words and lip movement hold.
Measure speed. We time the candidate, warm, on the same GPU.
Decide. We set a quality limit for each model on each type of GPU. A change is accepted only if it is faster and its scores stay above that limit.

Some changes we try fail this. One faster setting moved the frames too far from the original, so it is not our default. Another worked in short tests and broke on long, sharp clips. Those changes do not ship. A record of what failed is as useful as a record of what worked.

Why an autopilot needs this step

Deep Variance builds the auto-tuning inference stack: an inference stack on autopilot, designed for production diffusion workloads. It runs one loop:

Profile each model on the GPU that runs it: speed, memory and output.
Tune each model to its GPU: precision, attention and memory use.
Verify each change with the steps above.
Reassess when the model or the GPU changes, and tune again.

A loop that tunes on its own is safe only if it stops changes that make the output worse. The comparison is that stop. Without it, autopilot is only a faster way to make mistakes.

The loop and its gate. A change that fails goes back to Tune. A change that passes ships.

Why it matters for enterprise deployments

When we deploy our stack on a team's own GPUs, verification is part of the work, not an extra:

We measure the model on the team's GPUs and traffic before we change anything.
We compare each change with the original model's output on that hardware.
We record the result, so the team can see what changed and what it cost in quality.

A team that runs a video model in production cannot afford a silent drop in quality. A cheaper output that customers like less is not cheaper. Verification shows, in numbers, what each saving costs in quality, so the team can decide with the facts.

The takeaway

Speed-ups are easy to claim and hard to trust. When anyone offers you faster diffusion inference, ask three questions:

Faster than what, measured where, and on which GPU?
Compared with what output, and by which measures?
Written down where, so you can verify it later?

We ask ourselves the same questions for every change. If you run diffusion models in production and want a deployment you can verify, write to us.

Originally published on the Deep Variance blog.