Four cliffs of local video generation on an M1 Max 64GB — Wan2.2 I2V unified-memory cliffs and SeedVR2 not running on MPS

Originally published on Zenn (Japanese). Cross-posted here.

I'm going to write down every pitfall I hit trying to add "real camera work" to a still image with local Wan2.2 I2V. On an M1 Max 64GB with ComfyUI, this is the record of making a 5-second camera-flight clip from a single landscape photo and getting it onto the web. Every number I give is actually measured (seconds from logs, ffprobe, browser measurements) — no guesses.

Four things to take away up front:

1. On Apple Silicon's unified memory, generation time is not proportional to workload. Past a certain resolution, you fall off a cliff.
2. If you interrupt a long generation, restart ComfyUI before you fire the next one. The interrupted run's GPU reservation lingers, and the next generation starts "already on top of the cliff."
3. Don't "re-describe the picture" in an I2V prompt. The photo is the first frame. What you should write is only "the motion to add." Re-describing makes it drift to a different scene.
4. The SeedVR2 video upscaler doesn't finish on Apple Silicon (conv3d unimplemented). The alternative is ffmpeg lanczos.

Up front: this article is a record of settings and measurements. Power consumption, and how far you can push LTX further, are unmeasured (I cut off at one retry). The numbers I give are only values I actually measured — generation logs, ffprobe, browser seek measurements, etc.

Cliff ①: bumping resolution up "a little" went 27 s/step → 1693 s/step

Here are measurements from the same workflow (Wan2.2 I2V A14B Q5_K_M, MoE 2 experts, Lightning 4-step LoRA), varying only resolution and frame count.

Resolution × frames	pixels × frames	vs. min	measured s/step	time ratio	result
480×320×25f	3,840,000	1.0×	27.6 / 26.5	1×	normal (finished in 182.7s total)
720×480×81f	27,993,600	7.29×	461 (high) / 615 (low)	17×	finished
864×576×81f	40,310,784	10.50×	1693 → 2317 (worsening)	61×→84×	broke down, interrupted at 41 min

The workload is 7.3× and 10.5×, but the time is 17× and 61×. Not linear. What was happening on the 61× side: swap filled to 21.9GB / 22.5GB (539MB free) and it was thrashing. Past a certain line, unified memory melts time into disk round-trips instead of computation. 864×576 was pushing this machine's Wan2.2 A14B too hard.

Cliff ②: an interruption kills "the next generation"

This is the trap I got stuck on the worst. I interrupted 864×576 at 41 minutes, dropped to 720×480, and re-fired — and with the same settings, the first step took 28 minutes. Chasing the reason through measurement, the GPU's reserved memory looked like this.

Timing	GPU alloc system memory
right after interrupting 864×576	72,276,721,664 B (~72.3GB, exceeding the physical 64GB)
after stopping the ComfyUI process	42,425,729,024 B (~42.4GB) = ~30GB freed
MPS free right after restart (log)	52.70GB free / 64.00GB total

The interrupted run's reservation wasn't released, and the next generation started still over the physical limit — on top of the cliff from the start. Restart ComfyUI, get back to 42GB, and 720×480 ran at a normal 461 s/step. Miss this and you'll step on "same settings but slow" forever. The lesson is simple: if you interrupt a long generation, restart the process before firing the next. For the record, whether this un-released reservation is ComfyUI's leak, its handling of the interrupt signal, or MPS's unified-memory management, I did not pin down here (the observed fact is "a restart returned 30GB and normalized the speed").

Cliff ③: "re-describing the picture" in the prompt flies you to another world

I compared Wan2.2 with its rival LTX-2.3, on the same keyframe, same-intent prompt, same length (81 frames). But LTX's first pass threw away the input waterfall photo and turned into an unrelated landscape — a palm plantation, mountains, and sky.

LTX-2.3, first-pass failure. frame0 (far left) is the original waterfall. Everything after drifts to a different landscape.

Writing this off as "LTX's loss" would be inaccurate. The cause was my prompt. The source image is a ground-level waterfall, yet I had written aerial cinematic shot flying over rainforest canopy. Wan prioritized the image and absorbed the contradiction; LTX simply followed the text.

The primary source articulated why this happens, precisely.

Your photo is the first frame. If you spend your prompt re-describing the picture, you have told the model nothing about what should change, and you get generic drift.
— from segmind / VEED's Wan I2V prompt guide

In I2V the image is the first frame — the model already sees the subject, the place, the light. So every word of the prompt should go to "the motion to add"; describe the picture, or add a viewpoint that isn't shown (aerial on a ground-level photo), and it will rebuild the frame to create that = drift. My fixes:

• Match the prompt to the image's content (ground level, waterfall, mossy rocks, mountain stream) and write only the motion
• --image <path> 0 1.0 to anchor frame0 at strength 1.0 (the first pass had no strength specified)
• Raise cfg from 2.3 → 3.0 to strengthen adherence (※ the opposite of "lower cfg when it won't move." When the symptom is "departure," not "won't move," you swing the other way)

Re-comparing under fair conditions: numbers alone can't decide

I re-ran LTX with the conditions fixed and lined it up next to Wan. Top row is Wan2.2, bottom is LTX-2.3, left to right frame 0 / 20 / 40 / 60 / 80.

Re-comparison under fair conditions. Top = Wan2.2 / bottom = LTX-2.3. frames 0/20/40/60/80.

I measured divergence from frame 0 (resized to 256×171, RGB mean absolute difference; smaller = more of the original preserved) and change between adjacent samples (larger = more motion).

frame	frame0 divergence Wan	frame0 divergence LTX	change Wan	change LTX
10	0.0455	0.1239	0.0455	0.1239
20	0.0826	0.1352	0.0730	0.0626
40	0.1387	0.1385	0.1139	0.0832
60	0.1828	0.1395	0.1204	0.0833
80	0.2042	0.1420	0.1305	0.1025

This is the crux of the article. Look only at final divergence and Wan (0.204) > LTX (0.142) — Wan appears to "depart more from the original." But this metric can't distinguish "traveled far" from "became something else." The numbers alone can't judge it; they only gain meaning combined with eyeballing the frames.

• LTX: jumps at frame10 (0.124), then nearly flat = switches to a different waterfall early, then barely moves
• Wan: increases continuously, with change accelerating = keeps the same waterfall while advancing deeper

Verdict: it splits by use case, not superiority

	Wan2.2 I2V A14B	LTX-2.3 (MLX q8)
Preserving the input scene	◎ continuously preserved	△ moves to a different place
Sharpness	○	◎ clearly sharper
Generation resolution	720×480	896×576
Sampling time	~36 min	~20 min (68.75 s/step × 16)

For "make a beautiful landscape video from one image," LTX-2.3 wins. The picture is sharp, the resolution higher, and generation faster. Meanwhile, for "dive in while keeping the original landscape," it's Wan2.2. Here the use was a scroll-driven effect of entering into the world, and the fallback when the video can't load was a 2.5D parallax of the same still image. If the video showed a different place, the moment it switches it breaks, so scene preservation was the requirement = I adopted Wan2.2. Not a preference — an integration constraint.

The adopted Wan2.2 clip. It advances deeper while keeping the same waterfall, with foreground branches sweeping past.

Cliff ④: SeedVR2 doesn't finish on Apple Silicon (it melted 1 hour 42 minutes)

The intended finishing step was to generate at low resolution and upscale to 1080 with the dedicated upscaler SeedVR2. It doesn't finish.

NotImplementedError: convolution_overrideable not implemented  → F.conv3d
[INFO] Prompt executed in 01:42:03   ← zero output

The VAE decoder uses 3D convolution (conv3d), and MPS doesn't implement this operator. It's not out of memory, so lowering resolution or batch doesn't fix it. Both SeedVR2LoadDiTModel and SeedVR2LoadVAEModel offer only ['mps'] as the device choice, with no escape route to the CPU. The nasty part is that both the encode stage (≈104 s/batch × 27 = 47 min) and the DiT upscale itself (≈20 s/batch × 27 = 9 min) do finish. It looks like it's going fine, then the final decode turns it all to nothing.

But the output was intact. Because I'd ordered the pipeline to "save the raw output before the upscale step." That was a design change I made after wholesale losing 41 minutes of 864×576 work, and it protected the sampling result on the second accident (1 hour 42 minutes). In a multi-stage pipeline, always save the raw output before heavy post-processing.

The alternative is ffmpeg lanczos. Since the generation is natively 720 wide, upscaling beyond that just spends bytes on fake resolution. Measured: 960 = 3.8MB / 1152 = 4.8MB / 1280 = 5.5MB, so I adopted 960 width (1.33×).

Getting it on the web: for scroll-driven frame-stepping, use all-I-frames

I put this clip on a web page by moving the video's currentTime in sync with scroll amount (stepping frames). Encoding settings matter here. I exported the same material with different GOPs and measured latency by repeating seek 82 times in Chromium.

Encoding	avg	p95	max	size
all I-frames (-g 1)	20.6ms	22.9ms	40.3ms	6,081KB
GOP=10	48.3ms	66.7ms	71.9ms	4,514KB
GOP=30	74.5ms	126.3ms	271.1ms	4,150KB

With a long GOP, every seek re-decodes from the preceding keyframe, and GOP=30 is a worst case of 271ms = visibly janky. With all I-frames it stays around 20ms consistently and tracks the scroll. Size grows, but foliage's fine texture doesn't benefit much from inter-frame prediction, so the difference is small — taking the smoothness is worth it. The right answer is to squeeze size with crf and width, not by lengthening GOP.

On the real page, I also measured whether scroll progress (in-scene lA) matches the playback position. lA=0.05→0.251s / 0.2→0.983s / 0.35→1.76s = almost perfectly matching the theoretical value (lA × 5.06s). The frame-stepping works exactly as intended.

Wrap-up: measuring the "invisible cliffs" of local video generation

Unlike a cloud API, local video generation puts your own hardware's limits straight into the result. What I learned by measuring this time was four cliffs: time is not proportional to workload; an interruption kills the next run; don't re-describe the picture in the prompt; the upscaler doesn't always run. Each is a pitfall in the gap between "it ran" and "it ran as intended," and you'll step on them unless you measure. In the end, a 5-second clip diving into the depths of the forest while keeping the same waterfall plays at the entrance of a web page.