Plain-English takeaway (if you read nothing else): I spent a month testing whether a MacBook can generate AI video as fast as you'd watch it. It can't yet — my best result was about 11× too slow. Instead of hiding that, I published exactly where the time goes (mostly the final step that turns the AI's internal work into actual pixels), proof that a popular-sounding shortcut made videos worse, and evidence that the "same" AI video comes out different on different hardware. All of it is documented, and none of it is taken on trust.
Last month, I published a benchmark showing a 1.125× speedup from block-residual caching on 4-bit FLUX.
The main lesson was not the multiplier. It was that my original quality metrics had been measuring the wrong thing, and that acceleration claims often combine speed, trajectory preservation, and perceptual quality into one number.
For the follow-up, I chose a stricter target: real-time autoregressive diffusion video on an Apple M5 Max, with the definition of "real time" frozen before results were visible.
The tested configuration did not meet that target. The fastest claim-eligible result was 1.418 native generated frames per second, compared with a 16 FPS target. That is an 11.28× gap.
I am publishing the result because the measured bottleneck, one systems improvement, and two rejected hypotheses are useful even without a real-time result.
The evidence can be checked from a repository checkout:
git clone https://github.com/kkjcodes/liveframe
cd liveframe
python -m pip install liveframe
liveframe verify \
artifacts/liveframe-publication-claims.v1.json \
--artifacts-root .
liveframe recompute \
artifacts/liveframe-publication-claims.v1.json
The setup
LiveFrame evaluates Wan2.1-T2V-1.3B-based causal video models across NVIDIA H100 CUDA and Apple M5 Max MLX/Metal.
The experiments include:
- Causal Forcing++ for the clean M5 performance fixture
- Rolling Forcing for the CUDA-to-MLX portability study
- Frame-wise Causal Forcing++ for the H100 cache-reuse experiment
The clean M5 fixture produces 81 pixel frames at 480×832, corresponding to 5.06 seconds at the model's native 16 FPS.
Before holdout results were visible, the relevant protocols froze their prompts, seeds, content strata, horizons, thresholds, aggregation rules, and stop rules.
For the cross-runtime experiment, stochastic inputs were serialized once as BF16 tensors. CUDA and MLX consumed byte-identical tensors rather than relying on nominally matching random seeds.
LiveFrame separates four claim layers:
- Numerical trajectory: Does the candidate follow the reference latents?
- Intrinsic quality: Is the candidate video acceptable independently of the reference trajectory?
- Same-seed identity: Is it recognizably the same generated video?
- Complete-wall performance: How long does the measured generation pipeline take, including mandatory refresh, VAE decoding, materialization, synchronization, and encoding?
The clean M5 row excludes model loading and prompt encoding, so it is a complete measured generation wall rather than application startup time.
Result 1: A systems speedup with matching RGB digest records
Increasing MLX's bounded free-buffer cache from 1 GiB to 4 GiB reduced complete measured wall time on the frozen fixture from 69.167 to 57.127 seconds.
That is a 1.2108× speedup, or a 17.41% wall-time reduction.
All measured trials:
- Recorded the same SHA-256 digest for the 97,044,480-byte pre-encode RGB output
- Passed the pinned latent anchor
- Added zero swap
The raw RGB payload itself was not retained. The exact-output evidence therefore consists of repeated immutable trial records containing the same digest, not an independently redistributable copy of the raw pixels. The retained H.264 output and its decoded RGB have separate hashes.
The profile was more informative than the multiplier. In this M5 configuration, causal VAE decoding consumed 72.33% to 78.69% of complete measured wall time.
The transformer had already been distilled to one denoising step. For this configuration, decoder execution and memory reuse were the larger optimization targets.
Result 2: The measured gap to real time
With the allocator improvement included, the fixture reached 1.418 native generated FPS.
LiveFrame defines real time as at least 16 native generated FPS for a continuous 60-second stream, with time to first frame, p95 chunk latency, memory, thermal state, quality, identity, and drift disclosed.
Interpolated or duplicated presentation frames do not count toward native generation rate.
The measured gap is:
16 ÷ 1.4179056 = 11.2842×
This result has two important boundaries:
- It is an 81-frame clean fixture, not a sustained 60-second performance result.
- It is not an Apple Silicon impossibility result.
Quantization, lower-resolution tiers, different model families, custom Metal kernels, decoder redesign, and pipeline overlap remain open. Any interpolated presentation tier would also need to be reported separately from native generation throughput.
Result 3: Identical stochastic inputs did not preserve identity
The sealed cross-runtime holdout contained 128 H100 reference videos and 128 M5 candidate videos.
Each pair used byte-identical serialized stochastic inputs under the frozen Rolling Forcing schedule.
The results were:
- Same-seed identity: 0/128 videos and 0/32 prompt-horizon cells passed
- Intrinsic quality: 90/128 videos and 19/32 cells passed
- Pairwise temporal CLIP identity: 0/128 videos passed
Intrinsic failures increased with horizon:
- 15/64 videos failed at 15 seconds
- 23/64 videos failed at 60 seconds
The two runtimes consumed the same stochastic bytes but did not preserve the same autoregressive trajectory. Many MLX outputs remained independently acceptable under the intrinsic-quality policy, but none passed the complete same-seed identity contract.
This rejects the tested checkpoint and CUDA/MLX implementation pair under the frozen portability contract. It does not establish that every CUDA and MLX video implementation must diverge.
The experiment also does not isolate the first divergent operator or prove a specific numerical cause. Small implementation differences compounding through autoregressive feedback are a plausible explanation, but they were not causally identified here.
The practical lesson is narrower: visual inspection and nominal seed matching are insufficient for validating long autoregressive rollouts across runtimes.
Result 4: One cache-reuse mechanism failed its quality gate
I also tested a descendant of the FLUX caching idea: reuse cached keys and values across generated frames instead of across denoising steps.
The candidate computed a full cache anchor every third frame. For the next two frame refreshes, it substituted the anchor's normalized keys and values after repositioning the keys with temporal RoPE.
The experiment ran on an H100 using two content strata selected for high expected redundancy.
The mechanism passed 0/16 frozen prompt-horizon quality cells.
Additional results included:
- Pairwise temporal CLIP failed 64/64 videos
- Pairwise LPIPS failed 59/64 videos
- Intrinsic motion LPIPS failed 56/64 videos
Representative failures included reduced motion and changed same-seed trajectories.
Under the frozen protocol, numerical quality was a prerequisite for authoritative timing. Because the mechanism failed that gate, no timing claim was made for it.
The bounded conclusion is that this specific mechanism was not quality-safe:
- Raw K/V substitution
- All transformer layers
- Fixed reuse period
- Temporal RoPE repositioning only
The experiment did not separately isolate stale keys, stale values, skipped projections, skipped normalization, reuse depth, or reuse period. Adaptive, partial-layer, trained, corrected, and denoise-side reuse remain open questions.
What survives
Three things came out of the project.
1. The methodology
LiveFrame separates numerical trajectory, intrinsic quality, same-seed identity, and complete-wall performance.
Its evidence workflow includes frozen thresholds, serialized stochastic inputs, digest-bound provenance, separate numerical and timing eligibility, and fail-closed validation.
These controls prevented the target miss, cross-runtime divergence, and cache-reuse failure from being broadened beyond what the experiments support.
2. The verifier
The first PyPI release provides the evidence-verification CLI:
pip install liveframe
Given the claims ledger and its separately distributed evidence artifacts, the verifier:
- Re-hashes every referenced artifact
- Rejects missing files and digest mismatches
- Rejects unsupported schemas and unsafe artifact paths
- Recomputes the supported speedup, reduction, and target-gap arithmetic
It validates the machine-readable claim surface. It does not automatically fact-check every sentence in this article.
3. A clearer next target
The M5 profile redirects attention toward the causal VAE decoder.
Flash-VAED independently reports roughly 6× decoder acceleration on Wan and LTX-Video VAE decoders using different hardware and methods. That result does not predict the achievable speedup on Apple MLX, but it provides a useful reason to test decoder-focused approaches prospectively.
The next LiveFrame experiments will examine decoder optimization, quantization, lower-resolution tiers, adaptive reuse, and clean sustained 60-second M5 performance under newly frozen quality and timing contracts.
Links
- Code and evidence bundle
- LiveFrame on PyPI
- Paper: arXiv link coming once the preprint is live
- The earlier FLUX benchmark
This work builds on the open Wan, Causal Forcing, Causal Forcing++, Rolling Forcing, MLX, and mlx-video ecosystems. Thank you to the teams behind them.
Real-time diffusion video claims are more useful when native generation rate, complete-wall timing, quality, trajectory identity, and provenance are reported separately. Negative results with frozen denominators can help establish those boundaries.
Top comments (0)