DEV Community: Atlas Cloud

Writing High-Performance Kernels in TileLang, from GEMM to MLA

Atlas Cloud — Tue, 26 May 2026 08:50:38 +0000

If you write GPU kernels, you live somewhere on a spectrum. At one end is Triton: quick to write, but the compiler makes most of the layout and shared-memory decisions for you. At the other end is CUTLASS / CuTe: total control, at the cost of a lot of template machinery. TileLang sits in the middle. You write Python, but you say explicitly what lives in shared memory, how the pipeline is staged, and how warps split the work — and a layout inference pass fills in the rest.

In this post we'll cover the mental model, write a GEMM, and then build up to a real production kernel: DeepSeek's MLA decode, where the interesting decisions actually show up. The goal is not to be exhaustive. It's to show what you think about tiles, and where TileLang quietly does the hard parts for you. We'll finish with a more typical story from production — a kernel where the win wasn't speed at all.

The mental model

Here's the whole idea in three points.

A tile is a first-class object. A shaped chunk of data (block_M × block_K, say) is owned and operated on by a thread block, a warp, or a thread. You stop thinking purely at the thread-block level the way you do in Triton, and you stop hand-managing individual threads the way you do in CUDA.
You place buffers in the memory hierarchy yourself. You declare what goes to shared memory (T.alloc_shared), what goes to registers (T.alloc_fragment), and what's thread-local. This is the biggest difference from Triton, which hides shared-memory allocation and staging inside the compiler.
The compiler infers the thread mapping. Once you've said where a tile lives and what operation runs on it (a copy, a gemm, a reduce), a layout inference pass parallelizes it across threads and works out the register and shared-memory layouts. You can override it when you need to, but most of the time you don't. This pass is the load-bearing feature — by the time we get to MLA you'll see why.

If you're coming from Triton, here's the rough mapping.

	Triton	TileLang
Granularity	thread block + implicit vectorization	tile (block / warp / thread)
Shared memory	managed by the compiler	explicit alloc_shared + copy
Layout	the compiler decides	inferred, but you can annotate
Pipelining	tl.range + compiler	explicit T.Pipelined(num_stages=)
Tensor Core	tl.dot	T.gemm with a selectable warp policy
Backends	NVIDIA (mainly) / AMD	NVIDIA / AMD / CPU / WebGPU / CuTeDSL, plus Ascend & MUSA forks

The short version: if you want fine control over blocking, pipeline depth, and warp partitioning without writing CUTLASS, TileLang is the sweet spot. For simple elementwise or light fusion, Triton is still quicker to reach for.

Getting set up

conda create -n tilelang python=3.10 -y
conda activate tilelang
pip install tilelang                 # prebuilt wheel, easiest path

If you're going to touch the compiler passes, build from source instead (you'll need a local LLVM/CUDA toolchain):

git clone --recursive https://github.com/tile-ai/tilelang.git
cd tilelang && pip install -r requirements-dev.txt
pip install -e . -v --no-build-isolation

Let's write a GEMM

We'll start with the kernel everyone starts with: C = ReLU(A @ B). It's small, but it touches every primitive that matters — explicit buffers, parallel copy, software pipelining, the Tensor Core call, and an L2 swizzle.

import tilelang
import tilelang.language as T
import torch

@tilelang.jit
def matmul(M, N, K, block_M, block_N, block_K,
           dtype="float16", accum_dtype="float"):

    @T.prim_func
    def matmul_relu_kernel(
        A: T.Tensor((M, K), dtype),
        B: T.Tensor((K, N), dtype),
        C: T.Tensor((M, N), dtype),
    ):
        # grid dims: (#blocks along N, #blocks along M); 128 threads per block
        with T.Kernel(T.ceildiv(N, block_N), T.ceildiv(M, block_M),
                      threads=128) as (bx, by):

            # Say where each tile lives, explicitly.
            A_shared = T.alloc_shared((block_M, block_K), dtype)         # shared memory
            B_shared = T.alloc_shared((block_K, block_N), dtype)
            C_local  = T.alloc_fragment((block_M, block_N), accum_dtype) # register accumulator

            T.use_swizzle(panel_size=4, order="col")   # optional: better L2 reuse
            T.clear(C_local)                           # zero the accumulator

            for ko in T.Pipelined(T.ceildiv(K, block_K), num_stages=3):
                T.copy(A[by * block_M, ko * block_K], A_shared)   # global -> shared
                T.copy(B[ko * block_K, bx * block_N], B_shared)
                T.gemm(A_shared, B_shared, C_local)               # tile-level MMA

            for i, j in T.Parallel(block_M, block_N):             # fused ReLU
                C_local[i, j] = T.max(C_local[i, j], 0)

            T.copy(C_local, C[by * block_M, bx * block_N])        # write back

    return matmul_relu_kernel


M = N = K = 1024
kernel = matmul(M, N, K, block_M=128, block_N=128, block_K=64)
a = torch.randn(M, K, device="cuda", dtype=torch.float16)
b = torch.randn(K, N, device="cuda", dtype=torch.float16)
c = torch.empty(M, N, device="cuda", dtype=torch.float16)
kernel(a, b, c)

torch.testing.assert_close(c, torch.relu(a @ b), rtol=1e-2, atol=1e-2)
print("gemm ok")

Here is what each piece is doing.

Three buffers, three levels. A_shared and B_shared live in shared memory; C_local lives in registers. Accumulator in registers, operands staged through shared memory — that's the standard GEMM recipe, except here you write it down. That's the whole difference from Triton in one line.
T.copy is sugar for a parallel copy. It expands into a T.Parallel-style move, and the compiler derives a vectorized, coalesced global→shared transfer from it. When the copy sits inside T.Pipelined, it becomes cp.async automatically.
T.Pipelined(extent, num_stages=N) is a software pipeline. num_stages=3 means triple buffering — while you compute K-tile ko, the loads for ko+1 and ko+2 are already in flight. In Triton, this is a compile flag; here it's just the loop, which is easier to reason about.
T.gemm(A, B, C) is the tile-level matmul. It lowers to CuTe/MMA on NVIDIA and the matching intrinsically on AMD. It also takes transpose_A / transpose_B and a policy=T.GemmWarpPolicy.* that controls how warps split the output tile. Hold onto that policy argument — it's the whole story when we get to MLA.
T.use_swizzle reorders how thread blocks are scheduled so that blocks adjacent in L2 run close together in time. Usually a few percent of free bandwidth.

The figure below maps all of this onto the hardware. It's worth reading against the code, because the labeled spots are exactly where TileLang hands you control that Triton keeps for itself.

A few primitives you'll reach for

You can write most kernels with a small vocabulary.

Allocate: T.alloc_shared, T.alloc_fragment (registers), T.alloc_local.
Move and init: T.copy(src, dst) between any two levels; T.clear, T.fill.
Compute: T.gemm(...); T.Parallel(d0, d1, ...) for elementwise loops (this is the entry point for layout inference); T.reduce_max / T.reduce_sum; scalar math like T.exp, T.exp2, T.max, T.infinity.
Schedule: T.Pipelined(extent, num_stages=), T.use_swizzle(...), T.annotate_layout(...) when you need a specific layout (bank-conflict avoidance, custom swizzle).
Dynamic shape: M = T.dynamic("m") so you don't recompile per shape (it's called T.symbolic in some versions).

Checking your work

Two things you'll want often. To see what the compiler actually emitted:

print(kernel.get_kernel_source())     # generated CUDA / HIP

And to time it:

profiler = kernel.get_profiler(tensor_supply_type=tilelang.TensorSupplyType.Normal)
print(f"latency: {profiler.do_bench()} ms")

T.print(buf) prints a tile from inside the kernel, and the repo's examples/plot_layout draws the memory layout, which is handy when you're chasing a bank conflict or checking a swizzle.

Now a real one: MLA decode

The GEMM shows the mechanics. This next one shows why they matter. We'll walk through DeepSeek's MLA (Multi-Head Latent Attention) decode kernel, because it's the cleanest example of TileLang earning its keep. The TileLang reference lands at roughly FlashMLA's H100 performance (benchmarked at batch 64/128 in fp16, comfortably ahead of Triton and FlashInfer) in about 80 lines of Python. The interesting question is how, because the hard part of MLA isn't the math — it's register pressure.

Let's review the loop everyone knows. Every FlashAttention-family kernel has the same shape. Per query block, you stream over key/value blocks and keep a running max and denominator, so the full score matrix never lands in memory:

# acc_s : [block_M, block_N]  scores for this KV block
# acc_o : [block_M, dim]      output accumulator
for i in range(num_kv_blocks):
    acc_s = Q @ K[i].T
    m_prev       = scores_max
    scores_max   = max(m_prev, rowmax(acc_s))
    scores_scale = exp(m_prev - scores_max)
    acc_o *= scores_scale                       # rescale prior output
    acc_s  = exp(acc_s - scores_max)            # probabilities
    acc_o += acc_s @ V[i]

Both acc_s and acc_o want to stay in registers. For MHA or GQA, that's fine. For MLA, it isn't.

Here's where it gets hard. MLA's head dimensions are big: query and key are 576 wide (a 512-wide "nope" part with no positional encoding, plus a 64-wide "rope" part), and value is 512. So acc_o = [block_M, 512], and it has to stay resident in registers across the whole KV loop.

Now bring in the hardware. On Hopper, the fast path is wgmma.mma_async, which ties 4 warps (128 threads) into one warpgroup and requires a minimum M of 64. So the smallest M one warpgroup can own is 64, which means one warpgroup would be holding a 64 × 512 accumulator. That's too big for a single warpgroup's register file. It spills, and performance falls off a cliff.

The fix is to split the output across two warpgroups. You can't shrink M below 64, so the only axis left is dim. Use two warpgroups: WG0 owns acc_o[:, :256], WG1 owns acc_o[:, 256:]. Now each one holds a 64 × 256 accumulator, which fits. That creates a second problem, though: the P @ V step (with policy=FullCol, each warpgroup producing one column slab of the output) needs the complete acc_s, but in Q @ K each warpgroup only naturally computed half of it. The resolution is a shared-memory swap. During Q @ K, each warpgroup writes its half of acc_s to shared memory and reads back the other warpgroup's half, so afterward both hold the full acc_s and can each compute their slab of acc_o. The diagram above is exactly that: split the scores, swap through S_shared, split the output.

In CuTe you'd hand-write the layouts, the swizzles, the Tensor Core alignment, and the producer/consumer sync to pull this off. The reason it collapses to ~80 lines here is layout inference.

Let's break down what layout inference does. You annotate intent on the T.gemm calls, and it propagates the constraints through the program for you:

policy=FullCol on P @ V means each warpgroup needs the full acc_s, so acc_s = [block_M, block_N].
That propagates back to the staging buffer, so S_shared in T.copy(S_shared, acc_s) is also [block_M, block_N].
And forward into Q @ K: with FullCol, each warpgroup's score slab is [block_M, block_N/2].

The key insight is that you never write any of those shapes. You pick the warp policy and write the math; the shapes, the swizzled layouts, and the warp-specialized producer/consumer code all come out of inference.

The kernel skeleton. In MLA decode the query splits into a "nope" part (Q, dim 512) and a "rope" part (Q_pe, dim 64), and the compressed latent serves as both K and V. So the score is a sum of two GEMMs, and the output is one more. The inner loop looks like this (a representative skeleton, not line-exact — see example_mla_decode.py):

# acc_s = Q_nope @ KV^T + Q_rope @ K_pe^T
T.gemm(Q_shared,    KV_shared,   acc_s, transpose_B=True,
       policy=T.GemmWarpPolicy.FullCol, clear_accum=True)
T.gemm(Q_pe_shared, K_pe_shared, acc_s, transpose_B=True,
       policy=T.GemmWarpPolicy.FullCol)

# online softmax
T.copy(scores_max, scores_max_prev)
T.fill(scores_max, -T.infinity(accum_dtype))
T.reduce_max(acc_s, scores_max, dim=1, clear=False)
# ... exp, rescale acc_o by scores_scale, reduce_sum into logsum ...

# acc_o += P @ V  (V is the same latent KV)
T.copy(acc_s, acc_s_cast)
T.gemm(acc_s_cast, KV_shared, acc_o, policy=T.GemmWarpPolicy.FullCol)

The S_shared exchange between the two warpgroups is the part inference inserts for you, once the FullCol policies force acc_s to be full per warpgroup.

The nice part: the optimizations are one line each. This is where TileLang pays off — the whole performance toolkit is one-liners, and the messy lowering is handled for you.

Threadblock swizzling for L2 reuse: T.use_swizzle(panel_size, order="row").
Shared-memory swizzling for bank conflicts: T.annotate_layout({S_shared: T.layout.make_swizzled_layout(S_shared)}) — XOR-style address remapping so concurrent accesses spread across banks instead of serializing.
Warp specialization: you write a plain script, and it's lowered into a producer warpgroup (TMA loads) plus consumer warpgroups, with all the mbarrier sync generated. None of it shows up in your code.
Pipelining: T.Pipelined(range, num_stages) overlaps loads with compute — more stages, more overlap, but more shared memory, so it's a knob.
Split-KV (FlashDecoding-style): when the batch is small and the SMs are idle, split the KV context across SMs and merge. It's a num_split parameter plus a combine kernel.

So the genuinely hard reasoning — register budget against the M≥64 floor, who owns what across warpgroups, the shared-memory swap — you express by choosing a policy and writing the math. Everything that would be hundreds of fragile lines in CuTe is inference and codegen. That's the pitch, and MLA is where it's most convincing.

One of our own: a drop-in RMSNorm at AtlasCloud

The last example is one of our own production kernels at AtlasCloud, from the Wan video-generation VAE on H100/H200. It's a great illustration of the other thing TileLang is excellent at: covering a config a hand-tuned kernel can't reach, with a clean drop-in.

The setup. We already ship a hand-tuned fused RMSNorm + SiLU kernel. It's fast, and it's compiled for the hidden dims D ∈ {96, 192, 384} that one model config uses. A newer config needs channel widths like {160, 256, 320, 512, 640, 1024}, so on that config the hand-tuned fast path can't run. We wrote a TileLang drop-in to cover exactly that gap.

The TileLang kernel. A drop-in with the same interface (BTHWC in/out, same math, same eps) that supports any C that's a multiple of 32. Two passes, fully coalesced, FP32 accumulator:

@T.prim_func
def main(X:     T.Tensor((M, C), dtype),      # M = B*T*H*W rows
         gamma: T.Tensor((C,),  dtype),
         Y:     T.Tensor((M, C), dtype)):
    with T.Kernel(T.ceildiv(M, BLOCK_M), threads=128) as bm:
        X_chunk = T.alloc_shared((BLOCK_M, BLOCK_C), dtype)
        ss      = T.alloc_fragment((BLOCK_M,), accum_dtype)   # FP32 sum-of-squares
        # pass 1: loop over C in BLOCK_C chunks, accumulate sum of squares in FP32
        # rinv = rsqrt(ss / C + 1e-5)
        # pass 2: re-load X, y = silu(x * gamma * rinv), write back

BLOCK_C is 128/64/32 depending on C, to respect the TMA boxDim ≤ 256 limit, and the FP32 accumulator keeps the sum of squares from overflowing in FP16. Dispatch keeps the hand-tuned path where it works and only falls back when it has to:

_ATLAS_SUPPORTED_D = (96, 192, 384)

def rms_silu_dispatch(x, gamma, out):
    if x.shape[-1] in _ATLAS_SUPPORTED_D:
        atlas_rms_norm_silu(x, gamma, out=out)        # keep the hand-tuned path
    else:
        tilelang_rms_silu_bthwc(x, gamma, out=out)    # cover the gap

What it gained us. All upside, and it's a true drop-in — same interface, same math, same eps, so it slots in behind the existing dispatch with no call-site changes.

What	Gain
Previously-unsupported config	0 → 1 — it runs now (the headline win)
Attention-block RMSNorm vs the eager PyTorch norm it replaced	42 μs → 20 μs (~2× faster)
End-to-end VAE at production resolution (720×1280, 21 frames)	~1.79× encode, ~1.78× decode

The first row is the real point: TileLang let us serve a model config that previously had no fast path at all, without touching the hand-tuned kernel that already works for the other config. One drop-in, written in Python, and a whole model path went from "throws" to "ships."

Where TileLang shines

Fine control over blocking, pipeline stages, and warp partitioning, without writing CUTLASS/CuTe.
Structurally complex, layout-sensitive kernels: GEMM variants, the FlashAttention family, MLA, linear attention, dequant-fused quant GEMM, MoE routing.
Covering an op or config your hand-tuned kernels don't reach (a hidden dim outside the instantiated set, an unusual layout) — and beating an eager fallback while you're there.
One kernel body across backends (NVIDIA / AMD / vendor forks).
The whole optimization toolkit is one call at a time — T.use_swizzle, T.annotate_layout, T.Pipelined, warp specialization, split-KV — with the lowering handled for you.

Wrapping up

The cool part of TileLang is that the hard reasoning stays in your head, not in boilerplate. You decide how to split work across warps, where buffers live, and how deep the pipeline runs — and then layout inference and warp specialization turn that into the register layouts, the swizzles, and the producer/consumer dance that would otherwise be hundreds of lines of CuTe. You pick a policy and write the math. That's the whole pitch, and it's why an 80-line MLA kernel can sit next to a hand-tuned CUTLASS one.

Long video generation blog: How We Shipped SVI in Production

Atlas Cloud — Thu, 07 May 2026 09:38:34 +0000

In Part 1, we surveyed six approaches to long video generation — TTT, LoL, Self Forcing, Self Forcing++, Infinite Talk, and Helios — and landed on SVI as the only path that ships today without retraining a 14B model. This post is about what building with it actually looked like: how the clip-stitching loop works, why Error-Recycling matters, and the production numbers from our first deployment on TurboWan.

The choice: SVI (Stable Video Infinity)

SVI's core philosophy is to turn infinite-length generation into stitching together a finite number of short clips with carefully designed memory transfer. That sounds modest until you realize it cleans up most of the engineering pain points at once: no base-model retraining (a small LoRA mounted on TurboWan), constant VRAM, composable with existing speed-distillation, and official LoRA weights are public.

SVI's mental model. (a) Standard video generative models have a Train-Test Hypothesis Gap — they train on clean inputs but face noisy, error-accumulated inputs at inference. (b) Image restoration models are robust to errors but cannot generate new content. (c) SVI's Error-Recycling Fine-Tuning bridges both — using self-generated errors as supervisory signals so the model actively learns to identify and correct its own generation errors.

How clip stitching works

Each clip is 81 frames (5s @ 16fps). Generation is just a loop: condition the next clip on a global identity anchor and a short-term motion bridge from the previous clip, then concatenate.

Clip 1. Inputs: ref image + empty motion memory. Output: a 5s clip. Extract motion memory: the latent of the last 4 frames. Clip 2. Inputs: ref image + motion memory from clip 1. Output: a 5s clip. Extract motion memory from its tail. ... Repeat for N clips, then concatenate clip 1 + clip 2 + … + clip N into the long video.

The clean part is that no DiT attention modification is needed. Historical context is concatenated at the input level as latents, and a small LoRA teaches the model to actually use that prefix.

Anchor latent. User-provided reference image, encoded by the VAE → keeps subject / character appearance globally consistent. Motion latent. Latent of the last 4 / 8 / 12 frames of the previous clip → tells the model how the last segment ended. Padding. Aligns the input shape so the DiT sees one tidy concatenated sequence: anchor + motion + padding.

Error-Recycling Fine-Tuning

The detail that makes SVI hold up over many clips is how its LoRA is trained. Standard inference always starts denoising from pure Gaussian noise — but in long-video stitching, errors from earlier clips contaminate the conditioning for later clips. If you only ever train on clean reference inputs, you have baked in the train-inference gap.

Standard training: every clip's reference inputs are clean ground truth → the model never sees the kind of noisy historical context it actually faces at inference, and discontinuities accumulate.

Error-Recycling: during training, deliberately inject the model's own past errors into the reference inputs, so the LoRA explicitly learns to operate on noisy historical context. Visual discontinuities at clip boundaries drop sharply.

SVI identifies two core error types. (a) Error-Free Flow Matching is the training-time trajectory. (b) Single-Clip Predictive Error — the per-clip drift between the denoising path and the ideal trajectory. (c) Cross-Clip Conditional Error — error-contaminated reference images cause cascading drift across clips. Error-Recycling explicitly injects both.

SVI training framework. (a) Inject DiT's self-generated errors into the latent space to break the error-free assumption. (b) Efficiently compute bidirectional errors via one-step forward / backward integration. (c) Store errors in a Replay Memory and dynamically resample for reuse, forming a closed-loop error supervision cycle.

SVI separates two error types. Single-clip Predictive Error is the per-clip drift between the denoising path and the ideal trajectory. Cross-clip Conditional Error is the cascading drift caused when error-contaminated reference images flow into the next clip. Error-Recycling injects both, so the LoRA learns explicit error tolerance.

LoRA variants

SVI ships three variants — SVI-Shot for static-image → short-clip, SVI-Dance for human motion (it can also take a pose-sequence input), and SVI-Film for multi-shot / scene-transition long video. Hyperparameters: 81 frames per clip, num_motion_frames ∈ {4, 8, 12}, LoRA rank typically 16–64.

Stacking on TurboWan

We mount SVI's LoRA on top of TurboWan (an speedup version of Wan optimized by Atlas), and we keep our specialized LoRA in the stack for style control. At inference, multiple LoRA weights are superimposed at once.

Base. TurboWan LoRA 1. specialized LoRA — content / style control. LoRA 2. SVI LoRA — long-video consistency. Combined. TurboWan speed + SVI long-video continuity + Spicy style, all in one inference pass.

The full inference flow is straightforward: encode the reference into an anchor latent, concatenate it with the previous clip's motion latent and padding, run TurboWan's denoise, decode, append, and update the motion latent from the tail of the freshly-generated clip. After N iterations, concatenate everything into one video.

1. Encode ref image → anchor latent.

2. y = concat(anchor latent, motion latent, padding).

3. Run TurboWan's 5-step denoise conditioned on y and the text embedding.

4. VAE-decode the clip and append to the output list.

5. Set motion latent = tail (last num_motion_frames) of the just-generated clip.

6. Repeat for N clips, then concatenate all of them.

Some production numbers

Standard test: a single reference image and 3 prompts, generating ~15s output (3 clips × 5s):

Metric	Value
Generated duration	15s (3 clips)
Per-clip inference time	~14s (TurboWan fp8, single GPU)
Total inference time	~42s
Subject consistency	Good

A worked example: Cat Adventure

To make the cross-clip behavior concrete, we ran a 15-second case with one reference and three shots. The style prompt fixed a Pixar look with warm lighting; the character was an orange tabby kitten with big curious eyes; the three shots took it from windowsill, to sidewalk, to meeting a golden retriever, each with its own camera direction.

Clip 1 (0–5s): the orange Pixar kitten on a windowsill, with the camera slowly pulling back from a close-up. Style and character stay stable across frames.

Clip 2 (5–10s) at the transition boundary: the kitten's appearance matches Clip 1, then turns and shifts posture as it jumps down. The motion latent has carried the motion state across the boundary.

Clip 3 (10–15s): a golden retriever is introduced and the scene transitions toward an indoor / outdoor boundary. The kitten's Pixar style remains stable across all three clips.

Aggregate metrics for the run:

Metric	Value
Total duration	15s (3 clips × 5s)
Total frames	240 frames (16fps)
Total inference time	33s (TurboWan, single GPU)
Time-to-video ratio	2.2 s/s
Subject consistency	Pixar orange kitten stable throughout
Clip boundary discontinuity	No obvious jump cuts

That is a 15-second long video in 33 seconds on a single GPU, with cross-clip subject consistency — well within the ≤ 60s wait we set as our target. On a 14-case internal test set, 9 cases came back with no obvious issues (64% pass rate).

The honest closing observation is that in video generation, speed, length, and quality are three corners of an iron triangle. No single approach today leads on all three at once. The interesting work is in choosing which corner you can give up the least, given today's hardware and your training budget. SVI gives up a little length and a little boundary quality — and in exchange we ship long video with Wan2.2-class fidelity, on one GPU, today.

Long video generation blog: Six Approaches, One Decision

Atlas Cloud — Thu, 07 May 2026 09:35:29 +0000

A few months ago we set ourselves a deceptively simple goal: produce coherent, high-quality video longer than 15 seconds, on a single GPU, in well under a minute of wall-clock time. Today's video diffusion models like Wan2.2 are good at 3–5 second clips. Stretching that to 10s, 30s, or a minute is where things get interesting.

This post documents the route we actually took. We surveyed six approaches that show up in recent papers and tech reports — TTT, LoL, Self Forcing, Self Forcing++, Infinite Talk, and Helios — measured the trade-offs, and ultimately landed on SVI (Stable Video Infinity), wired up next to TurboWan in our DiffSynth Engine. We will go over each of those routes, then how SVI works, then the production numbers.

Why long video is hard

Three things break when you push past about five seconds.

The VRAM wall

Wan2.2 uses Full Attention with O(n²) cost in the number of latent tokens. The math is unforgiving:

5s (81 frames): ~32.7k tokens, attention matrix ~10 GB.

10s (165 frames): ~65.5k tokens, attention matrix ~40 GB — already spills off a single GPU.

30s (~500 frames): ~200k tokens, infeasible.

In practice, Self Forcing alone fills most of an H200's 129 GB at 165 frames just for the KV cache.

Temporal drift

Even when memory is fine, three drift modes show up. The Helios paper named them: position shift (subjects wandering across the frame), color shift (gradual hue and brightness drift), and restoration shift (the model overcorrecting and producing visible discontinuities).

Causal consistency

Standard video diffusion uses bidirectional Full Attention — every frame attends to every other. That means no streaming output: you cannot show frame 1 until frame N is done.

Our concrete target was modest: ≥15 second video, smooth visual continuity, stable subjects across the whole clip, total wait under 60 seconds, minimal training, and a strong preference for reusing weights we already have.

The survey

We looked at six families. The names are mostly paper titles; the categories will matter later.

Route 1 · TTT (Test-Time Training)

Paper: One-Minute Video Generation with Test-Time Training (arXiv 2504.05298, Apr 2025).

The idea is to fine-tune the model during inference so it remembers what it has already generated. A small TTT layer (a 2-layer MLP, plus a gate and a local attention) gets inserted after Attention in every Transformer Block, and the model is trained on a curriculum that pushes from short clips out to a full minute.

Per-block insertion: after the standard attention, splice in a Gate, a TTT Layer, and a Local Attention, then a LayerNorm.

Curriculum: train on progressively longer windows — 3s → 9s → 18s → 30s → 60s.

Cost: 256 H100s for ~50 hours.

TTT — left: insertion point (Gate + TTT Layer + Local Attention + LayerNorm, attached after standard Attention via residual). Right: video segmented into 3-second clips, each handled by Local Attention internally, with the TTT Layer carrying global memory across segments.

It works — the paper reaches 1-minute generation. But the training cost is enormous, the experiments only cover CogVideoX 5B (transfer to Wan2.2 14B is unproven), and the inserted TTT layers conflict with the kernel optimizations we already rely on. Verdict: not selected.

Route 2 · LoL (Longer than Longer)

Paper: LoL: Longer than Longer, Scaling Video Generation to Hour (arXiv 2601.16914, Jan 2026).

LoL targets a specific failure mode in autoregressive long video — sink-collapse, where multi-head attention all converges onto the anchor frame and the video periodically reverts to its initial state. The fix is Multi-Head RoPE Jitter: per-head random phase perturbations that break inter-head homogeneity. Training-free, plug-in.

Failure mode: sink-collapse — under autoregressive RoPE, distant frames' positional phases periodically realign with the anchor, attention concentrates, content snaps back to the anchor frame.

Fix: give each attention head its own small random phase shift. Heads can no longer collapse to the same column. No retraining required, drops into existing models.

L2 distance to anchor vs. frame index. Self-Forcing++ (red) and LongLive (blue), both with sink, repeatedly snap back at specific frame positions — those are sink-collapse events where the video reverts to the anchor. LoL's Phase Alignment (green) eliminates the snap-back.

Per-head attention maps. Top row: normal frames — heads have visibly different patterns. Bottom rows: during sink-collapse — every head looks the same, all collapsed onto the anchor frame's column. RoPE Jitter restores per-head diversity.

LoL hits 12-hour video on CogVideoX/HunyuanVideo with little quality loss. The catch is that all the demos are static-ish scenes; we don't know how it survives dance, sports, or anything with strong motion. Plus we'd need to modify Wan2.2's attention. Verdict: adaptation cost is too high for unproven gains on motion content. Not selected.

Route 3 · Self Forcing (Causal Wan2.2)

Paper: Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (arXiv 2506.08009, NeurIPS 2025 Spotlight).

Self Forcing replaces Wan2.2's bidirectional Full Attention with causal attention: a frame only attends to frames before it. That single change unlocks streaming generation — once chunk 1 is done, decode and ship it.

Bidirectional: every frame attends to every other → must finish all 40 denoise steps before any frame can be shown. Causal: a frame only sees its past → the first chunk can stream the moment it is done.

The training trick is what gives the paper its name. Instead of training on clean ground-truth context (Teacher Forcing) or with custom attention masks (Diffusion Forcing), Self Forcing rolls out the actual inference path with a rolling KV cache, so train and inference distributions match.

Generation loop: denoise the next small chunk of frames using DMD's compressed step schedule, conditioned on a rolling KV cache built from already-generated frames.

Stream: as soon as a chunk finishes, VAE-decode and emit it.

Carry-over: push the new chunk's latents into the KV cache for the next chunk to attend to.

Three training paradigms compared: (a) Teacher Forcing trains on clean frames — at inference, noisy frames cause out-of-distribution drift; (b) Diffusion Forcing uses custom attention masks but still has train-inference mismatch; (c) Self Forcing replays the true inference process using a rolling KV cache, fully aligning training and inference.

We measured it on the FastVideo framework, single H200:

Length	Frames	Time	VRAM
5s	81 frames	70s	—
10s	165 frames	168s	129 GB (near capacity)
20s	321 frames	287s	129 GB (KV cache capped at 42 frames)

This is architecturally the cleanest answer, and we genuinely like it. But 10s already saturates an H200's VRAM, quality drops at 165 frames, the original model needs causal-attention fine-tuning, and true streaming also needs a Causal Conv3D in the VAE.

Verdict: wait for the community to chip away at VRAM and quality. Not adopted for now.

Route 4 · Self Forcing++

Paper: Self-Forcing++: Towards Minute-Scale High-Quality Video Generation (arXiv 2510.02283, Oct 2025).

Builds on Self Forcing with three additions: Backward Noise Initialization (each new chunk starts from noise back-integrated from already-generated frames, removing chunk-boundary discontinuities); Extended DMD alignment (slice 5s windows from a long rollout and align them against a teacher's short-window output); and a GRPO stage with optical-flow reward to push for more dynamic motion.

Step 1. Self-rollout the student for far longer than 5 seconds, accumulating a long draft using a rolling KV cache. Step 2. Slice random 5s windows out of that draft, run them through Extended DMD against the teacher's short-window distribution to align. Step 3. Refine with GRPO using optical-flow magnitude as reward, nudging the model toward more dynamic motion. Trick. Each new chunk starts from noise back-integrated from the previous chunk, not from fresh Gaussian — so chunk boundaries no longer pop.

Left to right: CausVid (fixed training duration, train-inference mismatch) → Self Forcing (fixed duration + DMD alignment) → Self-Forcing++ (extended duration + Backward Noise Initialization + Extended DMD alignment). Bottom rows show training-stage and inference-stage correspondence.

Result: minute-scale video (up to about 4m15s) on a 1.3B Wan2.1. Great paper. For production we hit two walls: content is mostly static (low motion), the base model is 1.3B (a long way below Wan2.2 14B), and there is no released code or weights to bootstrap from. Verdict: not selected for now.

Route 5 · Infinite Talk (A2V)

A different shape of problem entirely — Audio-to-Video, where audio drives continuous talking-head generation.

Per-chunk input bundle: the new chunk's noisy latents, the audio features for that time window, the user-provided reference image, the last frame of the previous chunk, and a soft conditioning weight. Reference identity: the reference image keeps long-term appearance stable. Adaptive constraint: the soft weight tightens or relaxes the reference based on similarity drift. Motion bridge: the previous chunk's last frame carries motion across boundaries.

It is good for what it is — talking heads, indefinitely. But the architecture differs enough from Wan2.2 that it requires dedicated training, and it does not generalize to general scenes. Verdict: valuable in a narrow lane, not a general long-video solution.

Route 6 · Helios

Paper: Helios: Real Real-Time Long Video Generation Model (PKU-YuanGroup, arXiv 2603.04379, Mar 2026).

As of writing, Helios is the SOTA for long video — 14B params, 19.5 FPS real-time on a single H100. The trick is to compress historical frames into a three-level pyramid and inject them into the current frame's denoising, so the token budget stays constant no matter how long the video gets.

Multi-Term Memory. Short-term history (last 3 frames) keeps full resolution; mid-term (last 20 frames) gets moderate compression; long-term (everything earlier) gets heavy compression. Total token budget is constant regardless of video length. Guidance Attention. Inside each DiT block, clean historical KVs and noisy current QKVs are processed separately so historical noise cannot contaminate current denoising. Pyramid Sampling. Sample at low resolution first to define structure, then refine to high resolution to add detail — about 2.3× fewer tokens overall.

Helios architecture. Left: Unified History Injection — short / mid / long-term history compressed at different ratios, concatenated with the current frame before entering the DiT. Right: Pyramid Unified Predictor-Corrector — low token count first to define structure, then high token count to refine details, reducing computation by ~2.3×.

The Helios paper systematically defines and visualizes three categories of drift in long-video generation: (a) position shift, (b) color shift, (c) restoration shift (noise), (d) restoration shift (blur). Guidance Attention is specifically designed to address all three.

Helios's measured throughput on H200 is striking — basically flat with length:

Length	Time	Throughput
240 frames (10s)	24s	~10 FPS
480 frames (20s)	42s	~11.4 FPS
960 frames (40s)	82s	~11.7 FPS
H100 single GPU (Helios-Distilled)	—	19.5 FPS

The catch is that Multi-Term Memory Patchification needs full retraining of a 14B model. There are no released weights — only a tech report — so we cannot just bolt on a LoRA. Verdict: a medium-to-long-term direction; not deployable today.

Route Comparison Summary

All six routes side by side, with SVI added as the row we ultimately committed to:

Approach	Max Duration	Quality	Training Required	Engineering Difficulty	Generality	Rec.
TTT	1 minute	High	Heavy training needed	High	Medium	★★☆
LoL	Hour-scale	Medium (static only)	Training needed	Medium	Medium	★★☆
Self Forcing	Theoretically unlimited	Medium (drops > 10s)	Existing model	High (VRAM issues)	High	★★★
Self Forcing++	Minute-scale	Low (mostly static)	Training needed	Very high (no code)	High	★☆☆
Infinite Talk	Unlimited	High (talking head)	Training needed	High	Low (A2V only)	★★☆
Helios	Theoretically unlimited	High (industry SOTA)	Full retraining	Very high (no weights)	High	★★★☆
SVI	Unlimited	Medium-High	Open-source LoRA	Medium	High	★★★★

A taxonomy that fell out of the survey

If you squint, every approach we surveyed falls into one of three buckets.

Type A — extend the attention range itself (Self Forcing, LoL, TTT). Have the model directly process longer sequences. Highest theoretical quality. VRAM grows quadratically, so engineering hits a wall around 10s today.

Type B — hierarchical history compression (Helios). Compress past frames and inject them as conditioning. Bypasses VRAM. Costs a full retraining of a 14B model.

Type C — stateful rolling generation (SVI, Infinite Talk). Decompose long video into short clips with overlapping state. Constant VRAM, unlimited length, LoRA-only training. The trade is possible discontinuities at clip boundaries and unbounded long-term drift you can manage but not eliminate.

For this quarter, Type C is what we ship. For next year, Type B is where we are watching the literature.

In the next post, we go into what shipping actually looked like — six approaches to ≥15-second video generation, why we picked SVI, and what the production numbers look like. Read Part 2 →