What interactive video world models actually do on one GPU (and a tool to check)

Interactive video "world models" are having a moment — Matrix-Game, HY-World, and others generate playable worlds frame by frame, conditioned on your actions. Almost every launch quotes a frame rate. Almost none name the hardware behind it.

So I built a small open tool, LiveFrame, that measures what a model actually delivers on the hardware you run it on, and binds every number to a digest ledger anyone can re-verify.

Here's what three published models did when I measured them on a single GPU:

Model	Claim (as published)	Measured	Why they differ
Matrix-Game 2.0	25 fps (1× H100)	19.7 fps	same H100 — consistent with the claim
ABot-World-0	16 fps (1× RTX 5090)	5.85 fps	same RTX 5090; optional fp8/sage stack off
Oasis-500M	20 fps (proprietary engine)	1.06 fps	open weights vanilla (A100 PCIe) — different artifact
HY-World 1.5	24 fps (streaming; unstated)	1.36 fps	released pipeline runs batch (8× A100)

This isn't a "gotcha." The vendors aren't wrong — their numbers assume clusters. The point is that the number you can act on depends entirely on the configuration, and that configuration is usually missing from the headline.

Why a single-GPU number matters even if you deploy on a cluster

LiveFrame measures any config — it times frames as they're delivered regardless of how the model uses hardware internally, so you can point it at your 8-GPU setup and it'll time that. I report single-GPU for a practical reason: world models are per-user. One player → one live world. You can't cheaply shard a single interactive stream across a cluster (latency and sync get in the way), so GPUs-per-concurrent-stream is the dominant cost, and single-GPU throughput is roughly your cost per user. The cluster figure tells you it's possible; the per-GPU figure tells you what it costs.

Why you can trust the numbers

Three design choices do the heavy lifting:

1. It errs low. LiveFrame times frames as they're delivered to the host and counts native decoded frames only — never interpolated or duplicated ones. It structurally cannot inflate a number; if anything it under-reports.
2. Receipts. Every run writes a self-contained report.html, a machine-readable report.json, and a digest-bound ledger.json. liveframe world verify <run>/ledger.json re-checks every artifact byte — change one byte and verification fails. That's what makes a report you didn't produce trustworthy.
3. Prove it on CPU first. A synthetic backend ships in the package, so you can exercise the whole pipeline in seconds before touching weights.

How this fits with existing benchmarks

To be clear about prior art: there's a healthy field of world-model benchmarks — VBench, WorldScore, WBench, WorldExam — and they're good at what they do: evaluating quality (controllability, visual fidelity, physics, multi-turn consistency). VBench even reports a single-GPU efficiency number. LiveFrame isn't the first tool to time a model, and it doesn't try to score quality.

What it adds is the complementary piece those benchmarks don't cover: a re-verifiable, bring-your-own-model runtime harness for interactive models — runtime, control, stability, and cost, measured on your config, with a ledger anyone can re-check. It even exports to the worldmark/wbench formats, so it feeds the quality benchmarks rather than competing with them. Quality benchmarks tell you if a model is good; LiveFrame tells you if it's fast enough and what it costs.

Try it in two minutes

pip install "liveframe[world]"

# Prove the loop end to end on CPU (no GPU, no model):
liveframe world run synthetic --suite smoke

# Re-verify any finished run, byte for byte:
liveframe world verify runs/synthetic/smoke/<timestamp>/ledger.json
# -> "verified": true

Bring your own model

Any model becomes a FILE.py:ClassName adapter — five methods and a declared-capabilities object. The harness treats it exactly like a real model: it validates every frame against what you declared, times every chunk, and fails the case (not the run) if the contract breaks.

Here's a complete, runnable stand-in:

# my_world.py
from __future__ import annotations
import time
from collections.abc import Iterator, Sequence
import numpy as np
from liveframe.engine import NativeFrame
from liveframe.world.backend import WorldBackendError, WorldCapabilities

class MyWorld:
    def __init__(self, *, height=144, width=256):
        self._h, self._w = height, width
        self._canvas, self._pos, self._emission, self._loaded = None, np.zeros(2), 0, False

    @property
    def capabilities(self) -> WorldCapabilities:
        return WorldCapabilities(
            name="my-world",
            interaction_mode="interactive",       # step() is called per chunk
            native_fps=16.0,                       # the rate it's DESIGNED to be viewed at
            chunk_frames=4,                        # pixel frames emitted per step()
            resolution=(self._h, self._w),         # exact; any other shape fails the case
            # Declare EVERY action the suites command, or the case fails with a clear error:
            actions=frozenset({"W", "S", "A", "D", "L", "R", "U", "N", "0"}),
            action_magnitudes={"W": 3.0, "S": 3.0, "A": 3.0, "D": 3.0},
            supports_seed=True,
        )

    def load(self):  # load weights here; cold start is reported as context, not generation time
        self._loaded = True

    def reset(self, image, prompt, seed):
        if not self._loaded:
            raise WorldBackendError("reset() before load()")
        self._canvas = np.tile(np.asarray(image, dtype=np.uint8), (3, 3, 1))
        self._pos, self._emission = np.array([float(self._h), float(self._w)]), 0
        self._rng = np.random.default_rng(seed)

    def step(self, actions: Sequence[str]) -> Iterator[NativeFrame]:
        delta = {"W": (-3., 0.), "S": (3., 0.), "A": (0., -3.), "D": (0., 3.),
                 "L": (0., -1.), "R": (0., 1.), "U": (-1., 0.), "N": (1., 0.), "0": (0., 0.)}
        frames = []
        for _ in range(self.capabilities.chunk_frames):
            for token in actions:
                self._pos += delta[token]
            self._pos[0] = float(np.clip(self._pos[0], 0, self._h * 2))
            self._pos[1] = float(np.clip(self._pos[1], 0, self._w * 2))
            r, c = int(self._pos[0]), int(self._pos[1])
            frames.append(np.ascontiguousarray(self._canvas[r:r+self._h, c:c+self._w]))
        # Emit one NativeFrame per step — never buffer and dump at the end.
        yield NativeFrame(index=self._emission, payload=np.stack(frames),
                          native_frame_count=len(frames),
                          metadata={"worker_ts": time.perf_counter()})
        self._emission += 1

    def run_script(self, script):   # batch-mode models implement this; interactive ones raise
        raise WorldBackendError("my-world is interactive; the harness calls step()")

    def close(self):
        self._canvas = None

liveframe world run my_world.py:MyWorld --suite quick
# Or run it inside your model's own environment (its torch/CUDA):
liveframe world run my_world.py:MyWorld --suite quick --python /path/to/model-env/bin/python

The three rules the harness enforces: capabilities must tell the truth (resolution is exact,every commanded action must be declared); step must emit work as it happens (no buffering tolook faster); and native frames only (no interpolation). Break any one and it fails the case with a plain-English reason.

Limitations, up front

• smoke/quick suites are short, so the real-time verdict reports null on purpose — the real-time rule needs a continuous 60-second stream (the soak suite).
• MG3/HY ran the short quick suite; HY ran batch + offloaded; MG3 on SDPA, not flash-attn.
• Forcing cluster-designed models onto one GPU with offloading isn't their intended config, it's a "can it run on commodity hardware" number. The fully fair comparison measures the vendor's config on its cluster and compares cost-per-stream. I haven't funded that yet.

All of this is recorded in the run bundles, not buried.

Backstory

This grew out of a month-long experiment trying to hit real-time diffusion video on a MacBook.
I came up about 11× short and published exactly where the time went instead of hiding it. The arXiv write-up got rejected 🙂 but the useful thing turned out to be the measurement method,so I kept building. Thanks to the Wan, Causal Forcing, Matrix-Game, HY-World, and MLX teams whose work this measures.

Links

• Code: https://github.com/kkjcodes/liveframe
• Quickstart: https://github.com/kkjcodes/liveframe/blob/main/QUICKSTART.md
• Install: pip install "liveframe[world]"

I'd really value your feedback. If you run it and find a wrong number or a bad assumption, please let me know, I'd love to hear from you.

Comments

[Sammi De Blas]

Binding every number to a re-verifiable ledger is the part most benchmarks skip. A frame rate quoted without the artifact, the precision stack and the GPU it ran on is not falsifiable, and the gap between 25 fps on a cluster and 1.36 fps on the released pipeline is exactly where the cost per user lives. Timing delivered native frames and structurally erring low is the right bias: a harness that can only under-report is more useful than one that can round up.

[Kumar K Jha]

Thank you, you've articulated the core idea better than I did in places. "Not falsifiable" is exactly the gap that bothered me: the numbers aren't necessarily wrong, there's just usually no artifact to check them against.

And yes on erring low, I'd rather publish a floor someone can beat than a ceiling someone has to argue with.

Is there anything that would make this more useful for you? If there's a model you'd like measured, or you want to point it at your own setup, the adapter is ~60 lines and I'm happy to help you get one running or to re-run anything with different settings if you think a config is unfair. Genuinely open to being told where it's wrong or can be improved.