Interactive video "world models" are having a moment — Matrix-Game, HY-World, and others generate playable worlds frame by frame, conditioned on your actions. Almost every launch quotes a frame rate. Almost none name the hardware behind it.
So I built a small open tool, LiveFrame, that measures what a model actually delivers on the hardware you run it on, and binds every number to a digest ledger anyone can re-verify.
Here's what three published models did when I measured them on a single GPU:
| Model | Claim (as published) | Measured | Why they differ |
|---|---|---|---|
| Matrix-Game 2.0 | 25 fps (1× H100) | 19.7 fps | same H100 — consistent with the claim |
| ABot-World-0 | 16 fps (1× RTX 5090) | 5.85 fps | same RTX 5090; optional fp8/sage stack off |
| Oasis-500M | 20 fps (proprietary engine) | 1.06 fps | open weights vanilla (A100 PCIe) — different artifact |
| HY-World 1.5 | 24 fps (streaming; unstated) | 1.36 fps | released pipeline runs batch (8× A100) |
This isn't a "gotcha." The vendors aren't wrong — their numbers assume clusters. The point is that the number you can act on depends entirely on the configuration, and that configuration is usually missing from the headline.
Why a single-GPU number matters even if you deploy on a cluster
LiveFrame measures any config — it times frames as they're delivered regardless of how the model uses hardware internally, so you can point it at your 8-GPU setup and it'll time that. I report single-GPU for a practical reason: world models are per-user. One player → one live world. You can't cheaply shard a single interactive stream across a cluster (latency and sync get in the way), so GPUs-per-concurrent-stream is the dominant cost, and single-GPU throughput is roughly your cost per user. The cluster figure tells you it's possible; the per-GPU figure tells you what it costs.
Why you can trust the numbers
Three design choices do the heavy lifting:
- It errs low. LiveFrame times frames as they're delivered to the host and counts native decoded frames only — never interpolated or duplicated ones. It structurally cannot inflate a number; if anything it under-reports.
-
Receipts. Every run writes a self-contained
report.html, a machine-readablereport.json, and a digest-boundledger.json.liveframe world verify <run>/ledger.jsonre-checks every artifact byte — change one byte and verification fails. That's what makes a report you didn't produce trustworthy. - Prove it on CPU first. A synthetic backend ships in the package, so you can exercise the whole pipeline in seconds before touching weights.
How this fits with existing benchmarks
To be clear about prior art: there's a healthy field of world-model benchmarks — VBench, WorldScore, WBench, WorldExam — and they're good at what they do: evaluating quality (controllability, visual fidelity, physics, multi-turn consistency). VBench even reports a single-GPU efficiency number. LiveFrame isn't the first tool to time a model, and it doesn't try to score quality.
What it adds is the complementary piece those benchmarks don't cover: a re-verifiable, bring-your-own-model runtime harness for interactive models — runtime, control, stability, and cost, measured on your config, with a ledger anyone can re-check. It even exports to the worldmark/wbench formats, so it feeds the quality benchmarks rather than competing with them. Quality benchmarks tell you if a model is good; LiveFrame tells you if it's fast enough and what it costs.
Try it in two minutes
pip install "liveframe[world]"
# Prove the loop end to end on CPU (no GPU, no model):
liveframe world run synthetic --suite smoke
# Re-verify any finished run, byte for byte:
liveframe world verify runs/synthetic/smoke/<timestamp>/ledger.json
# -> "verified": true
Bring your own model
Any model becomes a FILE.py:ClassName adapter — five methods and a declared-capabilities object. The harness treats it exactly like a real model: it validates every frame against what you declared, times every chunk, and fails the case (not the run) if the contract breaks.
Here's a complete, runnable stand-in:
# my_world.py
from __future__ import annotations
import time
from collections.abc import Iterator, Sequence
import numpy as np
from liveframe.engine import NativeFrame
from liveframe.world.backend import WorldBackendError, WorldCapabilities
class MyWorld:
def __init__(self, *, height=144, width=256):
self._h, self._w = height, width
self._canvas, self._pos, self._emission, self._loaded = None, np.zeros(2), 0, False
@property
def capabilities(self) -> WorldCapabilities:
return WorldCapabilities(
name="my-world",
interaction_mode="interactive", # step() is called per chunk
native_fps=16.0, # the rate it's DESIGNED to be viewed at
chunk_frames=4, # pixel frames emitted per step()
resolution=(self._h, self._w), # exact; any other shape fails the case
# Declare EVERY action the suites command, or the case fails with a clear error:
actions=frozenset({"W", "S", "A", "D", "L", "R", "U", "N", "0"}),
action_magnitudes={"W": 3.0, "S": 3.0, "A": 3.0, "D": 3.0},
supports_seed=True,
)
def load(self): # load weights here; cold start is reported as context, not generation time
self._loaded = True
def reset(self, image, prompt, seed):
if not self._loaded:
raise WorldBackendError("reset() before load()")
self._canvas = np.tile(np.asarray(image, dtype=np.uint8), (3, 3, 1))
self._pos, self._emission = np.array([float(self._h), float(self._w)]), 0
self._rng = np.random.default_rng(seed)
def step(self, actions: Sequence[str]) -> Iterator[NativeFrame]:
delta = {"W": (-3., 0.), "S": (3., 0.), "A": (0., -3.), "D": (0., 3.),
"L": (0., -1.), "R": (0., 1.), "U": (-1., 0.), "N": (1., 0.), "0": (0., 0.)}
frames = []
for _ in range(self.capabilities.chunk_frames):
for token in actions:
self._pos += delta[token]
self._pos[0] = float(np.clip(self._pos[0], 0, self._h * 2))
self._pos[1] = float(np.clip(self._pos[1], 0, self._w * 2))
r, c = int(self._pos[0]), int(self._pos[1])
frames.append(np.ascontiguousarray(self._canvas[r:r+self._h, c:c+self._w]))
# Emit one NativeFrame per step — never buffer and dump at the end.
yield NativeFrame(index=self._emission, payload=np.stack(frames),
native_frame_count=len(frames),
metadata={"worker_ts": time.perf_counter()})
self._emission += 1
def run_script(self, script): # batch-mode models implement this; interactive ones raise
raise WorldBackendError("my-world is interactive; the harness calls step()")
def close(self):
self._canvas = None
liveframe world run my_world.py:MyWorld --suite quick
# Or run it inside your model's own environment (its torch/CUDA):
liveframe world run my_world.py:MyWorld --suite quick --python /path/to/model-env/bin/python
The three rules the harness enforces: capabilities must tell the truth (resolution is exact,every commanded action must be declared); step must emit work as it happens (no buffering tolook faster); and native frames only (no interpolation). Break any one and it fails the case with a plain-English reason.
Limitations, up front
-
smoke/quicksuites are short, so the real-time verdict reportsnullon purpose — the real-time rule needs a continuous 60-second stream (thesoaksuite). - MG3/HY ran the short
quicksuite; HY ran batch + offloaded; MG3 on SDPA, not flash-attn. - Forcing cluster-designed models onto one GPU with offloading isn't their intended config, it's a "can it run on commodity hardware" number. The fully fair comparison measures the vendor's config on its cluster and compares cost-per-stream. I haven't funded that yet.
All of this is recorded in the run bundles, not buried.
Backstory
This grew out of a month-long experiment trying to hit real-time diffusion video on a MacBook.
I came up about 11× short and published exactly where the time went instead of hiding it. The arXiv write-up got rejected 🙂 but the useful thing turned out to be the measurement method,so I kept building. Thanks to the Wan, Causal Forcing, Matrix-Game, HY-World, and MLX teams whose work this measures.
Links
- Code: https://github.com/kkjcodes/liveframe
- Quickstart: https://github.com/kkjcodes/liveframe/blob/main/QUICKSTART.md
- Install:
pip install "liveframe[world]"
I'd really value your feedback. If you run it and find a wrong number or a bad assumption, please let me know, I'd love to hear from you.
Top comments (12)
Binding every number to a re-verifiable ledger is the part most benchmarks skip. A frame rate quoted without the artifact, the precision stack and the GPU it ran on is not falsifiable, and the gap between 25 fps on a cluster and 1.36 fps on the released pipeline is exactly where the cost per user lives. Timing delivered native frames and structurally erring low is the right bias: a harness that can only under-report is more useful than one that can round up.
Thank you, you've articulated the core idea better than I did in places. "Not falsifiable" is exactly the gap that bothered me: the numbers aren't necessarily wrong, there's just usually no artifact to check them against.
And yes on erring low, I'd rather publish a floor someone can beat than a ceiling someone has to argue with.
Is there anything that would make this more useful for you? If there's a model you'd like measured, or you want to point it at your own setup, the adapter is ~60 lines and I'm happy to help you get one running or to re-run anything with different settings if you think a config is unfair. Genuinely open to being told where it's wrong or can be improved.
What would make the ledger usable is publishing the harness with the numbers. A floor is only a floor if someone can reproduce it on the same artifact, otherwise it is a claim with a signature attached. The cheap version is a pinned commit plus the exact precision flags and the GPU model, so a reader can rerun and compare instead of trusting the table.
Fair standard, and I am glad to report it is almost entirely already met, just not visible enough in the post. The harness itself is the published artifact (pip install "liveframe[world]", source at github.com/kkjcodes/liveframe), and every evidence bundle in the repo carries what you list across both comments: per-case seeds, content-addressed suite digests with the input scenes shipping inside the wheel, the exact GPU model and driver in each report's environment block, model weight identity as HF snapshot commits, and the adapter pins for the model env. The precision flags are the committed adapter source itself. So the rerun path exists today: pip install, point at the same model per the pins, run the same suite, compare medians, verify the ledger.
You did catch one real gap: the report did not record the harness's own version, which it does as of today's commit. One honesty note for anyone who reruns: we measured about 36% variance between two same-model GPU instances, so compare against the stated band rather than the exact digit.
Loved the article! Tackling the unverified frame-rate claims in interactive world models is a brilliant angle, and building LiveFrame to bring verifiable receipts to single-GPU performance is incredible work. I especially liked how cleanly you positioned it alongside tools like VBench rather than trying to reinvent the wheel. Super crisp writing and a genuinely useful tool—well done!
Apologies for the slow reply. I did not rerun the harness myself, so I cannot report install friction first hand. The harness version pin you added today is the piece that makes the ledger comparable across machines.
A cheap fix: publish the seed, the input frames and the harness, so anyone can rerun the number on the same GPU.
Answered in the reply above, and thank you for taking the time to dig in.
Did you get a chance to try the install yourself? If anything was rough, from pip to the first smoke run, I would genuinely like to hear it.
Apologies for the slow reply. I did not rerun the harness myself, so I cannot report install friction first hand. The harness version pin you added today is the piece that makes the ledger comparable across machines.
Sorry for the late reply. The 36 percent variance between two same-model GPU instances is the honest number here. A ledger reporting a band rather than a digit is the one that survives a rerun.
No apology needed, thank you for coming back to it. I agree on the band. Two instances of the same GPU model differed by about 36% on the same run, so a single number would overstate the precision. Reporting a band is the plan from here on. On the version pin, it's on main now and goes out with the next release, so each report will say exactly which harness produced it. If you ever do try the install, I'd really like to hear how it goes.
A band plus the harness version in the same receipt makes two runs comparable. Without the pin, the band is just noise.