DEV Community

Cover image for What interactive video world models actually do on one GPU (and a tool to check)
Kumar K Jha
Kumar K Jha

Posted on AI-assisted

What interactive video world models actually do on one GPU (and a tool to check)

Interactive video "world models" are having a moment — Matrix-Game, HY-World, and others generate playable worlds frame by frame, conditioned on your actions. Almost every launch quotes a frame rate. Almost none name the hardware behind it.

So I built a small open tool, LiveFrame, that measures what a model actually delivers on the hardware you run it on, and binds every number to a digest ledger anyone can re-verify.

Here's what three published models did when I measured them on a single GPU:

Model Claim (as published) Measured Why they differ
Matrix-Game 2.0 25 fps (1× H100) 19.7 fps same H100 — consistent with the claim
ABot-World-0 16 fps (1× RTX 5090) 5.85 fps same RTX 5090; optional fp8/sage stack off
Oasis-500M 20 fps (proprietary engine) 1.06 fps open weights vanilla (A100 PCIe) — different artifact
HY-World 1.5 24 fps (streaming; unstated) 1.36 fps released pipeline runs batch (8× A100)

This isn't a "gotcha." The vendors aren't wrong — their numbers assume clusters. The point is that the number you can act on depends entirely on the configuration, and that configuration is usually missing from the headline.

Why a single-GPU number matters even if you deploy on a cluster

LiveFrame measures any config — it times frames as they're delivered regardless of how the model uses hardware internally, so you can point it at your 8-GPU setup and it'll time that. I report single-GPU for a practical reason: world models are per-user. One player → one live world. You can't cheaply shard a single interactive stream across a cluster (latency and sync get in the way), so GPUs-per-concurrent-stream is the dominant cost, and single-GPU throughput is roughly your cost per user. The cluster figure tells you it's possible; the per-GPU figure tells you what it costs.

Why you can trust the numbers

Three design choices do the heavy lifting:

  1. It errs low. LiveFrame times frames as they're delivered to the host and counts native decoded frames only — never interpolated or duplicated ones. It structurally cannot inflate a number; if anything it under-reports.
  2. Receipts. Every run writes a self-contained report.html, a machine-readable report.json, and a digest-bound ledger.json. liveframe world verify <run>/ledger.json re-checks every artifact byte — change one byte and verification fails. That's what makes a report you didn't produce trustworthy.
  3. Prove it on CPU first. A synthetic backend ships in the package, so you can exercise the whole pipeline in seconds before touching weights.

How this fits with existing benchmarks

To be clear about prior art: there's a healthy field of world-model benchmarks — VBench, WorldScore, WBench, WorldExam — and they're good at what they do: evaluating quality (controllability, visual fidelity, physics, multi-turn consistency). VBench even reports a single-GPU efficiency number. LiveFrame isn't the first tool to time a model, and it doesn't try to score quality.

What it adds is the complementary piece those benchmarks don't cover: a re-verifiable, bring-your-own-model runtime harness for interactive models — runtime, control, stability, and cost, measured on your config, with a ledger anyone can re-check. It even exports to the worldmark/wbench formats, so it feeds the quality benchmarks rather than competing with them. Quality benchmarks tell you if a model is good; LiveFrame tells you if it's fast enough and what it costs.

Try it in two minutes

pip install "liveframe[world]"

# Prove the loop end to end on CPU (no GPU, no model):
liveframe world run synthetic --suite smoke

# Re-verify any finished run, byte for byte:
liveframe world verify runs/synthetic/smoke/<timestamp>/ledger.json
# -> "verified": true
Enter fullscreen mode Exit fullscreen mode

Bring your own model

Any model becomes a FILE.py:ClassName adapter — five methods and a declared-capabilities object. The harness treats it exactly like a real model: it validates every frame against what you declared, times every chunk, and fails the case (not the run) if the contract breaks.

Here's a complete, runnable stand-in:

# my_world.py
from __future__ import annotations
import time
from collections.abc import Iterator, Sequence
import numpy as np
from liveframe.engine import NativeFrame
from liveframe.world.backend import WorldBackendError, WorldCapabilities

class MyWorld:
    def __init__(self, *, height=144, width=256):
        self._h, self._w = height, width
        self._canvas, self._pos, self._emission, self._loaded = None, np.zeros(2), 0, False

    @property
    def capabilities(self) -> WorldCapabilities:
        return WorldCapabilities(
            name="my-world",
            interaction_mode="interactive",       # step() is called per chunk
            native_fps=16.0,                       # the rate it's DESIGNED to be viewed at
            chunk_frames=4,                        # pixel frames emitted per step()
            resolution=(self._h, self._w),         # exact; any other shape fails the case
            # Declare EVERY action the suites command, or the case fails with a clear error:
            actions=frozenset({"W", "S", "A", "D", "L", "R", "U", "N", "0"}),
            action_magnitudes={"W": 3.0, "S": 3.0, "A": 3.0, "D": 3.0},
            supports_seed=True,
        )

    def load(self):  # load weights here; cold start is reported as context, not generation time
        self._loaded = True

    def reset(self, image, prompt, seed):
        if not self._loaded:
            raise WorldBackendError("reset() before load()")
        self._canvas = np.tile(np.asarray(image, dtype=np.uint8), (3, 3, 1))
        self._pos, self._emission = np.array([float(self._h), float(self._w)]), 0
        self._rng = np.random.default_rng(seed)

    def step(self, actions: Sequence[str]) -> Iterator[NativeFrame]:
        delta = {"W": (-3., 0.), "S": (3., 0.), "A": (0., -3.), "D": (0., 3.),
                 "L": (0., -1.), "R": (0., 1.), "U": (-1., 0.), "N": (1., 0.), "0": (0., 0.)}
        frames = []
        for _ in range(self.capabilities.chunk_frames):
            for token in actions:
                self._pos += delta[token]
            self._pos[0] = float(np.clip(self._pos[0], 0, self._h * 2))
            self._pos[1] = float(np.clip(self._pos[1], 0, self._w * 2))
            r, c = int(self._pos[0]), int(self._pos[1])
            frames.append(np.ascontiguousarray(self._canvas[r:r+self._h, c:c+self._w]))
        # Emit one NativeFrame per step — never buffer and dump at the end.
        yield NativeFrame(index=self._emission, payload=np.stack(frames),
                          native_frame_count=len(frames),
                          metadata={"worker_ts": time.perf_counter()})
        self._emission += 1

    def run_script(self, script):   # batch-mode models implement this; interactive ones raise
        raise WorldBackendError("my-world is interactive; the harness calls step()")

    def close(self):
        self._canvas = None
Enter fullscreen mode Exit fullscreen mode
liveframe world run my_world.py:MyWorld --suite quick
# Or run it inside your model's own environment (its torch/CUDA):
liveframe world run my_world.py:MyWorld --suite quick --python /path/to/model-env/bin/python
Enter fullscreen mode Exit fullscreen mode

The three rules the harness enforces: capabilities must tell the truth (resolution is exact,every commanded action must be declared); step must emit work as it happens (no buffering tolook faster); and native frames only (no interpolation). Break any one and it fails the case with a plain-English reason.

Limitations, up front

  • smoke/quick suites are short, so the real-time verdict reports null on purpose — the real-time rule needs a continuous 60-second stream (the soak suite).
  • MG3/HY ran the short quick suite; HY ran batch + offloaded; MG3 on SDPA, not flash-attn.
  • Forcing cluster-designed models onto one GPU with offloading isn't their intended config, it's a "can it run on commodity hardware" number. The fully fair comparison measures the vendor's config on its cluster and compares cost-per-stream. I haven't funded that yet.

All of this is recorded in the run bundles, not buried.

Backstory

This grew out of a month-long experiment trying to hit real-time diffusion video on a MacBook.
I came up about 11× short and published exactly where the time went instead of hiding it. The arXiv write-up got rejected 🙂 but the useful thing turned out to be the measurement method,so I kept building. Thanks to the Wan, Causal Forcing, Matrix-Game, HY-World, and MLX teams whose work this measures.

Links

I'd really value your feedback. If you run it and find a wrong number or a bad assumption, please let me know, I'd love to hear from you.

Top comments (12)

Collapse
 
analista_83 profile image
Sammi De Blas •

Binding every number to a re-verifiable ledger is the part most benchmarks skip. A frame rate quoted without the artifact, the precision stack and the GPU it ran on is not falsifiable, and the gap between 25 fps on a cluster and 1.36 fps on the released pipeline is exactly where the cost per user lives. Timing delivered native frames and structurally erring low is the right bias: a harness that can only under-report is more useful than one that can round up.

Collapse
 
kkjcodes profile image
Kumar K Jha •

Thank you, you've articulated the core idea better than I did in places. "Not falsifiable" is exactly the gap that bothered me: the numbers aren't necessarily wrong, there's just usually no artifact to check them against.

And yes on erring low, I'd rather publish a floor someone can beat than a ceiling someone has to argue with.

Is there anything that would make this more useful for you? If there's a model you'd like measured, or you want to point it at your own setup, the adapter is ~60 lines and I'm happy to help you get one running or to re-run anything with different settings if you think a config is unfair. Genuinely open to being told where it's wrong or can be improved.

Collapse
 
analista_83 profile image
Sammi De Blas •

What would make the ledger usable is publishing the harness with the numbers. A floor is only a floor if someone can reproduce it on the same artifact, otherwise it is a claim with a signature attached. The cheap version is a pinned commit plus the exact precision flags and the GPU model, so a reader can rerun and compare instead of trusting the table.

Collapse
 
kkjcodes profile image
Kumar K Jha •

Fair standard, and I am glad to report it is almost entirely already met, just not visible enough in the post. The harness itself is the published artifact (pip install "liveframe[world]", source at github.com/kkjcodes/liveframe), and every evidence bundle in the repo carries what you list across both comments: per-case seeds, content-addressed suite digests with the input scenes shipping inside the wheel, the exact GPU model and driver in each report's environment block, model weight identity as HF snapshot commits, and the adapter pins for the model env. The precision flags are the committed adapter source itself. So the rerun path exists today: pip install, point at the same model per the pins, run the same suite, compare medians, verify the ledger.
You did catch one real gap: the report did not record the harness's own version, which it does as of today's commit. One honesty note for anyone who reruns: we measured about 36% variance between two same-model GPU instances, so compare against the stated band rather than the exact digit.

Collapse
 
chiru_d_5f46fa62002963024 profile image
Chiru D •

Loved the article! Tackling the unverified frame-rate claims in interactive world models is a brilliant angle, and building LiveFrame to bring verifiable receipts to single-GPU performance is incredible work. I especially liked how cleanly you positioned it alongside tools like VBench rather than trying to reinvent the wheel. Super crisp writing and a genuinely useful tool—well done!

Collapse
 
analista_83 profile image
Sammi De Blas •

Apologies for the slow reply. I did not rerun the harness myself, so I cannot report install friction first hand. The harness version pin you added today is the piece that makes the ledger comparable across machines.

Collapse
 
analista_83 profile image
Sammi De Blas •

A cheap fix: publish the seed, the input frames and the harness, so anyone can rerun the number on the same GPU.

Collapse
 
kkjcodes profile image
Kumar K Jha •

Answered in the reply above, and thank you for taking the time to dig in.
Did you get a chance to try the install yourself? If anything was rough, from pip to the first smoke run, I would genuinely like to hear it.

Collapse
 
analista_83 profile image
Sammi De Blas •

Apologies for the slow reply. I did not rerun the harness myself, so I cannot report install friction first hand. The harness version pin you added today is the piece that makes the ledger comparable across machines.

Collapse
 
analista_83 profile image
Sammi De Blas •

Sorry for the late reply. The 36 percent variance between two same-model GPU instances is the honest number here. A ledger reporting a band rather than a digit is the one that survives a rerun.

Collapse
 
kkjcodes profile image
Kumar K Jha •

No apology needed, thank you for coming back to it. I agree on the band. Two instances of the same GPU model differed by about 36% on the same run, so a single number would overstate the precision. Reporting a band is the plan from here on. On the version pin, it's on main now and goes out with the next release, so each report will say exactly which harness produced it. If you ever do try the install, I'd really like to hear how it goes.

Collapse
 
analista_83 profile image
Sammi De Blas •

A band plus the harness version in the same receipt makes two runs comparable. Without the pin, the band is just noise.