Analyst focused on AI security
Adversarial evaluation of local LLMs, censorship and rejection rate, prompt injection testing, log analysis and defensive automation. Visit: sammideblas.com
Binding every number to a re-verifiable ledger is the part most benchmarks skip. A frame rate quoted without the artifact, the precision stack and the GPU it ran on is not falsifiable, and the gap between 25 fps on a cluster and 1.36 fps on the released pipeline is exactly where the cost per user lives. Timing delivered native frames and structurally erring low is the right bias: a harness that can only under-report is more useful than one that can round up.
Thank you, you've articulated the core idea better than I did in places. "Not falsifiable" is exactly the gap that bothered me: the numbers aren't necessarily wrong, there's just usually no artifact to check them against.
And yes on erring low, I'd rather publish a floor someone can beat than a ceiling someone has to argue with.
Is there anything that would make this more useful for you? If there's a model you'd like measured, or you want to point it at your own setup, the adapter is ~60 lines and I'm happy to help you get one running or to re-run anything with different settings if you think a config is unfair. Genuinely open to being told where it's wrong or can be improved.
Analyst focused on AI security
Adversarial evaluation of local LLMs, censorship and rejection rate, prompt injection testing, log analysis and defensive automation. Visit: sammideblas.com
What would make the ledger usable is publishing the harness with the numbers. A floor is only a floor if someone can reproduce it on the same artifact, otherwise it is a claim with a signature attached. The cheap version is a pinned commit plus the exact precision flags and the GPU model, so a reader can rerun and compare instead of trusting the table.
Fair standard, and I am glad to report it is almost entirely already met, just not visible enough in the post. The harness itself is the published artifact (pip install "liveframe[world]", source at github.com/kkjcodes/liveframe), and every evidence bundle in the repo carries what you list across both comments: per-case seeds, content-addressed suite digests with the input scenes shipping inside the wheel, the exact GPU model and driver in each report's environment block, model weight identity as HF snapshot commits, and the adapter pins for the model env. The precision flags are the committed adapter source itself. So the rerun path exists today: pip install, point at the same model per the pins, run the same suite, compare medians, verify the ledger.
You did catch one real gap: the report did not record the harness's own version, which it does as of today's commit. One honesty note for anyone who reruns: we measured about 36% variance between two same-model GPU instances, so compare against the stated band rather than the exact digit.
Loved the article! Tackling the unverified frame-rate claims in interactive world models is a brilliant angle, and building LiveFrame to bring verifiable receipts to single-GPU performance is incredible work. I especially liked how cleanly you positioned it alongside tools like VBench rather than trying to reinvent the wheel. Super crisp writing and a genuinely useful tool—well done!
Analyst focused on AI security
Adversarial evaluation of local LLMs, censorship and rejection rate, prompt injection testing, log analysis and defensive automation. Visit: sammideblas.com
Apologies for the slow reply. I did not rerun the harness myself, so I cannot report install friction first hand. The harness version pin you added today is the piece that makes the ledger comparable across machines.
Analyst focused on AI security
Adversarial evaluation of local LLMs, censorship and rejection rate, prompt injection testing, log analysis and defensive automation. Visit: sammideblas.com
Answered in the reply above, and thank you for taking the time to dig in.
Did you get a chance to try the install yourself? If anything was rough, from pip to the first smoke run, I would genuinely like to hear it.
Analyst focused on AI security
Adversarial evaluation of local LLMs, censorship and rejection rate, prompt injection testing, log analysis and defensive automation. Visit: sammideblas.com
Apologies for the slow reply. I did not rerun the harness myself, so I cannot report install friction first hand. The harness version pin you added today is the piece that makes the ledger comparable across machines.
Analyst focused on AI security
Adversarial evaluation of local LLMs, censorship and rejection rate, prompt injection testing, log analysis and defensive automation. Visit: sammideblas.com
Sorry for the late reply. The 36 percent variance between two same-model GPU instances is the honest number here. A ledger reporting a band rather than a digit is the one that survives a rerun.
No apology needed, thank you for coming back to it. I agree on the band. Two instances of the same GPU model differed by about 36% on the same run, so a single number would overstate the precision. Reporting a band is the plan from here on. On the version pin, it's on main now and goes out with the next release, so each report will say exactly which harness produced it. If you ever do try the install, I'd really like to hear how it goes.
Analyst focused on AI security
Adversarial evaluation of local LLMs, censorship and rejection rate, prompt injection testing, log analysis and defensive automation. Visit: sammideblas.com
Binding every number to a re-verifiable ledger is the part most benchmarks skip. A frame rate quoted without the artifact, the precision stack and the GPU it ran on is not falsifiable, and the gap between 25 fps on a cluster and 1.36 fps on the released pipeline is exactly where the cost per user lives. Timing delivered native frames and structurally erring low is the right bias: a harness that can only under-report is more useful than one that can round up.
Thank you, you've articulated the core idea better than I did in places. "Not falsifiable" is exactly the gap that bothered me: the numbers aren't necessarily wrong, there's just usually no artifact to check them against.
And yes on erring low, I'd rather publish a floor someone can beat than a ceiling someone has to argue with.
Is there anything that would make this more useful for you? If there's a model you'd like measured, or you want to point it at your own setup, the adapter is ~60 lines and I'm happy to help you get one running or to re-run anything with different settings if you think a config is unfair. Genuinely open to being told where it's wrong or can be improved.
What would make the ledger usable is publishing the harness with the numbers. A floor is only a floor if someone can reproduce it on the same artifact, otherwise it is a claim with a signature attached. The cheap version is a pinned commit plus the exact precision flags and the GPU model, so a reader can rerun and compare instead of trusting the table.
Fair standard, and I am glad to report it is almost entirely already met, just not visible enough in the post. The harness itself is the published artifact (pip install "liveframe[world]", source at github.com/kkjcodes/liveframe), and every evidence bundle in the repo carries what you list across both comments: per-case seeds, content-addressed suite digests with the input scenes shipping inside the wheel, the exact GPU model and driver in each report's environment block, model weight identity as HF snapshot commits, and the adapter pins for the model env. The precision flags are the committed adapter source itself. So the rerun path exists today: pip install, point at the same model per the pins, run the same suite, compare medians, verify the ledger.
You did catch one real gap: the report did not record the harness's own version, which it does as of today's commit. One honesty note for anyone who reruns: we measured about 36% variance between two same-model GPU instances, so compare against the stated band rather than the exact digit.
Loved the article! Tackling the unverified frame-rate claims in interactive world models is a brilliant angle, and building LiveFrame to bring verifiable receipts to single-GPU performance is incredible work. I especially liked how cleanly you positioned it alongside tools like VBench rather than trying to reinvent the wheel. Super crisp writing and a genuinely useful tool—well done!
Apologies for the slow reply. I did not rerun the harness myself, so I cannot report install friction first hand. The harness version pin you added today is the piece that makes the ledger comparable across machines.
A cheap fix: publish the seed, the input frames and the harness, so anyone can rerun the number on the same GPU.
Answered in the reply above, and thank you for taking the time to dig in.
Did you get a chance to try the install yourself? If anything was rough, from pip to the first smoke run, I would genuinely like to hear it.
Apologies for the slow reply. I did not rerun the harness myself, so I cannot report install friction first hand. The harness version pin you added today is the piece that makes the ledger comparable across machines.
Sorry for the late reply. The 36 percent variance between two same-model GPU instances is the honest number here. A ledger reporting a band rather than a digit is the one that survives a rerun.
No apology needed, thank you for coming back to it. I agree on the band. Two instances of the same GPU model differed by about 36% on the same run, so a single number would overstate the precision. Reporting a band is the plan from here on. On the version pin, it's on main now and goes out with the next release, so each report will say exactly which harness produced it. If you ever do try the install, I'd really like to hear how it goes.
A band plus the harness version in the same receipt makes two runs comparable. Without the pin, the band is just noise.