Building a client-side physics benchmark for embodied AI in the browser (Rapier/WASM + React Three Fiber)

I'm building VSArena, an open benchmark arena for evaluating embodied AI / VLA (Vision-Language-Action) policies — solo project, still early. This post is about one specific technical decision: running the physics simulation entirely client-side, in the browser, and what that has and hasn't solved.

The problem

LLMs have public, standardized arenas for comparing model quality (LMArena being the obvious example). Embodied AI doesn't have an equivalent yet. Most VLA papers report results on custom setups — different simulators, different tasks, different scoring — that aren't directly comparable to each other.

The goal for VSArena: one narrow, reproducible task (a 4-DOF arm stacking three cubes) that anyone can run and get scored against, without installing a simulator or setting up infrastructure.

Why client-side physics

The simulation runs entirely in the browser: Rapier.js (a Rust physics engine compiled to WASM) for rigid-body dynamics, React Three Fiber for rendering, targeting 60Hz physics steps decoupled from the render loop.

The upside is real: zero server cost to let someone watch or teleop the arm, and it removes the "clone a repo, install a simulator, configure a scene" friction that keeps most people from ever trying a benchmark in the first place.

The obvious problem: client-side physics can be tampered with by whoever runs it locally. That's fine for a spectator/dev demo — it's not fine for a leaderboard anyone is supposed to trust.

The fix: separating "watch" from "score"

Studio (the in-browser demo) is explicitly spectator/dev-only. It never writes to the public leaderboard, and that's labeled directly in the UI — anything run in the browser shows as "seed" or "not client-posted" rather than pretending to be a real result.

Public ELO only comes from a hosted harness: a server-side process that runs the same simulation authoritatively and validates submissions independently of whatever a client claims. That harness is the one piece not live yet — I'd rather ship it and then say "submissions are open" than claim it prematurely.

VLA vs. privileged state

The other separation worth mentioning: there are two tracks. The VLA track gives a policy only a 128x128 RGB camera frame and a language instruction — no object poses. A separate state track gives privileged poses directly, kept only for debugging/baselines, and explicitly does not write to the same public leaderboard. Scoring internally still uses real poses to grade spatial accuracy and task completion — but that's judge-only information, never exposed to the policy on the VLA track.

This distinction matters because a policy that "cheats" by getting exact coordinates isn't actually solving the problem a real VLA model has to solve (perceiving the world through pixels).

What's next

• Hosted harness for public ELO ingestion (server-authoritative scoring)
• Python SDK polish (pip install -e sdk/python works today in dry-run mode)
• A couple more baseline policies beyond the current IK baseline

Repo is public if you want to look at the protocol or the scoring code directly: https://github.com/NovaCoding-G/VSArena (MIT licensed). Feedback on the scoring approach — especially anything client-side physics still doesn't solve — is genuinely welcome.