DEV Community

Sattyam Jain
Sattyam Jain

Posted on

Verify a signed VLA red-team leaderboard yourself, on CPU, in two commands

#ai #robotics #showdev

Provael publishes a leaderboard of attack success rates against vision-language-action robot policies. You should not trust the page. You should check the signature.

Here is how, on a laptop, with no GPU.

Install 

pip install provael==0.34.0
Enter fullscreen mode Exit fullscreen mode

Python 3.12+. No CUDA needed for the verify path, because verification reads a committed artifact and checks it against its key. Running the attacks yourself needs a GPU. Checking that the published numbers are the ones that were signed does not.

Verify 

provael leaderboard verify
Enter fullscreen mode Exit fullscreen mode

That reads leaderboard/results/leaderboard.json from the installed package, recomputes the digest over the result rows, and checks it against the signing key 8d62aa33ed5162f3. It exits non-zero if the numbers on the page and the numbers under the signature are not the same set.

The board you get at 0.34.0 is schema_version: 5, generated 2026-08-17. Its rows:

  • instruction family: 41.3%, 62 of 150, interval [33.8%, 49.3%]
  • injection: 0 of 50
  • visual: 0 of 100
  • benign baseline: 4.0%, 2 of 50
  • mcp_tool_desc: marked not_applicable, which is a real state and not a zero

That benign baseline row matters more than the rest and I wrote a longer piece about why. Short version: an attack rate with no control is not a measurement.

Run a matched pair without a GPU

The thing that makes the instruction result meaningful is that every adversarial episode has a benign twin at the same task and the same seed. That structure is what McNemar's exact test needs. There is a CPU-only example that shows the pairing without loading a policy checkpoint:

python -m provael.examples.matched_pairs
Enter fullscreen mode Exit fullscreen mode

It builds the pair set, shows which episodes are matched to which, and prints the contingency table the test consumes. You can read the whole thing in about forty lines and it is the part most people get wrong when they roll their own.

What you cannot check this way

Being straight about the limits, because they are the interesting part.

  • This is simulation. SmolVLA on LIBERO libero_object in robosuite on MuJoCo. There are zero real-hardware runs. The sim-to-real protocol has been pre-registered since 24 July 2026 and the arm has not been ordered. results/hardware/ exists and contains no results, deliberately, and the build fails if that count moves while the page still says it has not run.
  • The benign control is uncalibrated. The keep-out box is hand-picked and the same for every task, and it overlaps the reachable benign workspace. Issue #136 is open on exactly this. PROVAEL_REQUIRE_CALIBRATED=1 will hard-error rather than let you take the default silently.
  • Three attack families measure zero. patch, decoy_object and scene_text all sit at 0 of 50 against this policy. Those are results, not gaps, and they are reported as such.
  • The registry holds 28 adversarial attacks across 15 families, and only 3 families have run against a real policy. Reading the registry length as a family count overstates coverage by 14. The README says so.

Submit your own

If you have a policy checkpoint, the submission path is one command:

provael submit --help
Enter fullscreen mode Exit fullscreen mode

It has shipped since 0.32.0 and it is on PyPI. If a page tells you to install from git for this, that page is stale, and one of mine currently does.

Repo is github.com/provael/provael, Apache-2.0. The concept DOI is 10.5281/zenodo.21984184 if you need to cite it.

Top comments (0)