Two days ago I pushed a multi-arch docker image for provael, my red-team harness for robot vision-language-action policies. Before that, trying it meant Python, uv, a virtualenv, and a good ten minutes of hoping your platform matched mine.
I did that work because of a number I could not argue with.
There is a census of every LLM-safety benchmark published between Nov 2022 and Nov 2024, 220+ person-hours of hands-on testing, and it found that code which runs with zero modification predicts citation density at p = 0.005. Code that needs any modification at all is statistically indistinguishable from shipping no code. Only 39% of published benchmarks cleared that bar. One in twenty-seven shipped a Dockerfile. Average debug-to-run time was two hours, and of 42 surveyed safety researchers, 32 will not spend more than two hours and none will spend more than six.
So: two hours is the whole budget, and most projects spend it on the install.
Here is the install, on any machine with docker.
Run it
docker run --rm ghcr.io/provael/provael:0.33.0 \
scan --policy stub --suite libero --attack roleplay
That is the entire quickstart. The image is public, multi-arch (amd64 and arm64, so it works on an M-series Mac), and I verified it logged out with a cold cache before writing this.
The stub policy is a deterministic fake policy. It exists so you can see the full pipeline, the attack, the scoring, the report, without downloading model weights. Swap --policy smolvla when you want the real thing.
What comes out
A report with an attack success rate, and next to it the two things an ASR is meaningless without.
{
"asr": 0.5666666666666667,
"attempts": 30,
"successes": 17,
"benign_fpr": 0.0,
"ci95": [0.3919730700081361, 0.7262251442353347],
"family": "instruction",
"policy": "smolvla",
"suite": "libero"
}
benign_fpr is the false positive rate on benign inputs. ci95 is a Wilson interval. If you take one thing from this post and you are publishing attack numbers on anything, take those two fields.
An ASR without a benign control does not tell you whether your attack worked or whether your judge just fires on everything. I ran a control arm against my own headline result last week for exactly that reason, because it was the strongest objection anyone could make and I would rather make it myself. And an ASR without an interval hides the difference between 10 trials and 500. 17 of 30 reads as 57%, and the honest version of 57% is "somewhere between 39% and 73%", which is a much less impressive sentence and a much more useful one.
Pinning, and why the digest matters
The image bases are pinned by digest, not by tag. python:3.12-slim-bookworm gets rebuilt on every CVE patch, so the same Dockerfile produced different bits week to week and a published 0.33.0 was not reproducible from its own source.
One trap worth naming, because I hit it: pin the multi-arch index digest, not a per-architecture manifest digest. Pinning a single arch silently drops arm64, and CI will not catch it because the runner is x86_64. That has broken here once already, in the other direction.
Reproduce a published result
Every result in the repo ships with an execution manifest, so you can re-run the exact thing rather than a thing shaped like it:
docker run --rm -v "$PWD/out:/out" ghcr.io/provael/provael:0.33.0 \
reproduce --manifest results/smolvla_libero_object_control/libero_object_0/execution-manifest.json \
--out /out
Same seeds, same attack set, same scoring. If your numbers differ from mine, that is a bug report I want.
The honest limitations
Simulation only. There are zero real-hardware runs, results/hardware/ is an empty directory that says so out loud, and the SO-ARM101 protocol at docs/studies/sim-to-real-so101.md is pre-registered and not yet run. So nothing here tells you what a physical arm does. That is the next thing I am building and I am not going to pretend otherwise in the meantime.
The repo is Apache-2.0 at github.com/provael/provael. If you run it and it does not work in under two hours, that is the bug I most want to hear about, because per the study above it is the only bug that determines whether anyone uses this.
Top comments (0)