TL;DR: I took lerobot/pusht — the same dataset used to train Diffusion Policy — injected 50 defective episodes across 5 defect classes (seed=42), kept 156 episodes untouched as controls, and ran RDA (v0.5.4, default thresholds, zero tuning) against it. Result: precision 1.000, recall 0.800 on a strict criterion, and zero false positives on all 156 clean controls. Broad criterion (including review signals): 50/50. Runtime for the full 206-episode audit: 2 seconds. All numbers below are from real runs; the honest caveats are published too.
The problem: benchmarks without ground truth prove nothing
I maintain RDA (Robot Data Audit), an open-source quality-audit tool for robot manipulation datasets. Its README claims 13 metrics, a three-tier verdict system, and a 12-dataset benchmark. But every one of those numbers is an observation on datasets whose ground truth we don't control. When the tool reports 83% idle ratio on a dataset, is that a real defect or just how that robot behaves? Without answers, "audit" is just a nicer word for "aggregate statistics".
The standard answer is a blind test: manufacture the ground truth, then see what the tool actually finds. This post documents the first one.
Why lerobot/pusht
Three reasons, all practical:
- It's famous. PushT is the manipulation dataset from the Diffusion Policy paper, hosted by HuggingFace's official LeRobot org. If a data-quality tool wants credibility, this is a dataset its audience already knows.
- It's standard. LeRobot v3.0 format — parquet + mp4 + episode metadata, exactly the format RDA supports natively.
- It's small. The full dataset is 16 files / 7.7 MB / 206 episodes / 25,650 frames — small enough to audit in full, no sampling shortcuts.
Download integrity was verified byte-for-byte: 25,650 frames = sum of episode lengths = video frame count; all LFS sha256 hashes matched.
The experiment design
Step 1 — Baseline. Audit the untouched dataset first. Result: 43 PASS / 163 REVIEW / 0 EXCLUDE. The 163 reviews all come from idle_ratio — PushT is a low-motion task (median idle ratio 82%), the robot spends most of its time repositioning. That's a property of the data, not a defect, and it matters later.
Step 2 — Inject. 5 defect classes × 10 episodes each (seed=42), leaving 156 episodes untouched as controls:
-
empty— delete all rows from the data parquet, leavemeta/episodesclaiming the episode still exists → expected: EXCLUDE -
nan_state— setobservation.stateto NaN on 15 random frames per episode → expected: EXCLUDE -
timestamp_reverse— reverse the second half of each episode's timestamps → expected: EXCLUDE -
frozen— freeze the entire episode at first-frame values → expected: REVIEW (statistical signal — shouldn't hard-fail) -
duplicate_frames— duplicate 5 random frames per episode, copies appended at the end → honest probe, see caveats
Step 3 — Audit blind. rda audit v0.5.4, default thresholds, no tuning, then compare every verdict against the manifest.
Results
Confusion matrix — strict criterion (EXCLUDE = flagged):
flagged not flagged
defective (50) TP = 40 FN = 10
control (156) FP = 0 TN = 156
Precision 1.000 · Recall 0.800 · Zero false positives on 156 untouched controls — every control episode's verdict is identical to its clean-baseline verdict, so the injection itself didn't perturb anything else.
Broad criterion (EXCLUDE + REVIEW = flagged): 50/50, recall 1.000.
Per class (strict / broad / detector):
-
empty — 10/10 strict · 10/10 broad ·
_zero_frame_guard, a regression probe for a P0 bug we fixed in v0.4.12 (zero-frame episodes used to silently PASS) -
nan_state — 10/10 strict · 10/10 broad ·
invalid_values -
timestamp_reverse — 10/10 strict · 10/10 broad ·
timestamp_validity -
frozen — 0/10 strict · 10/10 broad ·
idle_ratio(effective motion 0%) → REVIEW, by design -
duplicate_frames — 10/10 strict · 10/10 broad ·
timestamp_validity, only because of placement; see caveats
Runtime: 2 seconds for 206 episodes / 24,488 frames on a consumer laptop.
The honest caveats (the part most tool posts skip)
-
duplicate_framesdetection is an artifact of placement. The copies landed at the episode tail, creating 2–4 negative timestamp deltas each. A mid-stream duplicate with monotone timestamps would not be flagged by 0.5.4. I publish this because the class name sounds scarier than what was measured. - Frozen episodes get REVIEW, not EXCLUDE. RDA treats statistical anomalies as review signals, not hard failures. If you believe frozen arms must hard-fail, that's a one-line policy change — the thresholds are open, argue with us in an issue.
-
pusht is an unusually easy payload: 2-D state, 96×96 video, no multi-stream timestamps.
sensor_synchronizationandjoint_limitwere N/A the whole run — this blind test doesn't exercise them. -
121 of the 156 "clean" controls also got REVIEW (all from
idle_ratio). This is exactly why the tool reports distributions instead of bare pass rates — on a task like PushT, a bare pass rate would look terrible for reasons that have nothing to do with data quality.
Why this matters beyond one tool
Robot learning is hitting a data-quality wall: more labs are collecting manipulation data, and "the training set had silently corrupt episodes" is the kind of failure you discover after the policy fails to transfer. Blind-testing your audit tooling against a dataset your audience knows — with published confusion matrices and published weaknesses — is a much better baseline than vendor benchmarks with no ground truth. The full method, manifest schema, and per-episode verdicts are in the repo; the injection tool will ship as rda blindtest in a future release.
Reproduce
pip install robot-data-audit==0.5.4
# download lerobot/pusht (v3.0), apply the 5×10 injection with seed=42
rda audit <blindtest_dataset> --format json
Full experiment doc (tables, per-class detectors, caveats): docs/blind_test_20260831.md · Benchmark: docs/benchmark.md
RDA is MIT-licensed, runs fully locally — your data never leaves the machine.
If you maintain a robot dataset and want it audited (or blind-tested) as a public benchmark entry, open an issue on the repo — the benchmark grows one dataset at a time.
Top comments (0)