If you're working on robot learning, you already know the pain: a public dataset looks fine on paper, but after burning weeks of GPU time, your policy won't converge. The problem isn't the model — it's the data.
That's why we built RDA (Robot Data Audit), an open-source tool for auditing LeRobot-format datasets. It runs a four-layer diagnostic pipeline: L1 Integrity Gate for hard verdicts (PASS/REVIEW/EXCLUDE), L2 Trajectory Diagnostics for behavioral observations like action spikes and idle ratios, L3 Dataset Profiling for cross-episode distribution and coverage analysis, and L4 Dataset Summary for aggregated reporting. The design philosophy is separating measurement from judgment — L1 makes hard calls, while L2/L3 lay out the full picture so you can decide.
Open source tools evolve through conversations. Not the polite kind — the ones where someone digs into your code, runs it on real data, and comes back with evidence that something is wrong.
Last week, a developer filed two issues on our GitHub. One on LeRobot (huggingface/lerobot#4650), one on LIBERO (Lifelong-Robot-Learning/LIBERO#148). Both were technically precise, both came with his own reproducible analysis, and both led directly to changes in RDA v0.9.14.
Here's what happened and what we built.
The Action Unit Problem
The developer's first issue pointed out that we had the action unit wrong for the svla_so101_pickplace dataset. We said the values were in "raw STS3215 encoder steps" (0-4096 range). He checked the LeRobot driver layer and found it was already converting raw steps to physical degrees — so the [-100, 100] range was in degrees, not steps.
We went back and verified. He was right. The driver handles the conversion.
This seemed like a simple documentation fix at first. But it exposed a deeper problem: if you don't know what unit your action data is in, you can't meaningfully compare action discontinuity across datasets. One dataset uses degrees, another uses radians, another uses normalized [-1, 1], another uses raw encoder steps. The same spike looks completely different depending on the unit.
Proving Scale Invariance
The developer's intuition was that MAD-based z-scores should be invariant under scale transforms. This is a powerful claim — it would mean the spike detection results don't depend on what unit you measure in, as long as the transforms are linear.
We ran the test properly. We took the same svla action data and applied four different scale transforms:
Degrees (original)
Radians (degrees × π/180)
Raw steps (rescaled to 0-4096 range)
Normalized (rescaled to [-1, 1])
The result: spike_count was 260 in all four cases. Median idle ratio identical across all four.
This confirmed that MAD-based z-score detection is inherently scale-invariant. Whether your actions are in degrees or radians or encoder steps, the spike detection produces the same result. We formalized this as invariant INV-011 and added a guardian test (test_inv011_scale_invariance) to ensure no future code change breaks this property.
Per-Joint Spike Breakdown: Now at the Top Level
The developer didn't just point out the unit issue — he also shared his own per-joint action discontinuity analysis for the svla dataset. He computed spike counts for each joint individually: 347, 1080, 1279, 926, 497, 26.
This was exactly the kind of granular insight that matters for dataset quality analysis. The problem was that RDA already computed per-joint data, but it was buried inside the details layer — you had to dig for it.
In v0.9.14, we moved by_joint to the top-level measurement output. It's now sorted by spike_count descending, with a configurable top_k_joints parameter (default 0 means show all). The old details.by_joint is preserved for backward compatibility.
Running v0.9.14 on the full svla dataset gives per-joint totals of 351, 1075, 1179, 1227, 717, 25 — very close to the developer's independent calculation. The small deltas come from threshold tuning on our side, but the structure matches. That kind of independent reproduction and cross-validation is exactly what makes open source work.
Video Freeze: State Cross-Validation
The second issue (LIBERO#148) was about video freeze detection. The developer made two suggestions that both ended up in v0.9.14.
The first and more impactful one: add a --freeze-motion-source flag that lets the freeze detector cross-validate against state motion data, not just action data.
Previously, RDA's video freeze detection checked whether the robot's actions indicated motion during a visually frozen segment. But some datasets have unreliable action data, or no action data at all, while state data (joint positions) might be more trustworthy.
The new flag has three modes:
action (default): Pure vision-based, unchanged from prior behavior. All existing golden-set results reproduce exactly.
state: Cross-validates each visually-frozen segment against observation.state motion. If state motion exceeds the threshold during the frozen segment, the freeze verdict stays (robot commanded stop but video didn't update — real artifact). If state is also still, the segment downgrades to REVIEW (whole-machine stall, likely a data collection pause rather than a video glitch).
auto: Uses state cross-validation when state data is available, falls back to action mode otherwise.
The metric now reports freeze_motion_source (which mode actually ran) and state_cross_validated_segments (how many segments got downgraded) in the JSON output, so the decision path is transparent.
Thresholds in Frame Equivalents
The second suggestion from LIBERO#148 was simpler but genuinely useful: annotate thresholds in frame counts, not just seconds.
When RDA says "minimum freeze duration: 0.5 seconds," you have to do mental arithmetic to figure out how many frames that is at your dataset's frame rate. With the new version, the JSON and text reports annotate thresholds in frame counts for the configured fps. For example, at 30fps, the default 0.5s threshold shows as "≈15 frames." Small change, but it makes the output immediately readable.
The Bigger Picture
Four changes, all from one person's two issues. The developer clearly uses RDA seriously, understands the detection algorithms, and cares enough to run his own analysis and share the results. That's the best kind of open source feedback loop.
The action unit inference, per-joint exposure, scale invariance verification, and state cross-validation aren't just features — they're responses to real usage patterns from someone doing the same kind of dataset auditing work we're trying to support.
RDA v0.9.14 is on PyPI now: pip install robot-data-audit==0.9.14
GitHub releases: https://github.com/liesliy/rda/releases
If you're auditing robot datasets and have feedback, the issue tracker is open. Come with data.
Top comments (0)