Everyone in embodied AI says "data is the new code." Nobody audits it like code.
We run linters, static analyzers, and CI gates on our source code. Then we feed 50GB of teleoperation recordings into a policy network and hope for the best. So I built RDA (Robot Data Audit) — an open-source CLI that treats robot datasets the way ruff treats a Python repo.
pip install robot-data-audit
rda audit /path/to/lerobot/dataset
It works natively on LeRobot-format datasets (v2.1 + v3.0), checks every episode across two layers — integrity (NaN actions, timestamp reversals, missing frames) and behavior (action discontinuity, idle ratio, frozen segments) — and outputs a per-episode verdict: PASS / REVIEW / EXCLUDE.
The dogfooding surprise
Before releasing it, I ran RDA against the datasets everyone treats as ground truth — starting with aloha_sim_transfer_cube_human, the official ALOHA simulation demonstration set that ships with the LeRobot ecosystem.
Result from the behavior layer, across 50 episodes:
- Action discontinuity spikes: 1,535 (~30 per episode)
- Median idle ratio: 0.71** — the arm is effectively stationary 71% of frames
- Median effective motion ratio: 0.29
To be clear about what this does and doesn't mean: none of this is corruption. The integrity layer came back clean — no NaNs, no broken timestamps. The spikes are real discontinuities in the action space (large frame-to-frame joint jumps), and the high idle ratio likely reflects grasping/hover phases where the gripper holds still. Neither is necessarily a bug in the dataset.
But that's exactly the point. If you're benchmarking a policy on this data, or worse, fine-tuning on it, these numbers are context you didn't have. Is 30 spikes per episode normal for this task? Does 71% idle time skew your loss toward predicting "do nothing"? Nobody asks, because nobody measures.
The bug I found in my own tool (while writing this post)
Honesty section, because dev.to deserves better than marketing:
My first run reported all 50 episodes as PASS. Green across the board. Celebration ensued.
Then I cross-checked the behavior layer output against the verdicts and realized they weren't connected — the metrics were computing 1,535 spikes, and the verdict aggregator was ignoring behavior signals entirely. The tool had the evidence and wasn't reading it. The loudest silence in software is a metric that's computed but never consumed.
Fixed now: behavior signals feed the verdict through a dataset-utility layer, and metric-level findings carry human-readable reasons. The ALOHA run now correctly flags 49/50 episodes as REVIEW with the specific signals attached.
If you're building anything with a "signal producer → decision aggregator" architecture, test the wiring, not just the signals. I wrote a negative-control test for it before I trusted my own tool again.
Why this matters more than it sounds
Robot learning teams are drowning in data collection — teleop sessions, sim rollouts, fleet logs — with almost no tooling for "is this batch usable before I burn GPU hours on it." An episode with a frozen sensor or a corrupted timestamp doesn't fail loudly. It trains quietly.
RDA's philosophy: audit before train. Cheap checks first (seconds per episode, pure numpy/pandas), verdicts you can gate in CI:
rda audit ./my_dataset --format json -o report.json
# fail the pipeline if any episode comes back EXCLUDE
There's also a built-in UI (rda ui) for browsing verdicts without spelunking JSON — and as of v0.5.2 it's fully bilingual: one toggle switches the entire dashboard, backend recommendation copy included, between English and 中文.
What shipped since the first post
-
rda recommend— model-aware optimization advice. Tell it whether you're training a frame-wise model (MLP/BC) or a temporal one (ACT/Diffusion Policy), and it gives different answers for the same data — including an explicit DO_NOT_PRUNE guard for temporal models when valid-window ratio collapses. Every suggestion carries its experimental evidence: pruning cost our seq=10 temporal baseline +296% MSE, while trimmed ALOHA/PushT improved frame-wise baselines by 11–35%. -
Privacy-first architecture — metrics compute locally; only <1KB of aggregates reach the rules API.
rda auditstays 100% offline, always. - LeRobot v2.1 support — bridge-style layouts now load natively (first run on bridge data: median idle ratio 93.3%. Real robots spend a lot of time deciding.).
- A bug class worth naming: silent PASS. 1,690 zero-frame episodes in a popular dataset were passing because "no evidence of problems" was treated as "no problems." Now zero-frame episodes are explicit EXCLUDEs with a diagnosis.
What's next
- More behavior metrics (jitter, cycle anomalies, calibration drift)
- Trend dashboards across successive audits (already in the UI's History page — feedback wanted)
- Export-to-clean: one-click filtered dataset copy from surviving episodes
The project is early and hungry for real-world datasets to chew on. If you have a LeRobot-format dataset (v2.1 or v3.0), run rda audit on it and tell me what turns up — especially if it's boring. Boring results from real data are how a tool earns trust.
Issues, PRs, and "your idle-ratio threshold is wrong, here's why" comments all welcome.
(RDA is MIT-licensed. I also do paid data-quality deep dives and pipeline integration for teams that want the audit without the homework.)
Tool: https://github.com/liesliy/rda · PyPI: robot-data-audit · UI: rda ui (EN/中文)
Top comments (2)
This is a useful failure mode. The bug where the audit computed behavior spikes but the verdict ignored them is exactly why data-quality gates need a test for the gate itself. Otherwise the dataset gets a green badge and the policy pays the bill later.
😅Ha, fair point. And to be fully honest: when I wrote this post, that test didn't exist in the repo. I'd meant to commit it, then forgot somewhere between the debugging session and the write-up.
It's in now: github.com/liesliy/rda/tree/main/t... The idea is basically what you described — feed the classifier metrics that pass every rule but contain known anomalies (e.g. 150 action spikes in an episode, or 95% idle frames), then assert the verdict must come back REVIEW. Plus a boundary case making sure the behavior layer can't soften an EXCLUDE back to REVIEW — the gate shouldn't fail open in either direction.
"A test for the gate itself" is honestly a better one-liner than anything I had. May steal it for the README.