We ran an automated quality audit on two public xArm robot datasets, posted the report as a GitHub Issue, and got a response that changed how we think about data quality metrics.
Here's what happened, and what it means for anyone building robot learning pipelines.
The Setup
RDA (Robot Data Audit) is an open-source tool we built for automated quality auditing of LeRobot-format datasets. It runs 19 diagnostic checks across structural integrity, temporal consistency, trajectory quality, and visual health — and produces a layered report in under a minute.
The latest version, v0.9.12, introduced three new capabilities:
Gap detection in timestamp validation (detecting anomalous time intervals that suggest dropped frames)
Robust sampling jitter calculation (separating gap-induced noise from actual clock instability)
Path B timestamp inference for missing frame detection (when frame indices aren't available)
We wanted to test it on real-world data, so we picked two publicly available xArm datasets on HuggingFace: xarm_push_medium and xarm_lift_medium, each with 800 episodes of roughly 20,000 frames.
What the Audit Found
The structural integrity checks passed cleanly — no missing frames, no NaN values, no schema inconsistencies. The temporal checks also passed. But the trajectory diagnostics told a more interesting story.
xarm_push_medium:
845 action discontinuity spikes across 500 episodes (62.5% affected)
State space occupancy at just 1.7% median — the arm operates in a very narrow region of its joint space
83.3% median idle ratio — only 16.7% of frames show meaningful motion
xarm_lift_medium:
908 extreme acceleration spikes across 439 episodes (54.9% affected)
Similar low occupancy (2.4%) and high idle ratio
At first glance, these numbers look concerning. Hundreds of spikes? 83% idle? Is this data broken?
The Response That Changed Everything
I posted the audit report as Issue #158 on the xArm-Developer/xArm-Python-SDK repository. Three days later, an engineer from UFACTORY responded.
The engineer first point was a correction: those datasets weren't collected by UFACTORY. They came from Nicklas Hansen's TD-MPC paper (arXiv:2203.04955) and were later uploaded to the LeRobot platform.
But the second point was far more valuable:
"If operated with non-interpolated mode, then the spikes would appear easily."
And then he shared UFACTORY's standard practice:
"We normally recommend collecting data with online trajectory planning modes (6 or 7) to utilize the dynamic interpolation and velocity control in our control box. Under those modes, the accelerations would be continuous theoretically."
This was the missing context. The spikes weren't necessarily a data quality problem — they were a characteristic of how the data was collected.
Why This Matters for Metric Design
UFACTORY's feedback highlighted a fundamental tension in data quality tooling: metrics without context can be misleading.
The action_discontinuity metric
RDA detects action discontinuities by computing the second-order difference of the action array, then applying MAD (Median Absolute Deviation) z-score analysis. Spikes are frames where z > 5.0.
The MAD approach is robust — unlike standard deviation, it doesn't get pulled off course by the outliers it's trying to detect. But it's also context-blind. It doesn't know whether the data came from:
Direct teleoperation (no smoothing, every micro-movement recorded)
Online trajectory planning with dynamic interpolation (smooth by design)
A learned policy generating actions autoregressively
The same spike count means very different things depending on the collection mode. In non-interpolated teleoperation, spikes at grasp/release transitions are expected. In mode 6/7 with interpolation, they'd signal something genuinely wrong.
The idle_ratio metric
An 83.3% idle ratio sounds terrible. But in RDA's benchmark across 13 public datasets, 11 had median idle ratios above 63%. High idle ratios are normal for manipulation data.
A push task involves slowly approaching an object, making fine adjustments, waiting for contact — all of which produce tiny action changes that fall below the motion threshold. This isn't noise; it's the task.
The metric's role is observational: report the number, let the user decide based on their model architecture. Frame-level models like Diffusion Policy might want to trim idle frames. Temporal models might use them as useful context.
The velocity_acceleration metric
The 908 extreme acceleration spikes in xarm_lift_medium made sense once we understood the collection method. Without interpolation, rapid start-stop motions during pick-and-place create genuine acceleration discontinuities. With mode 6/7's built-in velocity control, these would be smoothed out.
The metric correctly detected the phenomenon. What it couldn't do — yet — was distinguish between "expected artifact of this collection mode" and "actual data quality problem."
What RDA Does Next
This Issue exchange crystallized three improvements for the next RDA release:
1. The gap_multiplier_used field (v0.9.13)
The gap detection algorithm uses a threshold of median_dt × 2.5. This 2.5 is an internal constant, not exposed to users. But audit reports should be transparent about what thresholds were used. Starting in v0.9.13, the output includes gap_multiplier_used: 2.5 so every report documents exactly how gaps were detected.
2. Control mode metadata
Future reports should include a "collection method" annotation. When action_discontinuity or velocity_acceleration values are reported, they should be interpreted relative to the expected characteristics of that collection mode. A spike rate of 0.04 might be normal for non-interpolated teleoperation but alarming for interpolated trajectories.
3. False Removal Rate research (v0.9.14)
The deeper challenge: how do you distinguish "real problems" from "expected features"? This requires building a labeled dataset of known-good and known-bad data across different collection modes, then measuring how often RDA's diagnostics correctly separate the two.
The Design Philosophy
RDA's architecture separates hard judgments from soft diagnostics:
Integrity Gate (L1) makes binary calls: missing frames, NaN values, time reversals → EXCLUDE. These are objective, verifiable problems.
Trajectory Diagnostics (L2) outputs numbers: spike counts, jitter values, idle ratios. No automatic verdicts.
Dataset Profile (L3) provides context: distributions, coverage, temporal structure.
This separation exists because "is this data good?" is not a question a tool can answer alone. It requires domain knowledge, task context, and model architecture considerations. RDA's job is to make the data's characteristics visible — not to pretend it knows more than it does.
UFACTORY's response validated this approach. If RDA had seen 845 spikes and automatically declared "poor data quality," it would have been wrong. The spikes were a feature of the collection method, not a bug in the data.
Practical Takeaways
If you're collecting xArm data for robot learning:
Use mode 6 or 7 (online trajectory planning) when possible. The control box's dynamic interpolation produces naturally smooth accelerations.
If using non-interpolated mode, expect higher spike counts in RDA reports. This is normal, not necessarily problematic.
Compare episodes using spike_rate (spikes per step), not spike_count. Episode lengths vary, and longer episodes naturally produce more spikes.
Don't panic about high idle ratios. For push-type tasks, 80%+ idle is typical. Whether to trim those frames depends on your model architecture.
Always check the collection metadata alongside audit numbers. Numbers without context are just numbers.
The Bigger Picture
Open-source tools improve through open feedback loops. We built RDA to audit datasets. We used it on public data. We shared the results publicly. And a hardware manufacturer gave us exactly the kind of contextual insight that makes the tool better.
This is how robust data quality tooling gets built — not in isolation, but through the messy, iterative process of applying metrics to real data and learning from domain experts when the numbers don't mean what you thought they meant.
RDA v0.9.12 is available on PyPI. The GitHub repository is at github.com/liesliy/rda.
This article is based on a real GitHub Issue exchange between the RDA team and UFACTORY engineers. The audit report was posted as Issue #158 on the xArm-Developer/xArm-Python-SDK repository.
Top comments (0)