Robot learning is getting better very quickly.
We now have better policy architectures, more capable simulation environments, standardized dataset formats such as LeRobot, and an increasing number of public robot manipulation datasets.
But there is a basic question that is surprisingly difficult to answer:
How do we know whether a robot training dataset is actually usable?
A dataset can be perfectly readable and still be a poor training dataset.
It can have valid Parquet files, correct schemas, and complete metadata while containing excessive idle motion, action discontinuities, sampling problems, distribution anomalies, or episodes that deserve human review.
I built RDA (Robot Data Audit) to explore this problem.
I recently ran it across 12 local robot datasets and 4,959 episodes.
Here is what I found.
"Can I Load It?" Is Not the Same as "Can I Train on It?"
Robot datasets can contain:
- camera observations
- joint states
- actions
- timestamps
- task metadata
- robot configuration
- multiple sensor streams
Most data pipelines start with a simple question:
Can I load the dataset?
That's necessary, but it isn't enough.
I think robot data quality needs to be considered in several layers:
| Layer | Question |
|---|---|
| Structural integrity | Does the data actually exist and load correctly? |
| Schema & temporal integrity | Are fields, values, and timestamps valid? |
| Behavioral quality | Does the robot motion contain suspicious patterns? |
| Task relevance | Are those patterns actually harmful for this task? |
RDA currently focuses primarily on the first three layers and produces structured evidence that can be used to investigate the fourth.
This distinction matters.
A statistical anomaly is not automatically a failed demonstration.
And a dataset that passes structural validation is not automatically good training data.
What RDA Actually Measures
RDA is an open-source, local-first auditing tool for LeRobot-format robot manipulation datasets.
The core audit runs locally and does not require a GPU.
The current version checks 13 metrics across three layers.
Layer 1 — Structural Integrity
Deterministic checks include:
- missing frames
- NaN / Inf values
- schema consistency
- timestamp validity
- joint limits
Hard structural failures can result in an EXCLUDE verdict.
Layer 2 — Temporal & Motion Quality
These checks look for statistical risk signals such as:
- sensor synchronization
- sampling jitter
- velocity / acceleration anomalies
- action discontinuity
- temporal sufficiency
These are generally signals for investigation rather than proof that an episode is unusable.
Layer 3 — Dataset Utility
This includes:
- idle ratio
- distribution characteristics
- state-space coverage
These metrics are particularly contextual.
For example, a high idle ratio may be perfectly reasonable for one task and problematic for another.
PASS, REVIEW, EXCLUDE — But Don't Confuse Them with Ground Truth
One design decision became particularly important during development.
RDA separates workflow verdicts from evidence levels.
The workflow verdict is:
PASSREVIEWEXCLUDE
The evidence level is:
HARD_FAILRISK_SIGNALUNVERIFIABLE
These are deliberately different concepts.
For example:
High action discontinuity →
RISK_SIGNAL
does not mean:
The demonstration is corrupted.
Likewise:
Missing frames →
HARD_FAIL
does not necessarily mean:
The entire upstream dataset is bad.
The purpose is to keep an audit tool from turning weak statistical evidence into overly confident labels.
4,959 Episodes: Almost Half Triggered a Review Signal
I ran RDA across 12 local dataset copies.
The overall result was:
| Verdict | Episodes | Percentage |
|---|---|---|
| PASS | 1,711 | 34.5% |
| REVIEW | 2,473 | 49.9% |
| EXCLUDE | 775 | 15.6% |
These numbers are not a confusion matrix.
There was no independent human ground truth for this benchmark, so they cannot be used to claim precision or recall.
The interesting observation is simply:
A large fraction of the tested episodes contained signals worth investigating before blindly feeding them into training.
That alone is useful.
Finding #1: Idle Data Is Everywhere
The median idle ratio across the tested datasets ranged from:
20.8% to 93.3%.
Ten of the twelve datasets had median idle ratios above 65%.
That raises a surprisingly important question:
If a large fraction of your training data represents the robot doing very little, what exactly is the policy learning?
It might be useful temporal context.
Or it might simply teach the model that doing nothing is common.
The answer depends heavily on the task and model architecture.
This led to our first optimization experiment.
Finding #2: The Same Robot Can Have Very Different Data Characteristics
Two datasets from the same xArm platform showed a large difference.
xArm lift
- Median idle ratio: 20.8%
- 767 / 800 episodes received PASS
xArm push
- Median idle ratio: 83.3%
- 562 / 800 episodes received REVIEW
Same robot platform.
Different task.
The important lesson is:
Data quality is not completely independent of task structure.
A high idle ratio is not automatically a collection failure.
It may simply reflect how the task is performed.
That makes universal rules like:
"Delete every episode above 70% idle."
dangerous.
Finding #3: Action Discontinuity May Reveal the Controller
Another interesting pattern appeared in action discontinuity.
Some datasets showed spikes in essentially every episode.
Another xArm dataset had only a handful across 800 episodes.
For example:
- simulated ALOHA datasets showed frequent spikes
- the SO-100 dataset also showed frequent spikes
- xArm lift had only 6 spike-containing episodes out of 800
This suggests that some data-quality signals may tell us as much about the collection or control stack as about the demonstrations themselves.
That could matter for:
- smoothness-regularized policies
- sim-to-real transfer
- controller comparisons
- teleoperation system evaluation
But again:
A discontinuity is a signal, not automatically a failure.
Finding #4: A Dataset Can Be Structurally Clean and Still Need Review
One of the most useful observations from the benchmark was the separation between integrity and behavior.
Outside of the intentionally corrupted ALOHA fixture, the tested datasets were largely clean at the basic integrity layer.
Yet the behavioral layer still flagged many episodes for review.
In other words:
"The files are valid" and "the demonstrations are ideal for training" are different questions.
This seems obvious after writing it down.
In practice, however, many pipelines stop at the first question.
Finding #5: Sometimes the Dataset Copy Is the Problem
One of the most interesting cases was LIBERO.
The local metadata declared:
1,693 episodes
But only:
920 episodes
could actually be read from the local Parquet files.
The remaining:
773 episodes
were missing from the local copy.
An earlier loader implementation could interpret this kind of metadata/data-layout mismatch as zero-frame episodes.
The loader was changed to scan the actual Parquet layout and build a fallback episode index.
The result was more honest:
The local copy is incomplete.
Not:
LIBERO contains 773 empty episodes.
This distinction matters.
A broken local copy is not necessarily a broken upstream dataset.
Then I Compared RDA with Other Tools
I also wanted to understand how different robot-data tooling behaves on deliberately corrupted data.
So I ran a small cross-tool benchmark involving:
- RDA
- trajlens
- ORBIT
- lerobot-doctor
The test included known injected defects.
One result looked like this:
| Defect | RDA | trajlens | ORBIT | lerobot-doctor |
|---|---|---|---|---|
| NaN actions | EXCLUDE | WARN | Crash | FAIL |
| Timestamp reversal | EXCLUDE | FAIL | Crash | FAIL |
| Frozen actions | PASS | — | — | FAIL |
The important result wasn't that RDA "won."
It didn't.
The frozen-action case exposed a real blind spot.
The Frozen-Segment Blind Spot
RDA's current idle_ratio metric is primarily episode-level.
That means a dataset can have a reasonable overall idle ratio while still containing a long locally frozen segment.
Another tool had a dedicated consecutive-identical-actions check and caught it.
This gives RDA a very concrete next feature:
frozen_segment
A good audit tool should be able to identify its own blind spots.
This one is now on our priority list.
Can Data Auditing Actually Improve Training?
Finding suspicious data is only half the problem.
The more interesting question is:
Can audit results tell us what to do with the dataset?
I ran several experiments around idle-data pruning.
The result was much more nuanced than I initially expected.
Moderate Pruning Helped — Aggressive Pruning Hurt
In one experiment:
| Strategy | Data retained | MSE change |
|---|---|---|
| Baseline | 100% | — |
| Keep 50% | 50.1% | -28.9% |
| Keep 40% | 40.1% | -12.5% |
| Keep 30% | 30.1% | -13.3% |
| Keep 20% | 20.0% | +11.5% |
| Keep 10% | 10.0% | +98.2% |
There was a clear dose-response pattern.
Moderate pruning helped in this experiment.
Aggressive pruning eventually became harmful.
So the conclusion is not:
"Remove idle data."
It is:
There may be a useful pruning region, but it depends on how the remaining data is used.
Then We Tested Different Model Architectures
This was probably the most interesting result.
We compared a frame-wise MLP with a temporal Transformer.
The result was almost the opposite.
MLP
Idle-data pruning:
-21.6% MSE
Temporal Transformer
Idle-data pruning:
+95.5% MSE
Why?
Because temporal models need context.
Suppose:
- idle ratio ≈ 70%
- active segments are scattered
- sequence length = 10
After pruning, many active segments become shorter than the required temporal window.
The result:
You removed "low-value" frames and accidentally removed the context required to construct training samples.
There is another issue.
A temporal policy may need to learn the transition:
idle → movement
If all idle context is removed, the model may never see how movement starts.
The Practical Lesson
This changed how I think about automated dataset optimization.
A rule like:
idle_ratio > 70% → delete
is too simplistic.
The better question is:
What model are you training, and what temporal structure does that model need?
The current experiments suggest:
- gentle trimming can be reasonable for frame-wise models
- aggressive idle pruning can be harmful
- temporal models need special handling
- dataset/task domain matters
- audit signals should not automatically become deletion rules
This is why RDA's recommendation layer takes the intended model type into account.
A Local-First Architecture
Robot datasets can contain commercially sensitive information.
They may reveal:
- robot configurations
- manipulation strategies
- factory environments
- camera observations
- proprietary tasks
- operator behavior
RDA's core audit therefore runs locally.
It can generate a blind report containing:
- anonymized paths
- dataset statistics
- episode/frame counts
- metrics
- verdicts
without sending the raw trajectories or images.
The goal is simple:
Share evidence without sharing the dataset.
What RDA Does NOT Claim
This is probably the most important section.
The current experiments do not establish:
- RDA precision
- RDA recall
- universally optimal thresholds
- that every
EXCLUDEepisode should be deleted - that action discontinuity means corruption
- that high idle ratio means useless data
- that RDA improves training success rate
- that RDA certifies compliance with any data-quality standard
The benchmark is based on real local dataset runs, but it does not have independent ground-truth labels.
So the next experiment is obvious:
Get real human or customer QC labels and compare them against RDA without exposing the RDA verdict first.
The Next Experiment: Blind Human Review
The proposed validation workflow is:
- Select 100–500 episodes from one robot platform.
- Generate anonymized samples.
- Have reviewers independently label them:
- KEEP
- REVIEW
- REMOVE
- Do not show them RDA's verdict.
- Preserve reviewer decisions and notes.
- Reveal RDA results only after labeling.
- Compare the two sets of evidence.
That would allow us to measure:
- overlap between RDA
HARD_FAILand humanREMOVE - overlap between RDA
RISK_SIGNALand humanREVIEW - additional defects found by RDA
- reviewer time per episode
- review queue reduction
- potential downstream training impact
Until that experiment is completed, these remain validation targets rather than marketing claims.
Try It
RDA is open source and currently supports LeRobot v2.1 and v3.0.
Install:
pip install robot-data-audit
Audit a dataset:
rda audit /path/to/lerobot/dataset -v
Generate a blind report:
rda audit /path/to/lerobot/dataset --blind --format json
Generate optimization recommendations:
rda recommend /path/to/dataset --policy frame-wise
or:
rda recommend /path/to/dataset --policy temporal
Launch the optional UI:
pip install robot-data-audit[ui]
rda ui
GitHub:
https://github.com/liesliy/rda
PyPI:
https://pypi.org/project/robot-data-audit/
What I Want to Test Next
The next step isn't another synthetic benchmark.
I want to test RDA against real robot data with independent QC labels.
In particular, I'm looking for:
- real-world teleoperation datasets
- datasets with existing QC labels
- datasets containing known failure cases
- datasets used for sim-to-real experiments
- robot companies or research groups willing to run a local audit
The raw data does not need to leave your environment.
A small validation set — even 100–500 episodes from one robot and 1–3 tasks — would already be extremely useful.
If you work with robot manipulation data and are willing to challenge these results, I'd love to compare notes.
Final Takeaway
After auditing thousands of robot episodes, my conclusion is not that robot datasets are "bad."
It is more subtle:
Robot datasets contain many properties that are invisible to a simple "can I load this file?" check.
And those properties can matter differently depending on:
- the robot
- the task
- the controller
- the dataset format
- the policy architecture
- the intended use of the data
The goal of RDA is not to produce another mysterious quality score.
It is to make the path from:
measurement → diagnosis → human review → optimization
more reproducible.
And eventually, hopefully, to make robot data quality something we can discuss with evidence rather than intuition.
Before the GPU bill arrives, audit the data.
Top comments (1)
One thing I deliberately did not include in this article is an accuracy number.
RDA has been tested on real public dataset copies, but we don't yet have independent human/customer ground truth.
So I don't want to claim "X% precision" just because the benchmark looks good.
The next experiment I'm looking for is simple:
100–500 episodes + independent human QC + blind comparison with RDA.
If you're working with real robot manipulation data and already have KEEP / REVIEW / REMOVE labels, I'd be very interested in comparing them.
The raw data can stay completely local.
I'm especially interested in datasets with known failure cases, teleoperation data, or sim-to-real workflows.