DEV Community: liesliy

I Audited 26 Episodes of a 59,000-Episode Open Robot Dataset. The Data Is Fine — the Metadata Isn't.

liesliy — Fri, 06 Nov 2026 03:18:05 +0000

A few numbers first, because they set up everything else.

There is an organisation called OpenLET publishing real-robot manipulation datasets on AtomGit (a Chinese Git hosting platform). Their headline claim is "over 100,000 minutes of open-source real-robot data". The main repo, LET-Base-Dataset, advertises 60,976 rosbag files and 926.3 hours.

I wanted to use this data. Before writing a single line of training code, I wanted to know where it would break. So I audited it.

Scope, stated up front: 26 episodes from the Kuavo4Pro platform, spanning 25 task directories — car sheet metal loading, SPS parts picking and sorting, iron block tooling, conveyor sorting, grocery loading, microwave placement, folder handover, bottle sorting, shampoo application. Real industrial and retail scenes, not toy tasks. That is still a small sample — 0.05% of the dataset. I therefore split every finding below into two buckets:

Exact verification — computed from the full file tree, independent of sampling. Reproducible to the byte.
Sample observation — describes these 26 episodes only. Not extrapolated to population proportions.

The tool is RDA (robot-data-audit), which I wrote. Treat that as a conflict of interest and check my numbers — every figure below traces back to a per-episode report I still have on disk.

Layer 1: Counting files is cheaper than you think

The first thing worth auditing is never the model. It's the file list.

You can pull the full directory tree without downloading a single binary:

git clone --no-checkout --depth 1 https://atomgit.com/OpenLET/LET-Base-Dataset.git
cd LET-Base-Dataset

git ls-tree -r HEAD --name-only | grep -c '\.bag$'
# 60976  ← matches the README exactly

Good. Now filter out the noise:

git ls-tree -r HEAD --name-only | grep '\.bag$' | grep -vc '/\._'
# 59045

1,931 files begin with ._.

These are macOS AppleDouble resource forks. If you have ever copied a directory to an external drive from a Mac, you have seen them: hidden sidecar files holding Finder metadata, tens to hundreds of bytes each, not rosbag data.

They are committed into the repository and counted as episodes.

Three consequences:

The real episode count is 59,045, not 60,976 — 3.2% lower.
Any script that walks the file list and feeds each path to a rosbag reader will crash on them. Mine did, first run.
Any duration estimate built on file count inherits the 3.2% inflation.

One nuance I only caught on a second pass: those 59,045 include 7,920 simulation bags (four directories under rosbag/sim/). Pure real-robot count is 51,125 (Kuavo4Pro 50,622 + Kuavo5W 503). The volume math later in this post uses the 59,045 denominator; treat it as "all bags, sim included".

The annotation files are clean, for what it's worth:

git ls-tree -r HEAD --name-only | grep -c '\.json$'
# 30746  ← matches the README

Task directory distribution is highly uneven: 82 directories, ranging from 11 to 3,015 episodes, median 316. If you sample uniformly by directory you will massively over-weight the small tasks.

And two more dimensions of composition, also counted exactly from paths:

Dimension	Distribution
Platform	Kuavo4Pro 58,542 (99.1%), Kuavo5W 503
Annotation split	Labelled 28,821 (48.8%), Unlabelled 24,304 (41.2%), three standalone task dirs 5,920

Both matter more than they look:

99% of the data comes from a single robot model. "Open real-robot data" sounds like a multi-embodiment collection; it's one platform at volume. Cross-embodiment transfer can't be validated on this.
41% carries no annotation. That's not a defect — unlabelled data is useful for self-supervised pretraining. But it means every annotation complaint below applies to only half the dataset.

Layer 2: 97.3% of the stated duration is fitted, not measured

The README's 926.3 hours decomposes like this:

2,049 trajectories have durations measured from real timestamps in their annotation JSON.
59,348 have durations inferred from a fitted bag-size ↔ duration relationship.

So 97.3% of the headline duration is an estimate, not a measurement. That's not inherently wrong — bag size and duration are strongly correlated. The problem is that the number gets quoted downstream as if it were measured.

There's also a self-consistency issue:

2,049 + 59,348 = 61,397

But there are only 60,976 bags. 421 too many. Two different counting bases were used and never reconciled.

Then there's the gap to the headline claim:

Repo	Bags	Self-reported duration
LET-Base-Dataset	60,976 (really 59,045)	926.3 h ≈ 55,578 min
LET-Dex-Dataset	2,908	≈20 h ≈ 1,200 min
LET-Body-Dataset	1,093	not published
Total	64,977	≈56,778 min

Publicly verifiable: about 57k minutes. Headline claim: 100k minutes. The missing 43k could be contributed data, unpublished subsets, or the Body repo's undisclosed duration — I have no evidence it doesn't exist. But the verifiable half is half.

I recomputed duration by a completely independent route, trusting no stated number:

Measured bitrate (bytes ÷ real duration across 26 bags): 21.9 MB/s
59,045 episodes × 1.19 GB/episode (mean of sampling pool) ≈ 70.2 TB
70.2 TB ÷ 21.9 MB/s ≈ 53,485 minutes ≈ 891 hours

Within 4% of the README's 926.3 hours. So the 926-hour figure is basically honest. The 100k-minute figure is a different, broader scope. The mistake is mixing them.

(That 70 TB also explains why nobody audits this dataset in full. At my measured 5–13 MB/s, a complete download is 1,500+ hours. Hence 26 episodes.)

Sample durations: median 26.6 s, mean 27.4 s, p10 3.8 s, p90 48.0 s, range 3.1–70.6 s.

One methodological note, because it changes the arithmetic: I sampled smallest episodes first to save bandwidth, so my sample skews short. That's why the extrapolation uses the sampling pool's mean bag size, not the sample's mean duration. Mixing those two up would halve the estimate.

Layer 3: The timestamp problem — and it's not where you'd expect

This is the finding I didn't anticipate, and the most actionable one.

While computing per-topic publish rates, /sensors_data_raw (raw joint stream, nominally 500 Hz) came out anywhere between 466 Hz and 999 Hz across episodes. A 2× spread.

My first assumption was "the nominal rate is wrong". The real answer is more interesting. I compared two timestamps on the same camera stream — the bag record time (when the message hit disk) and the message header stamp (when the camera exposed the frame):

from rosbags.rosbag1 import Reader
from rosbags.typesys import Stores, get_typestore

store = get_typestore(Stores.ROS1_NOETIC)
rec, stamp = [], []
with Reader(path) as reader:
    for conn, t, raw in reader.messages():
        if conn.topic != "/cam_h/color/image_raw/compressed":
            continue
        msg = store.deserialize_ros1(raw, conn.msgtype)
        rec.append(t)
        stamp.append(msg.header.stamp.sec * 10**9 + msg.header.stamp.nanosec)

Result:

record interval   p1  3.87 ms   p50  33.26 ms   p99 170.63 ms
stamp  interval   p1 32.09 ms   p50  33.37 ms   p99  34.65 ms
record − stamp lag    p50 18.41 ms   p99 397.66 ms   max 511.9 ms

The header stamps are immaculate. p1 to p99 spans 32.1–34.7 ms — a rock-steady 30 Hz, drifting by less than a millisecond across 22 seconds.

The record times are a mess — intervals jumping from 3.87 ms to 170 ms, lag ranging 18 ms to 512 ms.

The conclusion is unambiguous:

The camera is sampling cleanly. The rosbag writer is what's jittery. Messages get buffered and flushed in batches, so record time reflects disk scheduling, not physical sampling.

Which means:

Any downstream pipeline that aligns modalities on bag record time is wrong on this dataset, by up to half a second.

Half a second of vision-action misalignment is fatal for imitation learning. Your policy learns "see A, act half a second later" instead of "see A, act".

Now the joint stream. Counting messages per wall-clock second:

Episode A: [1001, 998, 999, 997, 995, 1004, 596, 28, 79]      ← 1000 Hz, then collapse
Episode B: [502, 499, 500, 498, 500, 500, 497, 484, 501, ...] ← 500 Hz, stable for 24 s
Episode C: [1002, 799, 1184, 1000, 963, 41, 672, 704, 1130,
            943, 818, 671, 937, 1089, 993, 953, 604, 887,
            1193, 1729, 1003, 662, 68]                        ← 41 to 1729

Across 26 episodes:

20 episodes ≈ 500 Hz (466–500)
4 episodes ≈ 1000 Hz (991–999)
2 episodes in between (718, 858) — mixed within the episode

Same platform, same nominal spec, 2× difference in actual rate, plus intra-episode switching. That's a capture-side configuration inconsistency, not a measurement artifact.

And all three episodes show the same tail behaviour: the last 0.5–1.5 seconds collapse across every stream simultaneously — counts dropping to single digits per second, like a camera being stopped mid-recording. If that tail is consumed as normal data, multi-modal alignment in the final second is garbage.

Practical takeaway: don't assume a fixed rate. Resample on header.stamp, and drop the last 1.5 s of every episode.

Half the action vector is dead

/kuavo_arm_traj is 14-dimensional (two 7-DoF arms). I measured per-dimension variance:

10 of 26 episodes have only 7 dimensions varying. The other 7 are identically zero for the entire episode.

Cause: these are single-arm tasks. One arm's 7 slots are left empty, which is semantically reasonable. The directory names back this up — car_sheet_metal_square_right_hand, car_sheet_metal_loading_square_QR.

For training, it's a trap:

The model outputs 14 dims, 7 of which are always zero — half the output capacity wasted.
Worse, the model happily learns "emit a constant on half the channels", and validation loss still looks fine.
Which half is dead is not fixed. Checking per episode: 9 of the 10 dead-dim episodes are zeros at indices 0–6, one is zeros at indices 7–13. So this dataset contains both "only arm A moves" and "only arm B moves" variants. A dataloader must never hardcode "the first 7 dims are one arm" — detect per episode by variance.

(An earlier draft of this section claimed the layout also varies within a task category. I checked, and I can't support that: only one of the 25 directories has two samples, and both are 7-dim. Retracted. All I can say is that dimensionality and dead-dim position vary across tasks; within-task consistency needs a larger sample.)

Suggested handling: check per-episode variance at load time, build separate dataloaders by effective dimensionality, or mask the dead dims.

A place where I initially got the causation wrong, worth spelling out. I tried to auto-derive the action↔state mapping by cross-correlation (which action dim corresponds to which of the 28 state dims). 15 of 26 episodes resolved unambiguously; 11 did not.

I assumed the failures were the single-arm episodes — seven all-zero dims give cross-correlation nothing to lock onto. Checking: only 3 of the 11 are 7-dim. Eight are fully 14-dim, and they fail because a few individual dims have near-zero-lag correlation as low as 0.16–0.55 (median correlation is still 0.61–0.97).

Probable cause is tracking lag: action is the commanded target angle, state is the measured angle, and the measured value takes time to catch up. On high-inertia shoulder joints with gravity compensation, that lag is largest, which drags zero-lag correlation down. This is a hypothesis — I did not run a lag sweep to confirm it.

Either way, the downstream implication is the same: don't assume action ≈ next-frame state. On those joints in those 8 episodes, that assumption collapses command and response into one quantity — which is exactly the mapping the policy is supposed to learn.

Annotations carry less signal than the file count suggests

One JSON per episode, ~844 bytes. I parsed 25 of 26 (one episode has no JSON):

Schema is consistent: 12 keys (deviceSn, initSceneText, primaryScene, marks, …)
Median field fill rate: 37.5% — six of ten fields empty on average
1 episode has an empty marks array — no action annotation at all
Only 3 episodes are fully populated

marks is the interesting loss. It should carry the temporal "what is happening now" annotation — the task semantics. Empty means you have video plus trajectory: the model can learn what happens next, not why.

Usable semantics reduce to two levels: the directory name and taskName.

What actually passed

To keep the balance honest: the data itself is clean.

RDA runs 23 checks. All 26 episodes passed.

Check	Failures
Missing frames / dropout	0
Invalid values (NaN / Inf)	0
Schema consistency	0
Timestamp validity	0
Video frame integrity (frames vs parquet rows)	0
Video freeze detection	0
Video–state temporal alignment	0
Numeric sanity / sampling jitter / velocity	0

No dirty data. No corrupt frames. No NaNs. The format is sound. The problems are in the metadata and the time axis, not in the recordings. Both are fixable without re-collecting anything.

Two checks did fail:

Idle ratio: median 16.2%, but 8 episodes exceed 50% and 3 exceed 80%. A lot of standing-still footage. For imitation learning, long static stretches dilute the signal — consider dropping idle segments.
Action discontinuity: 5 episodes show spikes. I have not verified case-by-case whether these are legitimate gripper open/close transitions or genuine capture glitches.

Also worth noting: workspace coverage median 5.7%. That measures how much 3D volume the end effector traverses. 5.7% means motion is highly concentrated. Fine for single-task data; not enough if you want manipulation diversity.

What I could not audit

This matters as much as what I found.

I audited 1 of 6 visual streams. Each bag contains six: head / left wrist / right wrist, each with colour and depth. I converted only head colour, to control download volume. Wrist views and all depth data were not audited.
Joint limit checking never ran. It needs a Kuavo URDF limit table, which the repo doesn't ship. So whether the 14 arm dims exceed physical limits is unknown.
Sensor synchronisation was not checked. rosbag preserves per-topic timestamps; after conversion to LeRobot there is one timeline and the original inter-stream sync is gone. That check can't run on converted data.
Kuavo4Pro only, and Labelled only. All 26 episodes sit under the Labelled split (28,821 episodes total). The 24,304 Unlabelled episodes — 41% of the dataset — were not audited at all. Neither were the 503 Kuavo5W episodes.
26 episodes cannot be extrapolated. I sampled the smallest episodes first (bandwidth). Durations skew short. Do not read population proportions off this.

Six fixes I'd make if this were my dataset

All low-cost, high-leverage:

Delete the ._* files, or gitignore them. One command. Saves every user hours of debugging.
Separate measured from inferred durations in the README. Say which of the 926.3 hours is measured and which is fitted from file size. That's not a mark against the project — it's professionalism.
Ship a stats.json. Real duration, message counts, per-stream rates for every episode. Costs one pass over bag headers to generate; saves each consumer a day of work. I spent most of this audit building exactly that.
Fix the timestamps, or document them. Either write record time from header.stamp at capture, or state plainly in the README: use header.stamp, record time is unreliable. The second option is free.
Fill or mark inactive action dimensions. For single-arm tasks, either write NaN so downstream knows it's missing, or add active_dims: [7..13] to the metadata. Constant zero is the worst option — it's indistinguishable from "this joint genuinely shouldn't move".
Ship the URDF limit table. That single file turns joint-limit checking from a no-op into a real test, and it's the check most worth running.

Reproducing this

pip install robot-data-audit
rda audit /path/to/lerobot_dataset --full

Pulling the data needs no login — the LFS batch API hands out CDN links directly:

curl -s -X POST https://atomgit.com/OpenLET/LET-Base-Dataset.git/info/lfs/objects/batch \
  -H "Accept: application/vnd.git-lfs+json" \
  -H "Content-Type: application/vnd.git-lfs+json" \
  -d '{"operation":"download","transfers":["basic"],
       "objects":[{"oid":"<lfs-oid>","size":<bytes>}]}'

The returned href supports HTTP Range requests, and the file's sha256 equals its LFS oid, so transfer integrity is verifiable per episode.

Raw data is rosbag1; RDA reads LeRobot format only, so a conversion step is required (rosbags + av for H.264 encoding, 10 Hz resampling). Per-episode reports — check results, conversion details, raw stream statistics — are kept locally; ask if you want them for line-by-line verification.

Three caveats, repeated because they matter: the sample is 26 episodes, not a census; only Kuavo4Pro was touched; and RDA is my own tool, so treat its verdicts with appropriate suspicion and verify against the raw numbers.

The data is real, and the capture work is decent. **What this dataset lacks isn't data — it's an honest metadata layer.

How Two GitHub Issues Shaped RDA v0.9.14: Action Units, Per-Joint Analysis, and Smarter Freeze Detection

liesliy — Sun, 20 Sep 2026 08:05:23 +0000

If you're working on robot learning, you already know the pain: a public dataset looks fine on paper, but after burning weeks of GPU time, your policy won't converge. The problem isn't the model — it's the data.

That's why we built RDA (Robot Data Audit), an open-source tool for auditing LeRobot-format datasets. It runs a four-layer diagnostic pipeline: L1 Integrity Gate for hard verdicts (PASS/REVIEW/EXCLUDE), L2 Trajectory Diagnostics for behavioral observations like action spikes and idle ratios, L3 Dataset Profiling for cross-episode distribution and coverage analysis, and L4 Dataset Summary for aggregated reporting. The design philosophy is separating measurement from judgment — L1 makes hard calls, while L2/L3 lay out the full picture so you can decide.

Open source tools evolve through conversations. Not the polite kind — the ones where someone digs into your code, runs it on real data, and comes back with evidence that something is wrong.

Last week, a developer filed two issues on our GitHub. One on LeRobot (huggingface/lerobot#4650), one on LIBERO (Lifelong-Robot-Learning/LIBERO#148). Both were technically precise, both came with his own reproducible analysis, and both led directly to changes in RDA v0.9.14.

Here's what happened and what we built.

The Action Unit Problem

The developer's first issue pointed out that we had the action unit wrong for the svla_so101_pickplace dataset. We said the values were in "raw STS3215 encoder steps" (0-4096 range). He checked the LeRobot driver layer and found it was already converting raw steps to physical degrees — so the [-100, 100] range was in degrees, not steps.

We went back and verified. He was right. The driver handles the conversion.

This seemed like a simple documentation fix at first. But it exposed a deeper problem: if you don't know what unit your action data is in, you can't meaningfully compare action discontinuity across datasets. One dataset uses degrees, another uses radians, another uses normalized [-1, 1], another uses raw encoder steps. The same spike looks completely different depending on the unit.

Proving Scale Invariance

The developer's intuition was that MAD-based z-scores should be invariant under scale transforms. This is a powerful claim — it would mean the spike detection results don't depend on what unit you measure in, as long as the transforms are linear.

We ran the test properly. We took the same svla action data and applied four different scale transforms:

Degrees (original)
Radians (degrees × π/180)
Raw steps (rescaled to 0-4096 range)
Normalized (rescaled to [-1, 1])

The result: spike_count was 260 in all four cases. Median idle ratio identical across all four.

This confirmed that MAD-based z-score detection is inherently scale-invariant. Whether your actions are in degrees or radians or encoder steps, the spike detection produces the same result. We formalized this as invariant INV-011 and added a guardian test (test_inv011_scale_invariance) to ensure no future code change breaks this property.

Per-Joint Spike Breakdown: Now at the Top Level

The developer didn't just point out the unit issue — he also shared his own per-joint action discontinuity analysis for the svla dataset. He computed spike counts for each joint individually: 347, 1080, 1279, 926, 497, 26.

This was exactly the kind of granular insight that matters for dataset quality analysis. The problem was that RDA already computed per-joint data, but it was buried inside the details layer — you had to dig for it.

In v0.9.14, we moved by_joint to the top-level measurement output. It's now sorted by spike_count descending, with a configurable top_k_joints parameter (default 0 means show all). The old details.by_joint is preserved for backward compatibility.

Running v0.9.14 on the full svla dataset gives per-joint totals of 351, 1075, 1179, 1227, 717, 25 — very close to the developer's independent calculation. The small deltas come from threshold tuning on our side, but the structure matches. That kind of independent reproduction and cross-validation is exactly what makes open source work.

Video Freeze: State Cross-Validation

The second issue (LIBERO#148) was about video freeze detection. The developer made two suggestions that both ended up in v0.9.14.

The first and more impactful one: add a --freeze-motion-source flag that lets the freeze detector cross-validate against state motion data, not just action data.

Previously, RDA's video freeze detection checked whether the robot's actions indicated motion during a visually frozen segment. But some datasets have unreliable action data, or no action data at all, while state data (joint positions) might be more trustworthy.

The new flag has three modes:

action (default): Pure vision-based, unchanged from prior behavior. All existing golden-set results reproduce exactly.
state: Cross-validates each visually-frozen segment against observation.state motion. If state motion exceeds the threshold during the frozen segment, the freeze verdict stays (robot commanded stop but video didn't update — real artifact). If state is also still, the segment downgrades to REVIEW (whole-machine stall, likely a data collection pause rather than a video glitch).
auto: Uses state cross-validation when state data is available, falls back to action mode otherwise.

The metric now reports freeze_motion_source (which mode actually ran) and state_cross_validated_segments (how many segments got downgraded) in the JSON output, so the decision path is transparent.

Thresholds in Frame Equivalents

The second suggestion from LIBERO#148 was simpler but genuinely useful: annotate thresholds in frame counts, not just seconds.

When RDA says "minimum freeze duration: 0.5 seconds," you have to do mental arithmetic to figure out how many frames that is at your dataset's frame rate. With the new version, the JSON and text reports annotate thresholds in frame counts for the configured fps. For example, at 30fps, the default 0.5s threshold shows as "≈15 frames." Small change, but it makes the output immediately readable.

The Bigger Picture

Four changes, all from one person's two issues. The developer clearly uses RDA seriously, understands the detection algorithms, and cares enough to run his own analysis and share the results. That's the best kind of open source feedback loop.

The action unit inference, per-joint exposure, scale invariance verification, and state cross-validation aren't just features — they're responses to real usage patterns from someone doing the same kind of dataset auditing work we're trying to support.

RDA v0.9.14 is on PyPI now: pip install robot-data-audit==0.9.14

GitHub releases: https://github.com/liesliy/rda/releases

If you're auditing robot datasets and have feedback, the issue tracker is open. Come with data.

When We Audited Someone Else's Robot Dataset on GitHub, the Hardware Maker Responded

liesliy — Fri, 18 Sep 2026 03:16:29 +0000

We ran an automated quality audit on two public xArm robot datasets, posted the report as a GitHub Issue, and got a response that changed how we think about data quality metrics.

Here's what happened, and what it means for anyone building robot learning pipelines.

The Setup

RDA (Robot Data Audit) is an open-source tool we built for automated quality auditing of LeRobot-format datasets. It runs 19 diagnostic checks across structural integrity, temporal consistency, trajectory quality, and visual health — and produces a layered report in under a minute.

The latest version, v0.9.12, introduced three new capabilities:

Gap detection in timestamp validation (detecting anomalous time intervals that suggest dropped frames)
Robust sampling jitter calculation (separating gap-induced noise from actual clock instability)
Path B timestamp inference for missing frame detection (when frame indices aren't available)

We wanted to test it on real-world data, so we picked two publicly available xArm datasets on HuggingFace: xarm_push_medium and xarm_lift_medium, each with 800 episodes of roughly 20,000 frames.

What the Audit Found

The structural integrity checks passed cleanly — no missing frames, no NaN values, no schema inconsistencies. The temporal checks also passed. But the trajectory diagnostics told a more interesting story.

xarm_push_medium:

845 action discontinuity spikes across 500 episodes (62.5% affected)
State space occupancy at just 1.7% median — the arm operates in a very narrow region of its joint space
83.3% median idle ratio — only 16.7% of frames show meaningful motion

xarm_lift_medium:

908 extreme acceleration spikes across 439 episodes (54.9% affected)
Similar low occupancy (2.4%) and high idle ratio

At first glance, these numbers look concerning. Hundreds of spikes? 83% idle? Is this data broken?

The Response That Changed Everything

I posted the audit report as Issue #158 on the xArm-Developer/xArm-Python-SDK repository. Three days later, an engineer from UFACTORY responded.

The engineer first point was a correction: those datasets weren't collected by UFACTORY. They came from Nicklas Hansen's TD-MPC paper (arXiv:2203.04955) and were later uploaded to the LeRobot platform.

But the second point was far more valuable:

"If operated with non-interpolated mode, then the spikes would appear easily."

And then he shared UFACTORY's standard practice:

"We normally recommend collecting data with online trajectory planning modes (6 or 7) to utilize the dynamic interpolation and velocity control in our control box. Under those modes, the accelerations would be continuous theoretically."

This was the missing context. The spikes weren't necessarily a data quality problem — they were a characteristic of how the data was collected.

Why This Matters for Metric Design

UFACTORY's feedback highlighted a fundamental tension in data quality tooling: metrics without context can be misleading.

The action_discontinuity metric

RDA detects action discontinuities by computing the second-order difference of the action array, then applying MAD (Median Absolute Deviation) z-score analysis. Spikes are frames where z > 5.0.

The MAD approach is robust — unlike standard deviation, it doesn't get pulled off course by the outliers it's trying to detect. But it's also context-blind. It doesn't know whether the data came from:

Direct teleoperation (no smoothing, every micro-movement recorded)
Online trajectory planning with dynamic interpolation (smooth by design)
A learned policy generating actions autoregressively

The same spike count means very different things depending on the collection mode. In non-interpolated teleoperation, spikes at grasp/release transitions are expected. In mode 6/7 with interpolation, they'd signal something genuinely wrong.

The idle_ratio metric

An 83.3% idle ratio sounds terrible. But in RDA's benchmark across 13 public datasets, 11 had median idle ratios above 63%. High idle ratios are normal for manipulation data.

A push task involves slowly approaching an object, making fine adjustments, waiting for contact — all of which produce tiny action changes that fall below the motion threshold. This isn't noise; it's the task.

The metric's role is observational: report the number, let the user decide based on their model architecture. Frame-level models like Diffusion Policy might want to trim idle frames. Temporal models might use them as useful context.

The velocity_acceleration metric

The 908 extreme acceleration spikes in xarm_lift_medium made sense once we understood the collection method. Without interpolation, rapid start-stop motions during pick-and-place create genuine acceleration discontinuities. With mode 6/7's built-in velocity control, these would be smoothed out.

The metric correctly detected the phenomenon. What it couldn't do — yet — was distinguish between "expected artifact of this collection mode" and "actual data quality problem."

What RDA Does Next

This Issue exchange crystallized three improvements for the next RDA release:

1. The gap_multiplier_used field (v0.9.13)

The gap detection algorithm uses a threshold of median_dt × 2.5. This 2.5 is an internal constant, not exposed to users. But audit reports should be transparent about what thresholds were used. Starting in v0.9.13, the output includes gap_multiplier_used: 2.5 so every report documents exactly how gaps were detected.

2. Control mode metadata

Future reports should include a "collection method" annotation. When action_discontinuity or velocity_acceleration values are reported, they should be interpreted relative to the expected characteristics of that collection mode. A spike rate of 0.04 might be normal for non-interpolated teleoperation but alarming for interpolated trajectories.

3. False Removal Rate research (v0.9.14)

The deeper challenge: how do you distinguish "real problems" from "expected features"? This requires building a labeled dataset of known-good and known-bad data across different collection modes, then measuring how often RDA's diagnostics correctly separate the two.

The Design Philosophy

RDA's architecture separates hard judgments from soft diagnostics:

Integrity Gate (L1) makes binary calls: missing frames, NaN values, time reversals → EXCLUDE. These are objective, verifiable problems.
Trajectory Diagnostics (L2) outputs numbers: spike counts, jitter values, idle ratios. No automatic verdicts.
Dataset Profile (L3) provides context: distributions, coverage, temporal structure.

This separation exists because "is this data good?" is not a question a tool can answer alone. It requires domain knowledge, task context, and model architecture considerations. RDA's job is to make the data's characteristics visible — not to pretend it knows more than it does.

UFACTORY's response validated this approach. If RDA had seen 845 spikes and automatically declared "poor data quality," it would have been wrong. The spikes were a feature of the collection method, not a bug in the data.

Practical Takeaways

If you're collecting xArm data for robot learning:

Use mode 6 or 7 (online trajectory planning) when possible. The control box's dynamic interpolation produces naturally smooth accelerations.
If using non-interpolated mode, expect higher spike counts in RDA reports. This is normal, not necessarily problematic.
Compare episodes using spike_rate (spikes per step), not spike_count. Episode lengths vary, and longer episodes naturally produce more spikes.
Don't panic about high idle ratios. For push-type tasks, 80%+ idle is typical. Whether to trim those frames depends on your model architecture.
Always check the collection metadata alongside audit numbers. Numbers without context are just numbers.

The Bigger Picture

Open-source tools improve through open feedback loops. We built RDA to audit datasets. We used it on public data. We shared the results publicly. And a hardware manufacturer gave us exactly the kind of contextual insight that makes the tool better.

This is how robust data quality tooling gets built — not in isolation, but through the messy, iterative process of applying metrics to real data and learning from domain experts when the numbers don't mean what you thought they meant.

RDA v0.9.12 is available on PyPI. The GitHub repository is at github.com/liesliy/rda.

This article is based on a real GitHub Issue exchange between the RDA team and UFACTORY engineers. The audit report was posted as Issue #158 on the xArm-Developer/xArm-Python-SDK repository.

Designing Idle Detection for Robot Datasets: Why We Chose Per-Episode Adaptive Thresholds

liesliy — Thu, 17 Sep 2026 02:19:02 +0000

Recently someone raised a thoughtful question on our GitHub Issue:

Is the 'minimal state change' threshold in RDA using normalized units or raw STS3215 encoder steps (4096 per turn at 30 fps)? At step-level thresholds, slow fine motion near the brick reads as idle.

This question cuts to the core of idle detection design. Let me explain our reasoning and the trade-offs involved.

The Problem

In robot data auditing, idle detection is fundamental but deceptively complex. How do you define "idle" when auditing an episode?

Intuitively, a robot standing still is idle. But what does "still" mean?

Sensor noise causing micro-jitters — is that idle?
Slow approach before grasping — is that idle?
Holding pose while waiting for the next command — is that idle?

There's no universal answer.

Why Not Fixed Thresholds?

Our first attempt used a fixed threshold: ||Δaction|| < 0.01 means idle.

Problem: action scales vary wildly across datasets.

SO-101 with STS3215 servos: action range 0-4096 (encoder steps)
Franka Panda: normalized [-1, 1]
ALOHA: different scale again

A fixed threshold works for one dataset but fails catastrophically on another.

Normalization seems like the obvious fix, but normalizing to what? Each dataset has its own action distribution characteristics. Unified normalization loses dataset-specific information.

The Bimodal Gap Detection Approach

We realized that each episode has its own "motion distribution" .

A typical manipulation episode contains two types of frames:

Idle frames: robot stationary or micro-jittering, small action changes
Active frames: robot moving, significant action changes

These two types usually form a bimodal distribution in action change magnitudes.

Our algorithm:

Compute motion: ||Δaction|| (L2 norm of action first-difference)
Build a histogram (30 bins)
Find the valley between the "low cluster" (idle) and "high cluster" (active)
The valley position becomes the threshold

If no clear bimodal structure exists (e.g., all frames are slow movements), we fall back to 3 × MAD (Median Absolute Deviation) as a conservative estimate.

Answering the Issue Question

RDA's idle threshold operates in raw action space units, not normalized.

For SO-101, this means the threshold is at the encoder step level. But the threshold is computed per-episode adaptively, not a fixed global constant.

Benefits:

Preserves dataset-specific action scale characteristics
Each episode adjusts its own threshold based on its motion distribution
No cross-dataset normalization assumptions required

The cost?

Exactly what the Issue identified: slow fine motions can be misclassified as idle.

If a task involves entirely slow, precise operations (e.g., precision assembly), the action change magnitudes are uniformly small, the bimodal distribution may not exist, and the threshold becomes very small — causing many "meaningful but slow" motions to be classified as idle.

This is a real limitation, not a bug.

The Design Trade-off

We considered four approaches:

Fixed threshold — Simple and interpretable, but not comparable across datasets. A threshold that works for SO-101 fails on Franka.

Normalized threshold — Cross-dataset comparable, but loses dataset-specific characteristics. Normalizing to what? Each dataset has its own action distribution.

Per-episode adaptive (what we chose) — Adapts to each episode's motion distribution. The downside: slow tasks may have their fine motions misclassified as idle.

Learned threshold — Theoretically optimal, but requires labeled data and generalization is unknown.

We chose per-episode adaptive because:

RDA is a diagnostic tool, not a pass/fail gate. We output idle_ratio as a measurement, not a judgment. Users interpret the value based on their task characteristics.
Per-episode adaptive works well for most manipulation tasks. Typical pick-and-place, insertion tasks have clear bimodal distributions.
For special tasks (entirely slow operations), users should be aware that idle_ratio may be high. This itself is valuable diagnostic information.

What About Slow Tasks?

The Issue questioner's concern about "slow fine motion near the brick" is a real scenario.

If the task itself is like this, we suggest:

Don't just look at the absolute idle_ratio value. Compare idle_ratio across episodes of the same task — relative differences are more meaningful.
Look at action discontinuity alongside idle ratio. If idle_ratio is high AND action discontinuity is high, the data may have structural issues (e.g., control mode switching during teleoperation). If both are normal, it's likely just the task's inherent characteristic.
**Consider task-specific threshold tuning. **RDA's idle detection parameters are configurable (mad_multiplier, abs_threshold_floor), adjustable based on actual data.

Action Discontinuity: Another Perspective

Beyond idle detection, RDA also detects spikes in the action signal — sudden jumps that may indicate:

Control mode switching during teleoperation
Frame drops or interpolation issues in data collection
High-frequency controller oscillation

Spikes are detected on the second difference of the action signal (Δ²a), using MAD-based z-scores: z = 0.6745 × (x - median) / MAD, with a default threshold of |z| > 5.0.

This is also in raw action units, per-episode adaptive.

In our G1 dataset audit (300 episodes, 177,811 frames), 299/300 episodes had spikes (3,340 total). Combined with the 65.6% median idle ratio, this suggests the teleoperation data collection process has inherent characteristics worth understanding rather than "fixing."

Reflections

There's no universal "idle" definition.Different tasks, different hardware, different data collection methods — the understanding of "idle" varies. Tools can provide measurements, not judgments.
Adaptive thresholds come at the cost of uncertainty in edge cases.Bimodal detection works well on data with clear "still vs. moving" separation. It fails on "entirely slow" data. Users need to understand this boundary.
Open-source community value lies in these real discussions.The Issue made us re-examine idle detection's limitations and prompted us to think about how to better communicate applicable scenarios in documentation.

RDA is currently v0.9.12, and idle detection design is still iterating. If you have datasets with slow, precise operations, we welcome you to test it and discuss on GitHub.
Repository: github.com/liesliy/rda
Related Issue: huggingface/lerobot#4650

Detecting Frozen Video in Robot Datasets: Why First/Last Frame Comparison Isn't Enough

liesliy — Wed, 16 Sep 2026 02:22:17 +0000

While auditing robotics learning datasets, we stumbled onto a problem that seemed simple but turned out to have real depth: detecting frozen video frames.

The naive approach is obvious — compare the first and last frame of each episode. If they're identical, the video is frozen. This is what some popular tools do. But after testing it on real datasets, we found it misses the cases that actually matter.

What We Found in LIBERO

We ran our open-source audit tool RDA against the LIBERO benchmark — 10 manipulation tasks, 379 episodes total. The audit flagged 6 episodes with video freezes (1.6%), all short segments at episode boundaries.

But here's the thing: the first/last frame comparison method would have caught some of these by coincidence, while also producing false positives on episodes where the robot simply starts and ends in the same position (which is completely normal for manipulation tasks).

The real issue isn't whether frames are identical — it's whether the video fails to capture motion that should be happening.

The Action Cross-Reference Approach

This is the key insight: a frozen video isn't the problem. A frozen video while the robot is supposed to be moving — that's the problem.

RDA's video freeze detection works in four stages:

Decode to low-res grayscale (64×64) — noise reduction matters more than resolution
Adaptive thresholding — compute frame-to-frame pixel differences, use the p10 as a noise floor, set threshold at max(0.10, 0.25 × noise_floor). This automatically adapts to different camera qualities across datasets.
Continuous segment detection — only flag sequences of consecutive frozen frames (minimum 0.5s by default), filtering out single-frame noise
Action cross-validation — check the action data during frozen segments. If the robot's commanded actions show movement but the video doesn't change, it's a real freeze. If actions are also near zero, the robot is simply idle.

This distinction eliminates the two failure modes of simpler approaches:

False positives: Robot starts and ends at the same position → first/last frames match → incorrectly flagged as frozen
False negatives: Video freezes for 5 seconds in the middle of an episode → first/last frames are different → freeze completely missed

Validation on G1_WBT

To verify the method, we audited the Unitree G1_WBT_Brainco_Pickup_Pillow dataset — 300 episodes, 4 camera views (head stereo + wrist), 177,811 frames at 30fps.

Results on the 44 episodes with available video:

0 frozen segments detected across all camera streams
Camera sync: temporal offset = 0ms, drift rate = 0ms/min — excellent hardware synchronization
Episode 18's wrist camera had a visual quality dip (score 0.5/1.0, blurry frames), but RDA correctly classified this as "degraded quality" rather than "frozen" — because the frame pixels were changing, just not sharply

This is exactly where the action cross-reference matters: blurry frames have pixel variation, frozen frames don't. The tool distinguishes between them.

What Else We Found

Video freeze turned out to be the least interesting finding. The more impactful issues across these datasets:

LIBERO — Action discontinuities: 3,622 spikes across 100% of episodes
LIBERO — High idle ratio: Only 28.3% of frames show effective motion
G1_WBT — Missing video files: 278 out of 300 episodes have no video streams
G1_WBT — Joint limit violations: 7 out of 36 DOF exceed configured limits

These structural issues are harder to spot than frozen frames and likely have a bigger impact on policy training. Video freeze is visible and easy to understand; action discontinuity and low state-space occupancy are invisible but potentially more harmful.

The Bigger Picture

Robot learning datasets are growing fast — Open X-Embodiment, DROID, AgiBot World — but quality assurance hasn't kept up. Most datasets are published with basic sanity checks but no systematic audit of temporal integrity, action continuity, or state-space coverage.

We built RDA because we kept running into these issues during our own work. It's open source (v0.9.9 on PyPI), supports LeRobot v2/v3 format, and produces structured JSON reports. We've audited 12+ datasets so far and shared findings directly with dataset maintainers through GitHub issues.

If you're collecting robot data or training policies on open datasets, video freeze is worth checking — but it's probably not the biggest quality issue hiding in your data.

Tool: RDA (Robot Data Audit)https://github.com/liesliy/rda — pip install robot-data-audit

We Audited 13 Public Robot Datasets With One Tool and Zero Tuning. Here's What the Numbers Actually Tell Us.

liesliy — Tue, 15 Sep 2026 07:35:19 +0000

RDA v0.9.7 was run against 13 LeRobot-format datasets from HuggingFace Hub — 4,940 episodes total, default thresholds, zero per-dataset tuning. The results challenge the assumption that "clean data" and "good training data" are the same thing.

The Problem

When most people evaluate robot manipulation datasets, they check one thing: is the data broken? Missing frames? NaN values? Timestamp inversions? If none of those — "data's fine, let's train."

That logic has a gap.

We ran RDA (Robot Data Audit) across 13 public LeRobot-format datasets — simulation and real, bimanual and single-arm, scripted and teleoperated — and found that while all 4,940 episodes pass integrity checks (L1) , the behavioral profiles (L2) and training efficiency metrics (L3) vary dramatically. And those variations directly affect training outcomes, yet they're invisible to conventional quality checks.

This article isn't about ranking datasets. It's about showing what a multi-layer audit reveals when you stop asking "is this data broken?" and start asking "what is this data actually like?"

The Four-Layer Framework

RDA's audit runs every episode through four sequential layers. The design rule is strict: **only hard integrity checks can set an EXCLUDE verdict; diagnostic measurements never do.##

L1 — Integrity Gate · Deterministic hard checks (missing / NaN / limit / video-stream) · 9 metrics · ✅ PASS → REVIEW / EXCLUDE
L2 — Trajectory Diagnostics · Observational motion & video anomalies · 8 metrics · ❌ findings only
L3 — Dataset Profile · Training-data efficiency & coverage · 4 metrics · ❌ findings only
L4 — Dataset Summary · Dataset-level P10/P50/P90 aggregation · 📊 report only

This separation — measurement vs. judgment — is the core design philosophy of RDA v0.9.7. And it wasn't always this way.

Layer 1: Everything Passes (That's the Starting Line, Not the Finish Line)

All 13 datasets, all 4,940 episodes:

✅ missing_dropout — pass (4,940/4,940)
✅ invalid_values — pass (4,940/4,940)
✅ schema_consistency — pass (4,940/4,940)
✅ timestamp_validity — pass (4,940/4,940)
✅ video_frame_integrity — pass (all video episodes)
✅ video_timestamp_alignment — pass (all video episodes)

Zero corruption. Zero NaN. Zero timestamp inversions. If your audit stops here, the conclusion is: "all good."

But it doesn't stop here.

Layer 2: The Behavior Layer — Where the Interesting Stories Are

Finding 1: Same Robot, Same Lab, 4× Difference in Motion

xarm_lift_medium and xarm_push_medium are both collected on the same xArm platform, in the same lab, by the same team. But their behavioral profiles are radically different:

xarm_lift_medium:

Effective motion ratio (median): 79.2%
Idle ratio (median): 20.8%
Episodes with L2 findings: 72/800 (9%)
Action discontinuity (median): 0 spikes/ep

xarm_push_medium:

Effective motion ratio (median): 16.7%
Idle ratio (median): 83.3%
Episodes with L2 findings: 629/800 (78%)
Action discontinuity (median): 1 spike/ep

Lift is a large-amplitude pick-and-place task — the arm moves most of the time. Push is a small-force nudge-then-watch task — the arm spends most of its time waiting.

Neither is "bad data." But a loss function trained on a 75%-idle distribution is structurally biased toward predicting "do nothing." If you don't know this about your data, you won't know where to look when training plateaus.

Finding 2: Action Discontinuity Tracks the Controller, Not the Dataset's Reputation

Action discontinuity measures sudden jumps in the action sequence using MAD-based spike detection. The distribution across datasets reveals something counterintuitive:

aloha_sim_insertion_scripted — 50/50 (100%) · median 33 spikes/ep · Scripted controller discretization
aloha_sim_transfer_cube_scripted — 50/50 (100%) · median 49 spikes/ep · Scripted controller discretization
aloha_sim_transfer_cube_human — 47/50 (94%) · median 31 spikes/ep · Teleoperation corrections
cmu_stretch — 72/135 (53%) · median 22 spikes/ep · Binary gripper 0↔1 flips
jaco_play — 176/1085 (16%) · median 10 spikes/ep · Occasional corrections
libero_10 — 0/379 (0%) · median 10 spikes/ep · Smooth teleoperation
pusht — 0/206 (0%) · median 5 spikes/ep · Smooth simulation
xarm_lift_medium — 0/800 (0%) · median 0 spikes/ep · Extremely smooth

The ALOHA scripted datasets have 100% of episodes with action spikes — not because the data is corrupted, but because scripted controllers produce discrete command transitions that manifest as step functions in the action space.

If your policy uses smoothness regularization, this number decides your curriculum. You need different regularization strength for scripted vs. teleoperated data, and that's a Layer 2 insight that L1 alone would never reveal.

Finding 3: Video Freeze — A Signal That Needs Context

In libero_10, RDA detected 6 episodes with video_freeze (stuck frames, detected via inter-frame pixel difference analysis):

Ep 90 — 223 frames · ~0.5s freeze · wrist_image
Ep 198 — 235 frames · ~0.7s freeze · wrist_image
Ep 240 — 289 frames · ~0.8s freeze · wrist_image
Ep 254 — 227 frames · ~0.6s freeze · wrist_image
Ep 255 — 291 frames · ~0.9s freeze · wrist_image
Ep 300 — 237 frames · ~0.7s freeze · wrist_image

Initial reaction: "camera malfunction, discard these episodes."

But cross-validation using ffmpeg frame-by-frame extraction from both cameras (agentview + wrist_image) revealed that the frozen frames appear simultaneously in both cameras and are located at episode boundaries (start or end).

This pattern is consistent with the robot naturally pausing between episode segments — not a camera hardware failure.

RDA correctly flags these as REVIEW ("inspect before training") rather than EXCLUDE ("data is corrupted"). This distinction is the practical value of layered auditing: surface the signal, provide context, let the human decide.

Layer 3: The Training Efficiency Layer — The Most Overlooked Dimension

Finding 4: State Space Occupancy Is Uniformly Low

State occupancy measures how much of the discretized state space (10×10 grid) is covered by a dataset's episodes:

pusht — median 39.0% · P10: 28.0% · P90: 51.0%
libero_10 — median 6.6% · P10: 4.7% · P90: 8.3%
utokyo_pr2_tabletop — median 5.0% · P10: 4.4% · P90: 5.8%
droid_100 — median 4.8% · P10: 3.2% · P90: 11.0%
svla_so101_pickplace — median 4.5% · P10: 4.0% · P90: 5.0%
aloha_sim_transfer_cube_scripted — median 5.0% · P10: 5.0% · P90: 5.0%
aloha_sim_insertion_human — median 3.8% · P10: 3.3% · P90: 4.4%
jaco_play — median 2.8% · P10: 2.2% · P90: 3.4%
xarm_lift_medium — median 2.4% · P10: 2.1% · P90: 2.5%
cmu_stretch — median 1.9% · P10: 1.9% · P90: 2.0%
xarm_push_medium — median 1.7% · P10: 1.4% · P90: 2.1%

Outside of pusht (a deliberately designed 2-DOF simulation), most datasets occupy only 1.7%–6.6% of their state space. Episodes within each dataset heavily overlap in state space, suggesting limited exploration diversity.

This isn't a "quality" judgment. It's a measurement. But if your model needs to generalize to unseen states, this number tells you whether your current dataset has enough coverage — and you'd never know from L1 alone.

Finding 5: Idle Ratio Varies 4× Across Datasets

Median idle ratio across all 13 datasets:

20%–30%: xarm_lift (20.8%)
63%–72%: ALOHA sim (4 datasets), droid_100, jaco_play, libero_10
79%–87%: pusht (81.7%), utokyo_pr2 (83.6%), svla_so101 (86.8%)

11 of 13 datasets have median idle ratio above 63%. The implication for training: loss functions on these distributions learn to predict "do nothing" as the default. This isn't a bug — it's a dataset characteristic. But without L3 measurement, you wouldn't know to adjust your training strategy accordingly.

The Verdict Pipeline Change: Why Findings Don't Auto-Escalate

The Old Problem
In v0.5.x, idle_ratio findings above threshold automatically escalated the episode verdict to REVIEW:

libero_10: 247/379 (65%) REVIEW → Root cause: Low-motion task, not data quality issue
pusht: 163/206 (79%) REVIEW → Root cause: Low-motion task, not data quality issue

Cross-validation showed these were false positives — high idle ratio is a natural property of fine-grained manipulation tasks, not a sign of data corruption.

The New Architecture

v0.9.7 restructured the verdict pipeline into a three-layer aggregate model:

L1 Hard Checks → directly determine PASS / EXCLUDE
L2/L3 Findings → reported as RISK_SIGNAL, never auto-escalate to verdict
L4 Summary → aggregated statistics for reporting only

The Result

libero_10: v0.5.x → 132 PASS / 247 REVIEW → v0.9.7 → 373 PASS / 6 REVIEW
pusht: v0.5.x → 43 PASS / 163 REVIEW → v0.9.7 → 206 PASS / 0 REVIEW
All others: Various REVIEW counts → v0.9.7 → 100% PASS

libero_10's remaining 6 REVIEW verdicts come exclusively from video_freeze — an independent, more reliable signal.

The separation of measurement and judgment reduced false-positive REVIEWs by 97-100% on low-motion datasets, while preserving genuine defect detection.

Blind Test: Precision 1.000, Recall 0.800 — And an Honest Regression

Setup

Dataset: lerobot/pusht (206 episodes)
Injection: 5 defect classes × 10 episodes = 50 defective episodes
Controls: 156 unmodified episodes
Mode: --no-video (Fast Audit)
Tuning: Zero tuning, all defaults

Results

empty (stale metadata) — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10
NaN in state — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10
reversed timestamps — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10
frozen episodes — EXCLUDE 0/10 · REVIEW 0/10 · Strict 0/10 · Broad 0/10 ⚠️
duplicate frames — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10

Strict: TP=40, FN=10, FP=0, TN=156 → precision 1.000, recall 0.800
Zero false positives — all 156 controls matched clean baseline.

The Regression We're Not Hiding

Frozen episode detection is a known regression in v0.9.7. The idle_ratio metric still detects the signal (RISK_SIGNAL), but the new pipeline doesn't auto-escalate it to REVIEW. We're publishing this openly because a tool's credibility comes from transparency, not from claiming perfection.

This will be addressed in a future version — either by restoring conditional idle_ratio escalation, or by adding a dedicated frozen-episode detector that distinguishes between naturally low-motion tasks and genuinely frozen data.

Design Philosophy: Measure, Don't Judge

RDA v0.9.7 makes a deliberate choice: be a measurement tool, not a judge.

L1 answers: "Is the data broken?" (hard checks, can EXCLUDE)
L2 answers: "What does the data look like behaviorally?" (findings only)
L3 answers: "How efficient is this data for training?" (measurements only)
L4 answers: "What's the overall distribution?" (summary only)

This choice comes from a simple recognition: no universal threshold works for every task, every platform, every collection method. A 70% idle ratio is normal for a push task and suspicious for a lift task. 33 action spikes per episode is expected for scripted data and alarming for smooth teleoperation.

RDA surfaces these signals. The judgment — "is this acceptable for my training pipeline?" — stays with you.

What This Means in Practice

"Clean data" ≠ "ready to train." All 13 datasets pass L1. Their L2/L3 profiles differ dramatically. Ignoring the behavioral and efficiency layers means missing information that directly affects training.
Never discuss data quality without task context. Same xArm platform, 4× difference in idle ratio. Task nature, not collection quality, explains the gap.
Use findings, not verdicts, for dataset selection. The RISK_SIGNALs from L2/L3 give you actionable information about motion patterns, exploration coverage, and action smoothness — regardless of whether they trigger a verdict.
Velocity and occupancy are not cross-platform comparable without normalization. RDA classifies velocity as Tier-2 (normalizable) and occupancy as platform-dependent. You need scaling factors before comparing across robots.

All data is real, reproducible, and comes from public HuggingFace datasets. Full JSON reports and reproduction scripts are available in the repository.

RDA v0.9.7 on PyPI: pip install robot-data-audit==0.9.7
GitHub: github.com/liesliy/rda
Benchmark data: docs/benchmark.md

I Audited AgiBot's New RL Dataset with RDA

liesliy — Tue, 01 Sep 2026 09:33:05 +0000

AgiBot just released AGIBOT WORLD 2026 Phase 3, a real-robot reinforcement-learning dataset. I used it as an independent test of RDA: can a third-party audit tool run against a fresh, large dataset without any tweaks?

Key Finding: RDA Spikes Align with AgiBot's Human-Takeover Labels

The real-robot HG-DAgger package includes an official intervened column: frame-by-frame human-takeover labels. This gave us a rare chance to validate RDA.

I aligned all 1,712 RDA action-discontinuity spikes against these labels:

226 spikes (13.2%) fall within ±1 frame of a takeover transition
Random baseline: ~4.3%
That is a 3.1× enrichment overall (1.6×–5.2× per episode)

Most spikes are normal teleop motion — which is why RDA labels them REVIEW, not FAIL. But the clear enrichment at takeover boundaries shows the signal is grounded in real events, not pure noise.

Verdict Distribution: 0 EXCLUDE / 0 FAIL

Across 1,112 episodes (5 simulation tasks + one real-robot package):

Verdict	Count
PASS	31
REVIEW	1,081
EXCLUDE	0
FAIL	0

All 31 PASS episodes come from simulation tasks. Verdict layering works — it is not a blanket REVIEW.

What Was Covered

I did not settle for a single tiny split. I ran RDA over all 5 simulation tasks and the smallest real-robot RL package (3.57 GB — the others are 17–50 GB each and would not fit on my drive):

Dataset	Episodes	Notes
Simulation × 5 tasks	1,102	200–283 episodes per task
Real robot HG-DAgger	10	23,658 frames, 30 fps

Data Engineering Passed the Audit

4,448 integrity checks (missing frames, invalid values, schema, timestamps), 0 failures
Constant 30 fps
6 sampled info.json files, all LeRobot v2.1

The data is ready for training. REVIEW items are worth a spot-check, but nothing needs to be cleaned before use.

Run It Yourself

Grab the dataset from the official HF repo (simulation splits are a few hundred MB; the audit doesn't read video files, so you can skip the videos/ tarballs), then:

pip install robot-data-audit
rda audit ./your_dataset_dir
rda recommend ./your_dataset_dir --offline

The output is per-episode detail plus an overall verdict, in JSON so it plugs into any pipeline.

Scope Boundary (Said Honestly)

Tested: 5/5 simulation tasks (~500 MB) and 1/50 real-robot packages (2%).
Not tested: the remaining 49 real-robot packages (~3.9 TB), ImitationLearning 4.4 TB, and RichInteraction 2.3 TB.
Why the format conclusion generalizes: every sampled info.json uses LeRobot v2.1 with the same pipeline.
No video files were read, so you can skip the videos/ tarballs to save bandwidth.

The full 6 JSON audit reports are on hand for anyone who wants to reproduce or challenge the numbers.

RDA (Robot Data Audit) is an open-source tool I maintain. PyPI: robot-data-audit. GitHub: liesliy/rda.

I Poisoned 50 Episodes of LeRobot's PushT Dataset. The Audit Tool Caught 40 — With Zero False Alarms.

liesliy — Mon, 31 Aug 2026 03:51:54 +0000

TL;DR: I took lerobot/pusht — the same dataset used to train Diffusion Policy — injected 50 defective episodes across 5 defect classes (seed=42), kept 156 episodes untouched as controls, and ran RDA (v0.5.4, default thresholds, zero tuning) against it. Result: precision 1.000, recall 0.800 on a strict criterion, and zero false positives on all 156 clean controls. Broad criterion (including review signals): 50/50. Runtime for the full 206-episode audit: 2 seconds. All numbers below are from real runs; the honest caveats are published too.

The problem: benchmarks without ground truth prove nothing

I maintain RDA (Robot Data Audit), an open-source quality-audit tool for robot manipulation datasets. Its README claims 13 metrics, a three-tier verdict system, and a 12-dataset benchmark. But every one of those numbers is an observation on datasets whose ground truth we don't control. When the tool reports 83% idle ratio on a dataset, is that a real defect or just how that robot behaves? Without answers, "audit" is just a nicer word for "aggregate statistics".

The standard answer is a blind test: manufacture the ground truth, then see what the tool actually finds. This post documents the first one.

Why lerobot/pusht

Three reasons, all practical:

It's famous. PushT is the manipulation dataset from the Diffusion Policy paper, hosted by HuggingFace's official LeRobot org. If a data-quality tool wants credibility, this is a dataset its audience already knows.
It's standard. LeRobot v3.0 format — parquet + mp4 + episode metadata, exactly the format RDA supports natively.
It's small. The full dataset is 16 files / 7.7 MB / 206 episodes / 25,650 frames — small enough to audit in full, no sampling shortcuts.

Download integrity was verified byte-for-byte: 25,650 frames = sum of episode lengths = video frame count; all LFS sha256 hashes matched.

The experiment design

Step 1 — Baseline. Audit the untouched dataset first. Result: 43 PASS / 163 REVIEW / 0 EXCLUDE. The 163 reviews all come from idle_ratio — PushT is a low-motion task (median idle ratio 82%), the robot spends most of its time repositioning. That's a property of the data, not a defect, and it matters later.

Step 2 — Inject. 5 defect classes × 10 episodes each (seed=42), leaving 156 episodes untouched as controls:

empty — delete all rows from the data parquet, leave meta/episodes claiming the episode still exists → expected: EXCLUDE
nan_state — set observation.state to NaN on 15 random frames per episode → expected: EXCLUDE
timestamp_reverse — reverse the second half of each episode's timestamps → expected: EXCLUDE
frozen — freeze the entire episode at first-frame values → expected: REVIEW (statistical signal — shouldn't hard-fail)
duplicate_frames — duplicate 5 random frames per episode, copies appended at the end → honest probe, see caveats

Step 3 — Audit blind. rda audit v0.5.4, default thresholds, no tuning, then compare every verdict against the manifest.

Results

Confusion matrix — strict criterion (EXCLUDE = flagged):

                 flagged   not flagged
defective (50)   TP = 40   FN = 10
control   (156)  FP =  0   TN = 156

Precision 1.000 · Recall 0.800 · Zero false positives on 156 untouched controls — every control episode's verdict is identical to its clean-baseline verdict, so the injection itself didn't perturb anything else.

Broad criterion (EXCLUDE + REVIEW = flagged): 50/50, recall 1.000.

Per class (strict / broad / detector):

empty — 10/10 strict · 10/10 broad · _zero_frame_guard, a regression probe for a P0 bug we fixed in v0.4.12 (zero-frame episodes used to silently PASS)
nan_state — 10/10 strict · 10/10 broad · invalid_values
timestamp_reverse — 10/10 strict · 10/10 broad · timestamp_validity
frozen — 0/10 strict · 10/10 broad · idle_ratio (effective motion 0%) → REVIEW, by design
duplicate_frames — 10/10 strict · 10/10 broad · timestamp_validity, only because of placement; see caveats

Runtime: 2 seconds for 206 episodes / 24,488 frames on a consumer laptop.

The honest caveats (the part most tool posts skip)

duplicate_frames detection is an artifact of placement. The copies landed at the episode tail, creating 2–4 negative timestamp deltas each. A mid-stream duplicate with monotone timestamps would not be flagged by 0.5.4. I publish this because the class name sounds scarier than what was measured.
Frozen episodes get REVIEW, not EXCLUDE. RDA treats statistical anomalies as review signals, not hard failures. If you believe frozen arms must hard-fail, that's a one-line policy change — the thresholds are open, argue with us in an issue.
pusht is an unusually easy payload: 2-D state, 96×96 video, no multi-stream timestamps. sensor_synchronization and joint_limit were N/A the whole run — this blind test doesn't exercise them.
121 of the 156 "clean" controls also got REVIEW (all from idle_ratio). This is exactly why the tool reports distributions instead of bare pass rates — on a task like PushT, a bare pass rate would look terrible for reasons that have nothing to do with data quality.

Why this matters beyond one tool

Robot learning is hitting a data-quality wall: more labs are collecting manipulation data, and "the training set had silently corrupt episodes" is the kind of failure you discover after the policy fails to transfer. Blind-testing your audit tooling against a dataset your audience knows — with published confusion matrices and published weaknesses — is a much better baseline than vendor benchmarks with no ground truth. The full method, manifest schema, and per-episode verdicts are in the repo; the injection tool will ship as rda blindtest in a future release.

Reproduce

pip install robot-data-audit==0.5.4
# download lerobot/pusht (v3.0), apply the 5×10 injection with seed=42
rda audit <blindtest_dataset> --format json

Full experiment doc (tables, per-class detectors, caveats): docs/blind_test_20260831.md · Benchmark: docs/benchmark.md

RDA is MIT-licensed, runs fully locally — your data never leaves the machine.

If you maintain a robot dataset and want it audited (or blind-tested) as a public benchmark entry, open an issue on the repo — the benchmark grows one dataset at a time.

We Need a Unicode for Tactile Data — Here's Why I Built One

liesliy — Tue, 25 Aug 2026 07:13:42 +0000

I've spent the last few years working with tactile sensors in robotics. And if there's one thing that drives me crazy, it's this: every sensor speaks its own language, and nobody's translating.

GelSight gives you image sequences. BioTac streams 19-channel impedance data. PaXini sends you 8×8 taxel grids. If you work with two different sensors in the same project, you end up writing two completely different data pipelines. Switch robots? Throw away your labels and start over.

This isn't just annoying — it's actively holding back embodied AI. We're trying to train foundation models on tactile data, but the data is fragmented across dozens of incompatible formats. It's like trying to build GPT when every training corpus uses a different encoding.

So what if tactile data had... Unicode?
That's the idea behind TLabel — a unified annotation schema for tactile data across all sensor types.

Think about what Unicode did for text. Before Unicode, you had ASCII, Shift-JIS, Latin-1, GB2312... the same character, different encodings, constant headaches. Unicode didn't replace any of them — it gave them a shared identity. A Chinese character is a Chinese character, regardless of how your system stores it.

TLabel does the same thing for touch. It defines 14 semantic dimensions that any tactile sensor can map to:

Did something touch? (contact)
Where on the sensor? (contact_region, contact_centroid)
How hard? (force_magnitude)
Which direction? (force_vector)
Is it slipping? (slip_event)
What's the texture? (texture, friction)
Is the object deforming? (deformation_rate)
What phase of manipulation? (approach, grasp, hold, release)
And a few more...

The key insight: not every sensor can annotate all 14 dimensions. And that's fine. A simple force sensor gives you contact + force (Level 2). A vision-based sensor like GelSight can give you spatial info too (Level 3). Nobody's forcing you to pretend your sensor can do more than it can.

pip install tlabel

That's it. Three words.

Why this matters now
We're at a weird moment in tactile sensing. The hardware is getting good — there are dozens of capable sensors now. The ML models are getting better. But the data layer is still stuck in the dark ages.

Every lab has their own scripts. Every paper has its own format. Every startup that tries to build something on top of tactile data spends 3 months just writing data adapters before they can do anything interesting.

I built TLabel because I was tired of writing the same adapter code for the fifth time. And I think we've reached the point where the robotics community needs this — the same way the web community needed HTTP, the same way the text community needed Unicode.

What it actually looks like
Here's the real test: can you load data from a GelSight, a BioTac, and a PaXini, and get the same annotation structure back?

import tlabel

# These all return the same TLabelData structure
gelsight_data = tlabel.load("gel_sight_recording.pkl")
biotac_data = tlabel.load("biotac_recording.csv")
paxini_data = tlabel.load("paxini_recording.mat")

# Same schema. Same dimensions. Different sensors.
print(gelsight_data.frames[0].schema_v2.contact)
print(biotac_data.frames[0].schema_v2.contact)

That's the whole point. The downstream code — your model training, your quality checks, your data export — doesn't need to know which sensor it came from.

Where we are
TLabel is at v0.20.1 with 13 sensor adapters already built. It supports export to FTP-1 Zarr (for foundation models) and LeRobot format. It has a CLI, a quality scoring system, and a compliance level framework.

It's MIT licensed, actively maintained, and we're actively looking for collaborators — especially if you work with sensors we haven't covered yet.

👉 GitHub: github.com/liesliy/tlabel
· PyPI: pypi.org/project/tlabel

If you're working with tactile data and you're tired of writing the same adapter code for the fifth time — give it a shot. Or even just tell us what you think. The whole point of a standard is that it only works if people use it.

Built by the TouchLabel AI team. Open source, MIT license. Questions? Hit us up on GitHub Issues or drop a comment.

Robot Training Data Is Messier Than You Think: Auditing 4,959 Episodes with an Open-Source Tool

liesliy — Fri, 21 Aug 2026 08:48:10 +0000

Robot learning is getting better very quickly.

We now have better policy architectures, more capable simulation environments, standardized dataset formats such as LeRobot, and an increasing number of public robot manipulation datasets.

But there is a basic question that is surprisingly difficult to answer:

How do we know whether a robot training dataset is actually usable?

A dataset can be perfectly readable and still be a poor training dataset.

It can have valid Parquet files, correct schemas, and complete metadata while containing excessive idle motion, action discontinuities, sampling problems, distribution anomalies, or episodes that deserve human review.

I built RDA (Robot Data Audit) to explore this problem.

I recently ran it across 12 local robot datasets and 4,959 episodes.

Here is what I found.

"Can I Load It?" Is Not the Same as "Can I Train on It?"

Robot datasets can contain:

camera observations
joint states
actions
timestamps
task metadata
robot configuration
multiple sensor streams

Most data pipelines start with a simple question:

Can I load the dataset?

That's necessary, but it isn't enough.

I think robot data quality needs to be considered in several layers:

Layer	Question
Structural integrity	Does the data actually exist and load correctly?
Schema & temporal integrity	Are fields, values, and timestamps valid?
Behavioral quality	Does the robot motion contain suspicious patterns?
Task relevance	Are those patterns actually harmful for this task?

RDA currently focuses primarily on the first three layers and produces structured evidence that can be used to investigate the fourth.

This distinction matters.

A statistical anomaly is not automatically a failed demonstration.

And a dataset that passes structural validation is not automatically good training data.

What RDA Actually Measures

RDA is an open-source, local-first auditing tool for LeRobot-format robot manipulation datasets.

The core audit runs locally and does not require a GPU.

The current version checks 13 metrics across three layers.

Layer 1 — Structural Integrity

Deterministic checks include:

missing frames
NaN / Inf values
schema consistency
timestamp validity
joint limits

Hard structural failures can result in an EXCLUDE verdict.

Layer 2 — Temporal & Motion Quality

These checks look for statistical risk signals such as:

sensor synchronization
sampling jitter
velocity / acceleration anomalies
action discontinuity
temporal sufficiency

These are generally signals for investigation rather than proof that an episode is unusable.

Layer 3 — Dataset Utility

This includes:

idle ratio
distribution characteristics
state-space coverage

These metrics are particularly contextual.

For example, a high idle ratio may be perfectly reasonable for one task and problematic for another.

PASS, REVIEW, EXCLUDE — But Don't Confuse Them with Ground Truth

One design decision became particularly important during development.

RDA separates workflow verdicts from evidence levels.

The workflow verdict is:

PASS
REVIEW
EXCLUDE

The evidence level is:

HARD_FAIL
RISK_SIGNAL
UNVERIFIABLE

These are deliberately different concepts.

For example:

High action discontinuity → RISK_SIGNAL

does not mean:

The demonstration is corrupted.

Likewise:

Missing frames → HARD_FAIL

does not necessarily mean:

The entire upstream dataset is bad.

The purpose is to keep an audit tool from turning weak statistical evidence into overly confident labels.

4,959 Episodes: Almost Half Triggered a Review Signal

I ran RDA across 12 local dataset copies.

The overall result was:

Verdict	Episodes	Percentage
PASS	1,711	34.5%
REVIEW	2,473	49.9%
EXCLUDE	775	15.6%

These numbers are not a confusion matrix.

There was no independent human ground truth for this benchmark, so they cannot be used to claim precision or recall.

The interesting observation is simply:

A large fraction of the tested episodes contained signals worth investigating before blindly feeding them into training.

That alone is useful.

Finding #1: Idle Data Is Everywhere

The median idle ratio across the tested datasets ranged from:

20.8% to 93.3%.

Ten of the twelve datasets had median idle ratios above 65%.

That raises a surprisingly important question:

If a large fraction of your training data represents the robot doing very little, what exactly is the policy learning?

It might be useful temporal context.

Or it might simply teach the model that doing nothing is common.

The answer depends heavily on the task and model architecture.

This led to our first optimization experiment.

Finding #2: The Same Robot Can Have Very Different Data Characteristics

Two datasets from the same xArm platform showed a large difference.

xArm lift

Median idle ratio: 20.8%
767 / 800 episodes received PASS

xArm push

Median idle ratio: 83.3%
562 / 800 episodes received REVIEW

Same robot platform.

Different task.

The important lesson is:

Data quality is not completely independent of task structure.

A high idle ratio is not automatically a collection failure.

It may simply reflect how the task is performed.

That makes universal rules like:

"Delete every episode above 70% idle."

dangerous.

Finding #3: Action Discontinuity May Reveal the Controller

Another interesting pattern appeared in action discontinuity.

Some datasets showed spikes in essentially every episode.

Another xArm dataset had only a handful across 800 episodes.

For example:

simulated ALOHA datasets showed frequent spikes
the SO-100 dataset also showed frequent spikes
xArm lift had only 6 spike-containing episodes out of 800

This suggests that some data-quality signals may tell us as much about the collection or control stack as about the demonstrations themselves.

That could matter for:

smoothness-regularized policies
sim-to-real transfer
controller comparisons
teleoperation system evaluation

But again:

A discontinuity is a signal, not automatically a failure.

Finding #4: A Dataset Can Be Structurally Clean and Still Need Review

One of the most useful observations from the benchmark was the separation between integrity and behavior.

Outside of the intentionally corrupted ALOHA fixture, the tested datasets were largely clean at the basic integrity layer.

Yet the behavioral layer still flagged many episodes for review.

In other words:

"The files are valid" and "the demonstrations are ideal for training" are different questions.

This seems obvious after writing it down.

In practice, however, many pipelines stop at the first question.

Finding #5: Sometimes the Dataset Copy Is the Problem

One of the most interesting cases was LIBERO.

The local metadata declared:

1,693 episodes

But only:

920 episodes

could actually be read from the local Parquet files.

The remaining:

773 episodes

were missing from the local copy.

An earlier loader implementation could interpret this kind of metadata/data-layout mismatch as zero-frame episodes.

The loader was changed to scan the actual Parquet layout and build a fallback episode index.

The result was more honest:

The local copy is incomplete.

Not:

LIBERO contains 773 empty episodes.

This distinction matters.

A broken local copy is not necessarily a broken upstream dataset.

Then I Compared RDA with Other Tools

I also wanted to understand how different robot-data tooling behaves on deliberately corrupted data.

So I ran a small cross-tool benchmark involving:

RDA
trajlens
ORBIT
lerobot-doctor

The test included known injected defects.

One result looked like this:

Defect	RDA	trajlens	ORBIT	lerobot-doctor
NaN actions	EXCLUDE	WARN	Crash	FAIL
Timestamp reversal	EXCLUDE	FAIL	Crash	FAIL
Frozen actions	PASS	—	—	FAIL

The important result wasn't that RDA "won."

It didn't.

The frozen-action case exposed a real blind spot.

The Frozen-Segment Blind Spot

RDA's current idle_ratio metric is primarily episode-level.

That means a dataset can have a reasonable overall idle ratio while still containing a long locally frozen segment.

Another tool had a dedicated consecutive-identical-actions check and caught it.

This gives RDA a very concrete next feature:

frozen_segment

A good audit tool should be able to identify its own blind spots.

This one is now on our priority list.

Can Data Auditing Actually Improve Training?

Finding suspicious data is only half the problem.

The more interesting question is:

Can audit results tell us what to do with the dataset?

I ran several experiments around idle-data pruning.

The result was much more nuanced than I initially expected.

Moderate Pruning Helped — Aggressive Pruning Hurt

In one experiment:

Strategy	Data retained	MSE change
Baseline	100%	—
Keep 50%	50.1%	-28.9%
Keep 40%	40.1%	-12.5%
Keep 30%	30.1%	-13.3%
Keep 20%	20.0%	+11.5%
Keep 10%	10.0%	+98.2%

There was a clear dose-response pattern.

Moderate pruning helped in this experiment.

Aggressive pruning eventually became harmful.

So the conclusion is not:

"Remove idle data."

It is:

There may be a useful pruning region, but it depends on how the remaining data is used.

Then We Tested Different Model Architectures

This was probably the most interesting result.

We compared a frame-wise MLP with a temporal Transformer.

The result was almost the opposite.

MLP

Idle-data pruning:

-21.6% MSE

Temporal Transformer

Idle-data pruning:

+95.5% MSE

Why?

Because temporal models need context.

Suppose:

idle ratio ≈ 70%
active segments are scattered
sequence length = 10

After pruning, many active segments become shorter than the required temporal window.

The result:

You removed "low-value" frames and accidentally removed the context required to construct training samples.

There is another issue.

A temporal policy may need to learn the transition:

idle → movement

If all idle context is removed, the model may never see how movement starts.

The Practical Lesson

This changed how I think about automated dataset optimization.

A rule like:

idle_ratio > 70% → delete

is too simplistic.

The better question is:

What model are you training, and what temporal structure does that model need?

The current experiments suggest:

gentle trimming can be reasonable for frame-wise models
aggressive idle pruning can be harmful
temporal models need special handling
dataset/task domain matters
audit signals should not automatically become deletion rules

This is why RDA's recommendation layer takes the intended model type into account.

A Local-First Architecture

Robot datasets can contain commercially sensitive information.

They may reveal:

robot configurations
manipulation strategies
factory environments
camera observations
proprietary tasks
operator behavior

RDA's core audit therefore runs locally.

It can generate a blind report containing:

anonymized paths
dataset statistics
episode/frame counts
metrics
verdicts

without sending the raw trajectories or images.

The goal is simple:

Share evidence without sharing the dataset.

What RDA Does NOT Claim

This is probably the most important section.

The current experiments do not establish:

RDA precision
RDA recall
universally optimal thresholds
that every EXCLUDE episode should be deleted
that action discontinuity means corruption
that high idle ratio means useless data
that RDA improves training success rate
that RDA certifies compliance with any data-quality standard

The benchmark is based on real local dataset runs, but it does not have independent ground-truth labels.

So the next experiment is obvious:

Get real human or customer QC labels and compare them against RDA without exposing the RDA verdict first.

The Next Experiment: Blind Human Review

The proposed validation workflow is:

Select 100–500 episodes from one robot platform.
Generate anonymized samples.
Have reviewers independently label them:

KEEP
REVIEW
REMOVE
1. Do not show them RDA's verdict.
2. Preserve reviewer decisions and notes.
3. Reveal RDA results only after labeling.
4. Compare the two sets of evidence.

That would allow us to measure:

overlap between RDA HARD_FAIL and human REMOVE
overlap between RDA RISK_SIGNAL and human REVIEW
additional defects found by RDA
reviewer time per episode
review queue reduction
potential downstream training impact

Until that experiment is completed, these remain validation targets rather than marketing claims.

Try It

RDA is open source and currently supports LeRobot v2.1 and v3.0.

Install:

pip install robot-data-audit

Audit a dataset:

rda audit /path/to/lerobot/dataset -v

Generate a blind report:

rda audit /path/to/lerobot/dataset --blind --format json

Generate optimization recommendations:

rda recommend /path/to/dataset --policy frame-wise

or:

rda recommend /path/to/dataset --policy temporal

Launch the optional UI:

pip install robot-data-audit[ui]
rda ui

GitHub:
https://github.com/liesliy/rda

PyPI:
https://pypi.org/project/robot-data-audit/

What I Want to Test Next

The next step isn't another synthetic benchmark.

I want to test RDA against real robot data with independent QC labels.

In particular, I'm looking for:

real-world teleoperation datasets
datasets with existing QC labels
datasets containing known failure cases
datasets used for sim-to-real experiments
robot companies or research groups willing to run a local audit

The raw data does not need to leave your environment.

A small validation set — even 100–500 episodes from one robot and 1–3 tasks — would already be extremely useful.

If you work with robot manipulation data and are willing to challenge these results, I'd love to compare notes.

Final Takeaway

After auditing thousands of robot episodes, my conclusion is not that robot datasets are "bad."

It is more subtle:

Robot datasets contain many properties that are invisible to a simple "can I load this file?" check.

And those properties can matter differently depending on:

the robot
the task
the controller
the dataset format
the policy architecture
the intended use of the data

The goal of RDA is not to produce another mysterious quality score.

It is to make the path from:

measurement → diagnosis → human review → optimization

more reproducible.

And eventually, hopefully, to make robot data quality something we can discuss with evidence rather than intuition.

Before the GPU bill arrives, audit the data.

# Public Robot Dataset Health Check — as measured by RDA v0.5.2

liesliy — Wed, 19 Aug 2026 03:30:56 +0000

We ran RDA across 11 public LeRobot-format datasets — 4,909 episodes in total, spanning sim and real, scripted and human teleop, research arms and $100 hobby hardware. Summary:

Dataset	Type	Eps	Verdicts (P/R/E)	Action spikes	Idle median	Idle p5
aloha_sim_insertion_human	sim, human	50	5 / 45 / 0	1,338 (100% eps)	70.7%	65.1%
aloha_sim_transfer_cube_scripted	sim, scripted	50	5 / 45 / 0	2,489 (100% eps)	64.0%	44.2%
aloha_sim_insertion_scripted	sim, scripted	50	1 / 49 / 0	1,633 (100% eps)	63.7%	53.7%
droid_100	real Franka	100	34 / 66 / 0	1,428 (99% eps)	70.7%	55.7%
pusht	sim	206	43 / 163 / 0	1,148 (97% eps)	81.7%	59.5%
HuggingFaceVLA/libero	sim	1,693	0 / 3 / 1,690	34 (3 eps)	76.5% (3 eps)	74.6%
bridge_orig_lerobot (sampled)	real WidowX	25	4 / 21 / 0	91 (84% eps)	93.3%	20.5%
xarm_lift_medium	real xArm	800	767 / 33 / 0	6 (1% eps)	20.8%	12.5%
xarm_push_medium	real xArm	800	238 / 562 / 0	845 (62% eps)	83.3%	16.7%
svla_so101_pickplace	real SO-100	50	5 / 45 / 0	260 (100% eps)	86.7%	59.5%
jaco_play	real Jaco	1,085	390 / 695 / 0	11,958 (77% eps)	74.1%	52.7%

All integrity layers (NaN/Inf, timestamp validity, missing frames, schema) came back clean on all 11. libero note: 1,690 of 1,693 episodes read 0 frames — a dataset-side meta/layout mismatch, now flagged EXCLUDE instead of silently passing. bridge sampled 25 episodes; the other 10 datasets were audited in full. (aloha_sim_transfer_cube_human was audited earlier at v0.5.1 with matching results: 1/49/0, 1,535 spikes, 71.2% idle — it's now a gated repo, so we couldn't re-pull it.)

Five patterns worth knowing before you train

1. Median idle runs 20.8%–93.3%, and 8 of 11 datasets sit above 65%.

Loss functions trained on a 75%-idle distribution are structurally biased toward predicting "do nothing" unless you weight or curriculum it. Bridge data pushes it to 93%. Measure yours before the GPU bill, not after.

2. Same robot, same lab, four-fold idle difference.

xarm_lift_medium: 20.8% median idle, 767/800 episodes PASS. xarm_push_medium: 83.3% median idle, 562/800 REVIEW. Same xArm platform — the difference is task difficulty (lifting vs. pushing a flat object), not collection sloppiness. High idle isn't always a bug; it's a property you need to know and design around. RDA flags both sides of this honestly.

3. Action discontinuity tracks the controller, not the dataset's reputation.

Sim ALOHA and the SO-100 hobby setup spike in literally 100% of episodes; xArm lift data has 6 spikes across 800 episodes. If your policy uses smoothness regularization or you're doing sim-to-real action statistics, this number decides your curriculum.

4. Clean integrity ≠ good training data.

Integrity passed 11/11 — zero NaNs, zero timestamp reversals, zero missing frames anywhere. The behavior layer still flagged 45–98% of episodes for review in most datasets. Both layers matter; most pipelines check neither.

5. Cheap hardware produces the most expensive data.

The community SO-100 pick-place set: 86.7% median idle plus spikes in every episode. If you're fine-tuning on hobby-robot uploads, this is what you're inheriting.

Reproduce

pip install robot-data-audit
rda audit <any lerobot dataset> -v
rda recommend <dataset> --policy temporal   # or frame-wise, --lang en

Tool: https://github.com/liesliy/rda · PyPI: robot-data-audit · UI: rda ui (EN/中文)

Caveats: RDA flags statistical anomalies, not ground-truth errors. REVIEW means "look before you train," not "discard." All thresholds are open for debate — that's what the issue tracker is for.

I Open-Sourced a Data Quality Auditor for Robot Datasets — It Found 1,535 Action Spikes in the Official ALOHA Demo Set

liesliy — Wed, 19 Aug 2026 02:55:47 +0000

Everyone in embodied AI says "data is the new code." Nobody audits it like code.

We run linters, static analyzers, and CI gates on our source code. Then we feed 50GB of teleoperation recordings into a policy network and hope for the best. So I built RDA (Robot Data Audit) — an open-source CLI that treats robot datasets the way ruff treats a Python repo.

pip install robot-data-audit
rda audit /path/to/lerobot/dataset

It works natively on LeRobot-format datasets (v2.1 + v3.0), checks every episode across two layers — integrity (NaN actions, timestamp reversals, missing frames) and behavior (action discontinuity, idle ratio, frozen segments) — and outputs a per-episode verdict: PASS / REVIEW / EXCLUDE.

The dogfooding surprise

Before releasing it, I ran RDA against the datasets everyone treats as ground truth — starting with aloha_sim_transfer_cube_human, the official ALOHA simulation demonstration set that ships with the LeRobot ecosystem.

Result from the behavior layer, across 50 episodes:

Action discontinuity spikes: 1,535 (~30 per episode)
Median idle ratio: 0.71** — the arm is effectively stationary 71% of frames
Median effective motion ratio: 0.29

To be clear about what this does and doesn't mean: none of this is corruption. The integrity layer came back clean — no NaNs, no broken timestamps. The spikes are real discontinuities in the action space (large frame-to-frame joint jumps), and the high idle ratio likely reflects grasping/hover phases where the gripper holds still. Neither is necessarily a bug in the dataset.

But that's exactly the point. If you're benchmarking a policy on this data, or worse, fine-tuning on it, these numbers are context you didn't have. Is 30 spikes per episode normal for this task? Does 71% idle time skew your loss toward predicting "do nothing"? Nobody asks, because nobody measures.

The bug I found in my own tool (while writing this post)

Honesty section, because dev.to deserves better than marketing:

My first run reported all 50 episodes as PASS. Green across the board. Celebration ensued.

Then I cross-checked the behavior layer output against the verdicts and realized they weren't connected — the metrics were computing 1,535 spikes, and the verdict aggregator was ignoring behavior signals entirely. The tool had the evidence and wasn't reading it. The loudest silence in software is a metric that's computed but never consumed.

Fixed now: behavior signals feed the verdict through a dataset-utility layer, and metric-level findings carry human-readable reasons. The ALOHA run now correctly flags 49/50 episodes as REVIEW with the specific signals attached.

If you're building anything with a "signal producer → decision aggregator" architecture, test the wiring, not just the signals. I wrote a negative-control test for it before I trusted my own tool again.

Why this matters more than it sounds

Robot learning teams are drowning in data collection — teleop sessions, sim rollouts, fleet logs — with almost no tooling for "is this batch usable before I burn GPU hours on it." An episode with a frozen sensor or a corrupted timestamp doesn't fail loudly. It trains quietly.

RDA's philosophy: audit before train. Cheap checks first (seconds per episode, pure numpy/pandas), verdicts you can gate in CI:

rda audit ./my_dataset --format json -o report.json
# fail the pipeline if any episode comes back EXCLUDE

There's also a built-in UI (rda ui) for browsing verdicts without spelunking JSON — and as of v0.5.2 it's fully bilingual: one toggle switches the entire dashboard, backend recommendation copy included, between English and 中文.

What shipped since the first post

rda recommend — model-aware optimization advice. Tell it whether you're training a frame-wise model (MLP/BC) or a temporal one (ACT/Diffusion Policy), and it gives different answers for the same data — including an explicit DO_NOT_PRUNE guard for temporal models when valid-window ratio collapses. Every suggestion carries its experimental evidence: pruning cost our seq=10 temporal baseline +296% MSE, while trimmed ALOHA/PushT improved frame-wise baselines by 11–35%.
Privacy-first architecture — metrics compute locally; only <1KB of aggregates reach the rules API. rda audit stays 100% offline, always.
LeRobot v2.1 support — bridge-style layouts now load natively (first run on bridge data: median idle ratio 93.3%. Real robots spend a lot of time deciding.).
A bug class worth naming: silent PASS. 1,690 zero-frame episodes in a popular dataset were passing because "no evidence of problems" was treated as "no problems." Now zero-frame episodes are explicit EXCLUDEs with a diagnosis.

What's next

More behavior metrics (jitter, cycle anomalies, calibration drift)
Trend dashboards across successive audits (already in the UI's History page — feedback wanted)
Export-to-clean: one-click filtered dataset copy from surviving episodes

The project is early and hungry for real-world datasets to chew on. If you have a LeRobot-format dataset (v2.1 or v3.0), run rda audit on it and tell me what turns up — especially if it's boring. Boring results from real data are how a tool earns trust.

Issues, PRs, and "your idle-ratio threshold is wrong, here's why" comments all welcome.

(RDA is MIT-licensed. I also do paid data-quality deep dives and pipeline integration for teams that want the audit without the homework.)

Tool: https://github.com/liesliy/rda · PyPI: robot-data-audit · UI: rda ui (EN/中文)