Recently someone raised a thoughtful question on our GitHub Issue:
Is the 'minimal state change' threshold in RDA using normalized units or raw STS3215 encoder steps (4096 per turn at 30 fps)? At step-level thresholds, slow fine motion near the brick reads as idle.
This question cuts to the core of idle detection design. Let me explain our reasoning and the trade-offs involved.
The Problem
In robot data auditing, idle detection is fundamental but deceptively complex. How do you define "idle" when auditing an episode?
Intuitively, a robot standing still is idle. But what does "still" mean?
Sensor noise causing micro-jitters — is that idle?
Slow approach before grasping — is that idle?
Holding pose while waiting for the next command — is that idle?
There's no universal answer.
Why Not Fixed Thresholds?
Our first attempt used a fixed threshold: ||Δaction|| < 0.01 means idle.
Problem: action scales vary wildly across datasets.
SO-101 with STS3215 servos: action range 0-4096 (encoder steps)
Franka Panda: normalized [-1, 1]
ALOHA: different scale again
A fixed threshold works for one dataset but fails catastrophically on another.
Normalization seems like the obvious fix, but normalizing to what? Each dataset has its own action distribution characteristics. Unified normalization loses dataset-specific information.
The Bimodal Gap Detection Approach
We realized that each episode has its own "motion distribution" .
A typical manipulation episode contains two types of frames:
Idle frames: robot stationary or micro-jittering, small action changes
Active frames: robot moving, significant action changes
These two types usually form a bimodal distribution in action change magnitudes.
Our algorithm:
Compute motion: ||Δaction|| (L2 norm of action first-difference)
Build a histogram (30 bins)
Find the valley between the "low cluster" (idle) and "high cluster" (active)
The valley position becomes the threshold
If no clear bimodal structure exists (e.g., all frames are slow movements), we fall back to 3 × MAD (Median Absolute Deviation) as a conservative estimate.
Answering the Issue Question
RDA's idle threshold operates in raw action space units, not normalized.
For SO-101, this means the threshold is at the encoder step level. But the threshold is computed per-episode adaptively, not a fixed global constant.
Benefits:
Preserves dataset-specific action scale characteristics
Each episode adjusts its own threshold based on its motion distribution
No cross-dataset normalization assumptions required
The cost?
Exactly what the Issue identified: slow fine motions can be misclassified as idle.
If a task involves entirely slow, precise operations (e.g., precision assembly), the action change magnitudes are uniformly small, the bimodal distribution may not exist, and the threshold becomes very small — causing many "meaningful but slow" motions to be classified as idle.
This is a real limitation, not a bug.
The Design Trade-off
We considered four approaches:
Fixed threshold — Simple and interpretable, but not comparable across datasets. A threshold that works for SO-101 fails on Franka.
Normalized threshold — Cross-dataset comparable, but loses dataset-specific characteristics. Normalizing to what? Each dataset has its own action distribution.
Per-episode adaptive (what we chose) — Adapts to each episode's motion distribution. The downside: slow tasks may have their fine motions misclassified as idle.
Learned threshold — Theoretically optimal, but requires labeled data and generalization is unknown.
We chose per-episode adaptive because:
RDA is a diagnostic tool, not a pass/fail gate. We output idle_ratio as a measurement, not a judgment. Users interpret the value based on their task characteristics.
Per-episode adaptive works well for most manipulation tasks. Typical pick-and-place, insertion tasks have clear bimodal distributions.
For special tasks (entirely slow operations), users should be aware that idle_ratio may be high. This itself is valuable diagnostic information.
What About Slow Tasks?
The Issue questioner's concern about "slow fine motion near the brick" is a real scenario.
If the task itself is like this, we suggest:
Don't just look at the absolute idle_ratio value. Compare idle_ratio across episodes of the same task — relative differences are more meaningful.
Look at action discontinuity alongside idle ratio. If idle_ratio is high AND action discontinuity is high, the data may have structural issues (e.g., control mode switching during teleoperation). If both are normal, it's likely just the task's inherent characteristic.
**Consider task-specific threshold tuning. **RDA's idle detection parameters are configurable (mad_multiplier, abs_threshold_floor), adjustable based on actual data.
Action Discontinuity: Another Perspective
Beyond idle detection, RDA also detects spikes in the action signal — sudden jumps that may indicate:
Control mode switching during teleoperation
Frame drops or interpolation issues in data collection
High-frequency controller oscillation
Spikes are detected on the second difference of the action signal (Δ²a), using MAD-based z-scores: z = 0.6745 × (x - median) / MAD, with a default threshold of |z| > 5.0.
This is also in raw action units, per-episode adaptive.
In our G1 dataset audit (300 episodes, 177,811 frames), 299/300 episodes had spikes (3,340 total). Combined with the 65.6% median idle ratio, this suggests the teleoperation data collection process has inherent characteristics worth understanding rather than "fixing."
Reflections
There's no universal "idle" definition.Different tasks, different hardware, different data collection methods — the understanding of "idle" varies. Tools can provide measurements, not judgments.
Adaptive thresholds come at the cost of uncertainty in edge cases.Bimodal detection works well on data with clear "still vs. moving" separation. It fails on "entirely slow" data. Users need to understand this boundary.
Open-source community value lies in these real discussions.The Issue made us re-examine idle detection's limitations and prompted us to think about how to better communicate applicable scenarios in documentation.
RDA is currently v0.9.12, and idle detection design is still iterating. If you have datasets with slow, precise operations, we welcome you to test it and discuss on GitHub.
Repository: github.com/liesliy/rda
Related Issue: huggingface/lerobot#4650
Top comments (0)