DEV Community: Harpreet Sahota

EmbeddingGemma 2 in FiftyOne: Search Video by Text, Sound, or Your Webcam

Harpreet Sahota — Thu, 29 Oct 2026 00:16:16 +0000

One model, one 768-dimensional space, three LanceDB indexes: search 10,000 VGGSound clips with a sentence, a soundtrack, or a recording of your own, in a few lines of Python or a few clicks.

You have 10,000 video clips and a sound stuck in your head. A church bell, maybe. The clips have no labels for it, or the labels are wrong, or the thing you want lives in the audio track that no image model ever looked at.

Searching media by meaning has usually meant stitching models together: one for frames, one for audio, one for text, and a vector store for each. Then you compare scores across models that were never trained to agree.

EmbeddingGemma 2, released by Google DeepMind on October 6, 2026, removes the stitching. One 740M-parameter model maps text, images, video, and audio into a single 768-dimensional space. The sentence “church bell ringing,” a photo of a bell tower, and the sound of a bell all land near each other.

This post runs it end to end in FiftyOne on VGGSound, a dataset of 10-second YouTube clips. You’ll build three LanceDB indexes over the same clips, search them with text, search pictures with sound, and then use the Cross-Modal Retrieval panel to do the same thing with an upload, your microphone, or your webcam.

Key takeaways

EmbeddingGemma 2 (740M parameters, Apache 2.0) embeds text, images, video, and audio into one 768-dimensional space, and its Matryoshka training lets you truncate vectors to 512, 256, or 128 dimensions.
In FiftyOne, EmbeddingGemma 2 is a remote zoo model: register it once, then model.embed_dataset() fills one vector field per modality.
With EmbeddingGemma 2, VGGSound ends up with three vector fields (eg2_video, eg2_audio, eg2_av) and three LanceDB similarity indexes built with fob.compute_similarity(..., backend="lancedb").
Because EmbeddingGemma 2 puts every modality in one space, a text query can search all three indexes, and a clip’s soundtrack can search other clips’ frames.
The same EmbeddingGemma 2 vectors drive uniqueness, representativeness, UMAP layouts, and zero-shot classification of VGGSound’s 309 sound classes.
The Cross-Modal Retrieval panel installs with two fiftyone plugin commands and lets you search by text, selected samples, or media you upload or record. You can also try it in a Hugging Face Space without installing anything.

What is EmbeddingGemma 2?

EmbeddingGemma 2 is Google DeepMind’s open multimodal embedding model. It’s built on the Gemma 4 architecture and combines a 270M-parameter text model with optional vision (170M) and audio (300M) encoders. You load only the encoders you need.

EmbeddingGemma 2 maps text, images, video, and audio into one shared space where matching meanings sit together.

One space. You can compare any vector to any other, so cosine similarity works between a sentence and a sound.
Matryoshka dimensions. The first 512, 256, or 128 numbers of a vector are a usable embedding on their own, so you can use smaller indexes without retraining anything.
An 8K-token context. Per the launch post, that fits up to 5.5 minutes of audio, 29 images, or 58 video frames in a single input.

What can you do with EmbeddingGemma 2?

One shared space turns several separate projects into the same three steps: embed, index, query.

Find clips by describing them. “Rain on a window” against a pile of unlabeled video.
Search across modalities. A sound finds pictures, and a picture finds sounds. In a grouped dataset, a video’s vector can pull the music that fits.
Pick what to label or drop. Uniqueness and representativeness scores computed from the vectors rank clips by how redundant or typical they are.
Label without training. Compare each clip’s vector to the text of your class names for zero-shot classification.
Search with your own media. Record a clap, a photo, or a few seconds of video and use it as the query.

The dataset: VGGSound

The VGGSound dataset in the FiftyOne App, with label counts, uniqueness and representativeness scores, and an embeddings view.

VGGSound is a set of 10-second YouTube clips, each labeled with one of 309 sound classes. The sound source is visible in the clip. This FiftyOne build holds the first archive shard: 10,000 of the 199,467 clips in the full release.

Sample fields you’ll use: ground_truth (a Classification with the sound class), youtube_id, start_sec, and split. The Hub copy also carries the vectors this post computes: eg2_video, eg2_audio, and eg2_av, each 768-dimensional.

Keep it honest. VGGSound labels come from an automated pipeline (visual verification, audio verification, iterative noise filtering), not from human annotators. Treat ground_truth as a strong signal that can still be wrong.

Step 1: Register the model

import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
from fiftyone import ViewField as F

# One-time: point FiftyOne at the remote zoo model source
foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/EmbeddingGemma2", overwrite=True
)

# Load the vision and audio encoders so one model handles video, audio, and both
model = foz.load_zoo_model(
    "google/embeddinggemma-2",
    media_type="video",
    modalities=["vision", "audio"],
)

The weights download on first use, and the repo is public, so you don’t need a Hugging Face login. It runs on CPU, and a GPU makes embedding a video dataset much faster.

Step 2: Load VGGSound

from fiftyone.utils.huggingface import load_from_hub

dataset = load_from_hub("harpreetsahota/VGGSound")

Good to know. Do you have to recompute the embeddings? No. The Hub copy ships with the three eg2_* vector fields and the uniqueness and representativeness scores. Skip Step 3 and go straight to Step 4, which builds the LanceDB indexes on your machine from those vectors. Step 3 is the code to run on your own media. The full download is 16.6 GB.

Step 3: Embed the clips three ways

The three EmbeddingGemma 2 vector fields on VGGSound, explored as UMAP layouts in the Embeddings panel.

# Frames only: sampled at 1 fps, up to 32 frames spread over the clip
model.embed_dataset(dataset, "eg2_video", modality="video")

# Soundtrack only: the first 30 seconds, resampled to 16 kHz mono
model.embed_dataset(dataset, "eg2_audio", modality="audio")

# Frames and soundtrack together, as one vector
model.embed_dataset(dataset, "eg2_av", modality="video+audio")

One clip, three indexes: video embeds the frames, audio the soundtrack, and video + audio both as one vector.

Each call reads the filepath of every sample and stores one VectorField. FiftyOne’s compute_embeddings() only accepts image and video collections, so embed_dataset() is the route for audio and for video plus audio. Clips that fail to decode, such as a video with no audio track, get None and a logged warning.

Step 4: Build a LanceDB index for each field

index_settings = {
    "eg2_video": {"media_type": "video"},
    "eg2_audio": {"media_type": "audio"},
    "eg2_av": {"media_type": "video", "modalities": ["vision", "audio"]},
}

for key, model_kwargs in index_settings.items():
    fob.compute_similarity(
        dataset,
        embeddings=key,                   # reuse the stored vectors
        model="google/embeddinggemma-2",  # pass the zoo model name so text queries work
        model_kwargs=model_kwargs,
        backend="lancedb",
        brain_key=key,
    )

Each index is an ordinary FiftyOne similarity index backed by its own LanceDB table, so image, video, audio, and video + audio indexes of the same dataset sit side by side.

Good to know. Where do the tables live? FiftyOne’s LanceDB backend reads its storage location from your brain config, and the default is /tmp/lancedb. Many systems clear /tmp on restart. Set a permanent location in ~/.fiftyone/brain_config.json before you build:

{
    "similarity_backends": {
        "lancedb": {"uri": "~/fiftyone/lancedb"}
    }
}

Why LanceDB for a multimodal index?

LanceDB is an open-source vector database with FiftyOne Brain support built in.

The dataset keeps the vectors, a brain run records the index, LanceDB holds one table per index, and the panel, the App, and Python all search it.

Embedded, no server. It runs inside the FiftyOne process and stores tables as files, on local disk or in cloud storage.
Built for ML data. Tables use the Lance columnar format, designed for fast random access to large vectors.
Scales past memory. Tables live on disk, so an index doesn’t have to fit in RAM.
One table per index. A video, an audio, and a video + audio index of the same clips don’t interfere with each other.

Step 5: Search with text

Text queries in the Cross-Modal Retrieval panel return matches from the video, audio, and video + audio indexes at once.

view = dataset.sort_by_similarity("church bell ringing", k=10, brain_key="eg2_av")

session = fo.launch_app(view)

Text queries are prefixed with the model’s SearchQuery prompt, which tells it the text is a search query rather than a document. Run the same query against each index, and each one ranks the clips by different evidence:

The query “church bell ringing” against the video, audio, and video + audio indexes.

In this 100-clip sample, the video and video + audio indexes put four church bell clips in their top eight. The audio index puts an orchestra first and finds bells at ranks 3, 5, and 6. Colored bars are clips labeled church bell ringing.

Step 6: Let a sound search videos

Audio and video queries in the Cross-Modal Retrieval panel find VGGSound clips by what they show and how they sound.

Take the soundtrack vector of one clip and search the frames-only index, which contains no sound at all:

# A clip labeled "female singing"
clip = dataset.match(F("ground_truth.label") == "female singing").first()

# Its soundtrack vector queries the frames-only index; the clip itself is excluded
matches = dataset.exclude([clip.id]).sort_by_similarity(
    clip["eg2_audio"], k=10, brain_key="eg2_video"
)

Searching the frames-only video index with a clip's audio vector finds other clips of the same class.

The chart above shows a sample of 100 clips. On the full 10,000 clips, the top 10 for a female singing query is dominated by singing: male singing, female singing, and child singing, plus a few neighbors such as playing ukulele and people cheering. The query never saw a frame, and the index never heard a sound.

Step 7: Find the clips worth your time

The vectors you just built also answer curation questions. eg2_av embeds frames and sound together, so it’s the one to use here:

# Uniqueness: high scores mark clips with few close neighbors
fob.compute_uniqueness(
    dataset, embeddings="eg2_av", uniqueness_field="eg2_av_uniqueness"
)

# Representativeness: high scores mark clips near the center of a cluster
fob.compute_representativeness(
    dataset,
    embeddings="eg2_av",
    representativeness_field="eg2_av_representativeness",
    method="cluster-center",
)

# A 2D layout you can explore in the Embeddings panel (needs umap-learn)
fob.compute_visualization(
    dataset, embeddings="eg2_av", method="umap", brain_key="eg2_av_viz", seed=51
)

Sort by eg2_av_uniqueness to find outliers worth a look. Sort by eg2_av_representativeness to pick a small set that stands in for the whole. The Hub copy already stores both scores as sample fields for each of the three vector fields, for example, eg2_audio_uniqueness.

Step 8: Classify without training anything

# Use the 309 VGGSound class names as the label set
model.classes = dataset.distinct("ground_truth.label")

# Compare stored vectors to the class text; no media is decoded again
model.classify_dataset(dataset, "eg2_zeroshot", embeddings_field="eg2_av")

# Score the zero-shot labels against ground truth
dataset.evaluate_classifications(
    "eg2_zeroshot", gt_field="ground_truth", eval_key="eg2_eval"
)

Open the Model Evaluation panel, and the confusion matrix shows which sound classes the model separates and which it mixes up.

Keep it honest. eg2_zeroshot is model output, not ground truth. Expect near-neighbor classes to mix. VGGSound lists dog barking, dog bow-wow, and dog growling as separate classes, and a zero-shot model has little to separate them with.

Try it without code: the Cross-Modal Retrieval panel

The Cross-Modal Retrieval panel runs the same searches from the App, and it lets you search with media you have in your hands right now. For how plugins and panels are built, see FiftyOne Plugins for MCAP and LeRobot Data Curation.

Install

fiftyone plugins download https://github.com/harpreetsahota204/cross_modal_similarity_search
fiftyone plugins requirements @harpreetsahota/cross-modal-retrieval --install

Requirements:

FiftyOne 1.22.0 or later
lancedb, transformers>=5.18, torch, av, and huggingface-hub, all installed by the second command
The EmbeddingGemma 2 weights in your model zoo. Download them once (about 1.5 GB) so your first search doesn’t stall:

import fiftyone.zoo as foz

foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/EmbeddingGemma2", overwrite=True
)
foz.download_zoo_model("google/embeddinggemma-2")

A GPU is optional. The webcam and microphone options need the App on localhost or https, which is a browser rule.

Don’t want to install anything?

Try the EmbeddingGemma 2 in FiftyOne Space on Hugging Face and use the panel in your browser.

Open the panel and build an index

The Build Cross-Modal Index form, where you choose what part of each sample becomes its EmbeddingGemma 2 vector.

Open a dataset in the App, click the + next to the grid’s tab, and choose Cross-Modal Search. Indexes you built in Python with this model, such as the ones in Step 4, show up in the panel. To make a new one, click Build Cross-Modal Index.

When you build an index, you choose what part of each sample becomes its vector:

Embed the samples as	What goes into the vector	Finds samples by
Image	the image	what they show
Video	frames sampled from the video, with the sound ignored	what they show
Audio	the soundtrack, or the audio file	what they sound like
Video + audio	frames and soundtrack together, as one vector	both at once

The four ways to embed samples when you build a cross-modal index, with what goes into each vector and what it finds samples by.

The form also lets you pick Dimensions (768, 512, 256, or 128), frame sampling for video, and how many seconds of audio to use. Building runs in the background if you have delegated operations set up.

Search with text

Text searches in the Cross-Modal Retrieval panel, with a colored chip for each index's results.

Click New Search. Under Search against, pick one or more indexes. Each returns its own matches. Choose Text, describe what you want, and click Search. Try “church bell ringing,” “rain on a window,” or “a crowd cheering.”

The matches open in the grid, best first. A search across several indexes returns a colored chip for each index, and you can click chips to turn each index’s results on or off.

Search with other samples

Searching with selected samples, which the panel averages into one query.

Select samples in the grid, then choose Samples and search. The selected samples are excluded from the results. Several selected samples are averaged into one query, so selecting three clips of dogs barking finds “dogs barking” better than a single example does.

For video samples, embed your query as chooses what part of the clip becomes the query: its frames, its sound, or both. Search an audio index with the clip's frames to find what sounds like what the clip shows.

Search with your own media

Under Media, use something that isn’t in the dataset. Nothing you upload or record is added to the dataset.

Upload an image, video, audio, or MCAP file.
The microphone records up to 30 seconds of audio.
The webcam records up to 30 seconds of video with sound or takes a single photo.

A microphone recording searches the VGGSound clips directly, without being added to the dataset.

The browser asks for permission the first time you use the webcam or microphone.

Tune the search and keep your results

Set a Similarity threshold, and the panel returns every sample from any index whose cosine similarity to your query meets it. The default is 0.65, which suits most searches, and strong cross-modal matches usually score between 0.6 and 0.9. Lower it if a search comes back empty. Each index returns at most 500 results.

Every search is saved on the panel’s home page. Click one to bring its results back, Clone one to tweak and rerun it, or Delete the ones you don’t need.

Saved searches on the Cross-Modal Retrieval panel's home page, ready to reopen, clone, or delete.

Because every index is a standard FiftyOne similarity index, the App’s own similarity search and view.sort_by_similarity() work on the same indexes the panel uses.

Keep it honest

This is one shard of VGGSound. Everything here uses 10,000 of the 199,467 clips.
Read scores as a ranking. Text-to-clip similarity sits in a narrow band, and matching clips score only slightly higher than the rest. Compare scores within one search, not across searches.
Audio-only is the weakest index on VGGSound. The class names sound sources that are visible in the frames, so the video and video + audio indexes find them more reliably. Audio alone does better when the classes are defined by sound.
128 dimensions costs quality. The model card reports a large drop for multimodal tasks at 128. Use 768 or 512 unless storage forces your hand.
Retrieval figures in this post come from a 100-clip sample. The church bell and sound-to-frames charts illustrate behavior and aren’t benchmark results.

Try it yourself

EmbeddingGemma 2 in FiftyOne Space: the panel on VGGSound, no install
Cross-Modal Retrieval plugin: the panel and operators
EmbeddingGemma 2 FiftyOne integration: the remote zoo model, with a notebook
VGGSound on Hugging Face: 10,000 clips with vectors, uniqueness, and representativeness scores
EmbeddingGemma 2 model card and Google’s launch post
FiftyOne Brain similarity docs

Frequently asked questions

What is EmbeddingGemma 2?

EmbeddingGemma 2 is an open, 740M-parameter embedding model from Google DeepMind, released under Apache 2.0. It maps text, images, video, and audio into a single 768-dimensional vector space so that you can compare any modality with any other.

How do I use EmbeddingGemma 2 in FiftyOne?

To use EmbeddingGemma 2 in FiftyOne, register the remote zoo source with foz.register_zoo_model_source("https://github.com/harpreetsahota204/EmbeddingGemma2"), then load google/embeddinggemma-2 with foz.load_zoo_model(). Use dataset.compute_embeddings() for images and video, or model.embed_dataset() for audio and video plus audio.

What is cross-modal retrieval?

Cross-modal retrieval is searching one media type with another, such as finding video clips with a text query or finding pictures with a sound. It works when one model embeds every modality into the same space, which EmbeddingGemma 2 does.

How do I build a multimodal index in FiftyOne with LanceDB?

To build a multimodal index in FiftyOne with LanceDB, store embeddings in a vector field, then call fob.compute_similarity(dataset, embeddings="eg2_av", model="google/embeddinggemma-2", model_kwargs={...}, backend="lancedb", brain_key="eg2_av"). Each index becomes its own LanceDB table so that you can keep video, audio, and video + audio indexes of the same dataset together.

Can I search video by its sound?

Yes, EmbeddingGemma 2 can search video by its sound. Build an audio index (media_type="audio") or a video + audio index (modalities=["vision", "audio"]) and query with text, or query with another clip’s eg2_audio vector.

Do I need a GPU to run EmbeddingGemma 2?

No. EmbeddingGemma 2 runs on CPU in float32 and uses bfloat16 on CUDA. A GPU makes embedding a large video dataset much faster. float16 is refused because it returns NaN or degraded vectors.

Which embedding dimension should I use?

Start with 768, the full EmbeddingGemma 2 dimension. Use 512 or 256 to cut storage and search time. At 128, multimodal quality drops noticeably.

Where does FiftyOne store LanceDB vectors?

FiftyOne stores LanceDB vectors in the location set under similarity_backends.lancedb.uri in ~/.fiftyone/brain_config.json, which defaults to /tmp/lancedb. Point it somewhere permanent before you build indexes.

Why can’t I record from my webcam or microphone?

Browsers allow camera and microphone access only on localhost or https. Run the App locally or serve it over https.

One Camera Isn’t Enough: 9 Robotics Datasets You Can Play on One Timeline

Harpreet Sahota — Thu, 29 Oct 2026 00:10:05 +0000

We combine event cameras, radar, thermal, tactile pads, and lidar into nine robotics datasets, each converted into FiftyOne multimodal episodes for synchronized sensor analysis.

One of the six sequences in the DSEC sample has a mean image brightness of 26.5. The other five range from 78.5 to 92.5.

Same car, same cameras. The color images in that sequence are about a third as bright as the rest, and a pipeline tuned on the other five hasn’t seen anything like it. DSEC also records the road with a stereo pair of event cameras and ships lidar-derived disparity as ground truth, so when the color image degrades, you still have other sensors to check it against.

That is the setup for every dataset in this post. A robot that leaves the lab encounters real-world conditions such as low light, water, snow, global navigation satellite system (GNSS) dropouts, and objects it must touch. These datasets, with ground truth or robot state, help you evaluate and improve your algorithms effectively.

Each of the nine is a FiftyOne multimodal dataset: one .fo.mcap file per episode, with every sensor stream on one shared clock, making it straightforward to access and work with the data.

Dataset	Platform	Sensors beyond RGB	In this release	License
DSEC	Car	Stereo event cameras, LiDAR	6 of 53 sequences	CC BY-SA 4.0
M3ED	Car, quadrotor, and Spot	Stereo event cameras, LiDAR	3 sequences, one per robot	CC BY-SA 4.0
Edged-USLAM	Quadrotor	DAVIS346 event camera	All 13 runs	CC BY 4.0
ColoRadar	Handheld rig	Two FMCW radars, LiDAR	7 of 52 sequences	Apache 2.0
CitrusFarm	Clearpath Jackal	Thermal, red-green-NIR, depth, LiDAR	1 of 7 sequences	CC BY-SA 4.0
HapTile	UR5e arm	Vision-based tactile sensors, RGB-D	1,699 episodes	CC BY 4.0
NTNU Underwater Multi-Camera	BlueROV2 Heavy	Five monochrome cameras, barometer, rangefinder	All 8 runs	BSD 3-Clause
GrandTour	ANYmal D quadruped	HDR and depth cameras, LiDAR, GNSS/INS	3 of 49 missions	MIT
TUM RGB-D	Handheld and Pioneer robot	Kinect color and depth	47 main-table sequences	CC BY 4.0

The nine multimodal robotics datasets, with platform, sensors beyond RGB, how much of each release is included, and license.

Key takeaways

DSEC, M3ED, and Edged-USLAM record every event as a foxglove.PointCloud window of 1/30 s with x, y, z (milliseconds since the window opened), and polarity. In the spot_indoor_stairwell sequence, it runs at 63.8 Hz, a high temporal resolution.
ColoRadar, CitrusFarm, and HapTile cover radar, thermal and near-infrared (NIR), and touch: two frequency-modulated continuous-wave (FMCW) radars on a handheld rig, a thermal camera and a red-green-NIR camera on a Clearpath Jackal, and gel-pad tactile video on a UR5e gripper across 1,699 episodes.
GrandTour and NTNU Underwater Multi-Camera test assumptions about the environment. The indoor GrandTour mission ETH-2 has 0 GNSS fixes, against 44,601 in the forest mission ALB-3, and NTNU’s cameras sit behind a housing port that bends light underwater.
TUM RGB-D offers 47 sequences spanning six categories, including nine with moving people and eight designed to separate structure from texture, showcasing the datasets' broad applicability for various research needs.
Each of the nine robotics datasets is a FiftyOne dataset with media_type="multimodal" and plain sample fields such as peak_event_rate_mev_s, lighting, max_depth_m, and haptic_feedback, so you can sort and filter recordings before opening a single file.
Most of the nine robotics datasets are samples of larger releases: 6 of 53 DSEC sequences, 3 M3ED sequences, 7 of 52 ColoRadar sequences, 3 of 49 GrandTour missions, and 1 of 7 CitrusFarm sequences.

What does a camera that reports changes record?

An event camera doesn’t produce frames. Each pixel reports when its brightness changes, and nothing otherwise.

Three of the nine datasets use one, and they use it in three different places.

DSEC: the driving sequence where the color image goes dark

Voxel51/DSEC-Sample on Hugging Face | source: DSEC

DSEC in the FiftyOne multimodal viewer

DSEC is a driving dataset from the University of Zurich, featuring stereo Prophesee event cameras, global-shutter color cameras, LiDAR disparity ground truth, and optical flow in some sequences, totaling 6 sequences and over 2.2 billion events.

M3ED: one sensor head on a car, a quadrotor, and a Spot

Voxel51/M3ED-Sample on Hugging Face | source: m3ed.io

M3ED in the FiftyOne multimodal viewer

M3ED comes from the GRASP Laboratory at the University of Pennsylvania. The sensor head has a stereo pair of Prophesee EVK4 HD event cameras at 1280x720, a grayscale stereo pair, a color camera at 1280x800, an inertial unit, and an Ouster OS1-64 LiDAR. The robot changes, and the rig doesn’t. The sample contains one sequence per robot: 192 seconds and 3,631,437,863 events total. The Spot sequence alone, spot_indoor_stairwell, holds 1,883,139,312 events in 98.9 seconds.

Edged-USLAM: what an event camera records as the lights drop below 5 lux

Voxel51/Edged-USLAM-Event-Camera on Hugging Face | source: sebnem-byte/drone-navigation-event-camera

Edged-USLAM in the FiftyOne multimodal viewer

Edged-USLAM is a DAVIS346 event camera on a quadrotor flying in a motion-capture room, recorded for an ICRA 2026 paper. Five of its 13 runs vary the lighting: 30% lighting, 60% lighting, blinking lights, strong side light, and under 5 lux. The low_lit run has the lowest mean event rate of the five: 0.53 million events per second, compared with 0.69 to 0.80 for the others. The other eight runs include fly lines, squares, circles, aggressive turns, and manual flights, with the vehicle's Vicon pose recorded alongside them.

Good to know. How are events stored in an MCAP file? Each message holds one 1/30 s window as a foxglove.PointCloud. x and y are pixel coordinates, z is the time since the window opened (ms), and polarity is 1 for a brightness increase and 0 for a decrease. The message is stamped when its window closes, so an event’s time is the stamp, minus 1/30 s, plus z. A 3D panel set to the event's frame shows each window as a volume of pixel position against time. Each camera also gets an /event-frames channel, a video render with ON events white and OFF events black on gray.

What do radar, thermal, and touch see that a color camera can’t?

ColoRadar: two radars and a lidar, carried through an underground mine

Voxel51/ColoRadar-Sample on Hugging Face | source: ColoRadar

ColoRadar in the FiftyOne multimodal viewer

ColoRadar comes from the Autonomous Robotics and Perception Group at the University of Colorado Boulder. A handheld rig carries two FMCW radars, a TI MMWCAS-RF-EVM cascaded imaging radar and a TI AWR1843BOOST single-chip radar, next to an Ouster OS1-64 LiDAR and a Lord Microstrain 3DM-GX5-25 inertial unit. The release includes 52 sequences across seven settings: hallways, a lab, a large motion-capture space, outdoor built environments, the narrow and wide passages of an underground mine, and a fast ride along paths and roads. The sample takes the shortest sequence of each place: 7 sequences, 13m 46s, 8,263 LiDAR scans holding 382.6 million points, 4,145 cascaded radar frames, and 8,340 single-chip radar scans.

Keep it honest. The cascaded radar publishes a heatmap of 128 range by 128 azimuth by 32 elevation cells, not a point cloud. The /cascade-radar-points channel is derived from those heatmaps the way the release’s plotting tool derives them. It is not raw sensor output. The /cascade-radar-bev channel is a top-down render of the same heatmaps.

CitrusFarm: a Jackal in the orchard rows with thermal and near-infrared

Voxel51/CitrusFarm-Sample on Hugging Face | source: CitrusFarm

CitrusFarm in the FiftyOne multimodal viewer

CitrusFarm is a ground robot in the rows of a citrus farm, from the ARCS Lab at the University of California Riverside. A Clearpath Jackal carries a FLIR Blackfly monochrome camera, a FLIR ADK thermal camera, a Mapir Survey3 camera that records red, green, and near-infrared, a Stereolabs ZED 2i with depth, a Velodyne LiDAR, a Microstrain inertial unit, and a Piksi GPS-RTK receiver. The sample is the smallest of seven sequences: 06_14B_Jackal, 5m 03s over 357 m, with 3,024 thermal frames, 3,025 red-green-NIR frames, 2,998 LiDAR scans, and 3,024 GPS-RTK fixes. The ZED depth holds values for 47% of pixels (valid_depth_fraction = 0.4703).

HapTile: 1,699 episodes with the operator’s haptic feedback recorded

Voxel51/HapTile on Hugging Face | source: HapTile2026/HapTile

HapTile in the FiftyOne multimodal viewer

HapTile moves the question from seeing to touching. A UR5e arm with a Robotiq 2F-85 gripper works through 38 contact-rich tabletop tasks under teleoperation. Each finger carries a vision-based tactile sensor that films a marker-grid-printed gel pad, so contact shows up as markers moving. Two RGB-D cameras watch the scene, and we record the operator's haptic feedback alongside the robot state. That is 1,699 episodes, 664,045 frames, and 12.36 hours.

The feedback levels live in a haptic_feedback list on each sample. Firm appears in 1,481 episodes, soft in 575, and mild in 414. The 47 move_mobile_box episodes have no haptic channel at all.

What breaks when the robot goes underwater, indoors, or onto snow?

NTNU Underwater Multi-Camera: five cameras behind a housing port

Voxel51/NTNU-Underwater-Multicam on Hugging Face | source: ntnu-arl/underwater-datasets

NTNU Underwater Multi-Camera in the FiftyOne multimodal viewer

NTNU’s dataset is Ariel, a BlueROV2 Heavy from the Autonomous Robots Lab, piloted by hand through the Trondheim Fjord (6 runs) and the lab’s Marine Cybernetics Laboratory pool (2 runs). An Alphasense rig records five monochrome cameras at 720x540 and 20 Hz. The autopilot logs a second inertial unit, a barometer, a downward rangefinder, the battery, and eight thruster outputs. Each run includes a reference trajectory the authors computed with ReAqROVIO. In total, that is 63m 19s, 379,896 camera frames, and 759,943 inertial samples. The fjord runs reach 7.1 to 7.7 m deep. The two pool runs reach 0.5 and 0.6 m.

Keep it honest. The release calibrates the stereo pair underwater and the other three cameras in air only. Refraction at the housing port changes intrinsics once the vehicle is submerged, so those three carry in-air values. The depth_m track applies the release’s freshwater barometer mapping, which reads a little deep in the fjord’s salt water.

GrandTour: the same robot in a forest, a building, and snow

Voxel51/GrandTour-Sample on Hugging Face | source: leggedrobotics/grand_tour_dataset

GrandTour in the FiftyOne multimodal viewer

GrandTour is an ANYmal D quadruped from the Robotic Systems Lab at ETH Zurich carrying the Boxi sensor payload: HDR and depth cameras, a Hesai LiDAR, a tactical-grade inertial unit, and a GNSS/INS receiver, with a total station tracking a prism on the payload for reference. The release holds 49 missions. The sample carries 3 chosen to contrast: ALB-3 in a forest, ETH-2 inside the ETH main building, and SNOW-1 on the Jungfraujoch.

The indoor mission is where GNSS disappears. The GNSS field reads Working for the forest and the snow and No for ETH-2, which has 0 fixes against 44,601 and 37,201. The robot’s /legged-odometry, /lidar-odometry estimate, and /gnss-ins-odometry solution are separate pose channels, so you can overlay three estimates of the same walk and see where each one drifts from the others.

Keep it honest. The sample includes only the three HDR cameras and the upper front depth camera. The Alphasense cameras, ZED 2i, other depth cameras, Livox and Velodyne LiDARs, and the point cloud maps are in the full release but not here. For ETH-2, the release notes say its HDR timestamps are imprecise because they use the GMSL2 arrival time. That note is carried in the release_notes field.

What do you grade a trajectory against?

Every dataset in this post ships an answer key, and they are not the same kind of key. TUM RGB-D is the plainest case.

TUM RGB-D: a Kinect graded by an eight-camera motion-capture system

Voxel51/TUM-RGBD on Hugging Face | source: TUM RGB-D benchmark

TUM RGB-D in the FiftyOne multimodal viewer

TUM RGB-D comes from the Computer Vision Group at the Technical University of Munich. A Microsoft Kinect records color and depth at 640x480 and 30 Hz while an eight-camera motion-capture system tracks it at 100 Hz. Three Kinects, named freiburg1, freiburg2, and freiburg3, are carried handheld and on a Pioneer robot over scenes built to test structure against texture, moving people, and object reconstruction.

This conversion includes the 47 sequences in the benchmark’s main table: 9 freiburg1, 18 freiburg2, and 20 freiburg3, totaling 48m 12s, with 81,413 color frames and 80,683 depth frames. The category field groups them into Handheld SLAM (11 sequences), 3D Object Reconstruction (11), Dynamic Objects (9), Structure vs. Texture (8), Robot SLAM (4), and Testing and Debugging (4).

Keep it honest. The validation sequences, whose ground truth the benchmark withholds, are not included. The freiburg3 sequences have no accelerometer samples because the benchmark’s archives hold none.

What each dataset grades you against

Dataset	Ground truth
DSEC	LiDAR-derived disparity for both stereo pairs, and optical flow on some sequences
M3ED	Left event camera ground-truth poses and depth
Edged-USLAM	Vicon pose of the vehicle
ColoRadar	LiDAR-inertial trajectory, plus a Vicon pose on `aspen_run9`
CitrusFarm	A ground-truth position track
HapTile	Robot state: end-effector pose, joints, commanded targets, and gripper
NTNU	ReAqROVIO reference trajectory
GrandTour	Total-station prism positions, plus the three odometry estimates
TUM RGB-D	Motion-capture pose of the Kinect at 100 Hz

The ground truth or reference signal that ships with each of the nine multimodal robotics datasets.

Step 1: Load a multimodal robotics dataset from Hugging Face

Each dataset loads from the Hugging Face Hub with one call.

import fiftyone as fo
import fiftyone.utils.huggingface as fouh
from fiftyone import ViewField as F

dsec = fouh.load_from_hub(
    "Voxel51/DSEC-Sample",
    name="DSEC-Sample",
    persistent=True,
)

print(dsec.media_type)  # multimodal
print(len(dsec))        # 6

One sample is one sequence, and filepath points to its .fo.mcap file. The datasets have no FiftyOne label fields. Every signal, including ground truth, is a channel inside the file.

Step 2: Query the sample fields before you open a file

The sample fields summarize each recording, so you can find the one you want first.

# The sequence with the darkest color images
dsec.sort_by("mean_image_brightness").first().sequence  # zurich_city_09_b

# The Edged-USLAM run flown under 5 lux
edged = fouh.load_from_hub(
    "Voxel51/Edged-USLAM-Event-Camera",
    name="Edged-USLAM-Event-Camera",
    persistent=True,
)
dark = edged.match(F("lighting") == "under 5 lux")

# HapTile episodes where the operator felt firm contact
haptile = fouh.load_from_hub("Voxel51/HapTile", name="HapTile", persistent=True)
firm = haptile.match(F("haptic_feedback").contains("firm"))  # 1,481 episodes

Then launch the App and open a sample:

session = fo.launch_app(dsec)

The multimodal viewer lays out the channels as tiles with a shared playback clock, so a color frame, an event window, and a ground-truth plot move together.

Try it yourself

Each dataset above links to its Hugging Face repo and its source release. Pick the one that breaks your current pipeline, load it with the snippet in Step 1, and open a sample in the App.

Frequently asked questions

What is a FiftyOne multimodal dataset?

A FiftyOne multimodal dataset is a dataset with media_type="multimodal", where each sample is one recording stored as an MCAP file. The App’s viewer reads the file's channels and displays them as tiles on a single timeline. Each sample also carries ordinary fields, such as duration, num_events, or ground_truth_path_m, that you can query like any other FiftyOne field.

Do these datasets have labels I can evaluate a model against?

The nine robotics datasets have ground truth, but not as FiftyOne label fields. The ground truth, such as disparity, poses, depth, or reference trajectories, comes from each source release and lives in channels such as /ground-truth and /disparity-event. Nothing here is model output. The one derived channel is /cascade-radar-points in ColoRadar.

Are these the full datasets?

Four of the nine robotics datasets are complete, and five are samples. TUM RGB-D carries the benchmark’s 47 main-table sequences, NTNU carries all 8 runs, Edged-USLAM carries all 13 runs, and HapTile carries 1,699 episodes. DSEC (6 of 53 sequences), M3ED (3 sequences), ColoRadar (7 of 52), GrandTour (3 of 49), and CitrusFarm (1 of 7) are samples. The cards for those say what was left out.

What licenses do they use?

The nine robotics datasets use the licenses of their source releases. NTNU is BSD 3-Clause. M3ED, DSEC, and CitrusFarm are CC BY-SA 4.0. TUM RGB-D, Edged-USLAM, and HapTile are CC BY 4.0. ColoRadar is Apache 2.0, and GrandTour is MIT.

Which one should I start with?

Start with the robotics dataset that breaks your current pipeline. Low light points to Edged-USLAM, GNSS-denied walking to GrandTour ETH-2, underwater refraction to NTNU, and contact to HapTile.

Robots Can See. They Still Can't Feel: 8 Tactile LeRobot Datasets in FiftyOne

Harpreet Sahota — Thu, 29 Oct 2026 00:09:04 +0000

A field guide to touch as a robot-learning modality, plus 8 LeRobot v3.0 datasets that record it (1,679 episodes of gel-sensor video, taxel grids, and fingertip force/torque), loaded into FiftyOne with their exact keys and quirks.

Try tying your shoes with numb fingers. You can watch your hands the whole time and still fumble it, because the information you need lives at the point of contact. Is the lace slipping? Is the knot tight? How hard am I pulling? Your eyes can't answer any of that. Your fingertips answer it constantly, and you never notice.

That's roughly where most robot learning sits today. Vision-language-action models have gotten good at the "where": where's the cup, where's the gripper, where should the arm go next. They're still mostly blind to what's happening at the contact surface.

Tactile sensing fills that gap. Getting good tactile data, and then making sense of it, is one of the more interesting data problems in physical AI right now.

We pulled together 8 tactile datasets that ship as LeRobot v3.0, from a bimanual Flexiv rig packing earbuds into cases to a handheld gripper with two taxel pads, and loaded each one into FiftyOne. This guide covers why touch matters, what tactile data looks like on disk, why it's hard to collect and harder to read, and what to check before you train on it. Each dataset links to its FiftyOne version, its source repo, and its license.

Key takeaways

8 tactile LeRobot datasets, 1,679 episodes, and 1,877,818 frames, all loadable with fiftyone.utils.huggingface.load_from_hub. Xense's earbud-case insertion set alone holds 1,213,518 of those frames (65%).
Tactile data shows up in three shapes. 7 of the 8 store gel-sensor touch as video under observation.images.*, 4 carry force/torque vectors, and only Grabette stores taxel grids (6×6 and 4×8 int16). Tactile video plays in FiftyOne like any camera, and numeric tactile streams can be plotted in the episode viewer, but they stay in parquet and don't become sample fields you can filter on.
There's no naming standard for tactile keys, even inside these 8 datasets. observation.tactile is a 60-D fingertip force/torque vector in SharpaDex and the prefix of a taxel grid in Grabette.
In the one SharpaDex season here (assemble_gears_on_base, 410 episodes), the right index finger and right thumb register contact in 61% and 42% of 302,020 frames. Five of the ten fingertips rarely do.
Licenses: Apache-2.0 for the three Xense sets and Grabette, CC-BY-4.0 for SharpaDex, and none declared for the three Intern sets.
The evidence that tactile input helps robot policies is real but conditional. LeFlexiTac reports 7/30 → 23/30 on an in-bag pen task, FreeTacMan reports 21% → 71% average success, and an open T-Rex issue measured an offline benefit within ±0.9% on the full Sharpa data.

Why touch matters for robot learning

In all four cases the camera frames look the same. The tactile signal is where the difference shows up.

Vision fails at the moment of contact. When a gripper closes on an object, the fingers usually block the camera's view of what you care about. Inserting a plug, threading a cable, unscrewing a cap: the critical action happens in a region the camera can't see well.

Slip is invisible. An object starting to slide out of a grasp often doesn't look like anything until it's already falling. A tactile sensor picks up the shear well before that.

Force is not a pixel. Wiping a table, peeling a sticker, or holding an egg all need the right amount of pressure. You can't reliably infer pressure from an RGB frame.

Materials feel different than they look. A soft plastic bottle and a rigid one can look identical. They respond very differently when squeezed.

The early results back this up. LeFlexiTac added a FlexiTac sensor (a 12×32 taxel map) to the low-cost SO-100/SO-101 arm and fed it to four policy families through a single observation.tactile.primary stream. On an in-bag pen retrieval task, where the camera can't see what the gripper is reaching for, vision-only Action Chunking with Transformers (ACT) succeeded 7 of 30 times. With tactile input, it succeeded 23 of 30. FreeTacMan ran five contact-rich tasks with ACT: vision-only averaged 21% success, adding tactile input raised that to 55%, and adding their visuo-tactile pretraining raised it to 71%.

Keep it honest. An open issue on the T-Rex GitHub repo reports training T-Rex on the full Sharpa dataset (64 GPUs, six epochs) and comparing real tactile input against neutral input on 64 fixed samples. Overall action mean squared error (MSE) dropped from 0.0768 to 0.0295 across checkpoints, but the benefit of real touch over neutral input swung between −0.90% and +0.89%. That's offline action prediction, not closed-loop success, which is where touch is supposed to pay off. My read isn't that touch doesn't help. Touch helps when the task needs it and when the data is clean enough for a model to learn from it. That second condition is where most of the work is.

Three shapes of tactile data

The three shapes of tactile data across these 8 LeRobot datasets, with an example of each and how to view it.

"Tactile" isn't one data type. Depending on the sensor, it lands on disk in one of three shapes, and each one needs a different way of looking at it:

Images or video from camera-based gel sensors. These at least look like something, though a raw gel frame is hard to read without context.
Low-resolution arrays from taxel pads and gloves, such as a 12×32 or 6×6 grid of pressure values. You can't view these as an image until you turn them into a heatmap.
Time series of force and torque. These only make sense plotted over time, next to what the robot was doing.

How a gel sensor turns a press into an image

How a camera-based gel sensor turns a press into an image, and which LeRobot keys store the raw frame and the difference image.

Most of the touch in this roundup comes from camera-based sensors. A soft gel with a reflective coating sits on a clear support. LEDs light it from the edges, often in three colors, and a camera underneath films the coating. When an object presses in, the coating bends, the shading shifts, and the camera records it. Subtract a frame taken with nothing touching the sensor, and you get a map of where the gel moved.

That's why datasets often ship more than one stream per sensor. SharpaDex stores observation.images.tactile_raw (the camera's view), observation.images.tactile_deform (a deformation map), and a 60-D force/torque signal in observation.tactile. The Intern sets store observation.images.tactile_*_aug_diff, which by its name looks like a difference image, though the card only calls them "touch images."

No standard sensor

Four tactile sensor families, what each one outputs, and how each one wears out or fails.

Vision has converged on "an RGB camera." Touch hasn't converged on anything. You've got camera-based gel sensors (GelSight, DIGIT, Xense), taxel pads, fingertip sensors that report six-axis force and torque, and full-hand pressure gloves. The OmniViTac dataset alone uses four different visuo-tactile sensor types. Each produces data with a different shape, resolution, and meaning, and each fails in its own way.

Gels tear and degrade. The lighting inside camera-based sensors shifts. Baselines move between sessions. If you've done any work with lab assays, you'll recognize this immediately: it's a batch effect. Data collected on Tuesday with a fresh gel doesn't look like data collected on Friday with a worn one, and a model will happily learn that difference instead of the task.

Why tactile data is hard to collect and harder to read

Where tactile data goes wrong at each stage, from collection to training.

Teleoperators can't feel what the robot feels. Most demonstrations come from a human driving a robot remotely. Without haptic feedback, operators guess at contact forces, so demonstrations are only as good as those guesses. Projects such as HapTile are starting to feed haptic signals back to the operator, but that's still the exception.

Human gloves have an embodiment gap. Tactile gloves let you collect natural human manipulation at scale. But a human hand isn't a robot hand, and a glove's pressure map doesn't translate cleanly to a two-finger gripper or a 22-degree-of-freedom (DoF) dexterous hand.

Scale is still small. Open X-Embodiment pooled more than 1 million real robot trajectories across 22 embodiments. The largest public tactile datasets are in the thousands to tens of thousands of episodes: SharpaDex v1.0 has 28,993, and T-Rex has 5,464. The 8 datasets in this roundup add up to 1,679. Several announced tactile datasets, such as RoboTacDex and DexViTac, say they'll release their data later.

Formats are inconsistent, even within one standard. LeRobot has become the default format for robot-learning data, which helps. It doesn't settle where touch goes.

The same key, observation.tactile, holds a force vector in SharpaDex and a taxel-grid prefix in Grabette.

Inside these 8 datasets alone, observation.tactile means a force vector in one and a taxel-grid prefix in another. Xense names its overhead camera head in two sets and top in the third. All three Intern sources stored their task text as a pandas index in meta/tasks.parquet, which had to be rewritten before the standard loader would read it. Their tactile video was MJPEG inside MP4, which browsers can't play, so the FiftyOne versions are H.264 re-encodes. Outside this list, T-Rex encodes its tactile streams losslessly in an H.264 profile most browsers can't decode either.

None of these are bugs exactly. They're reasonable decisions made by different people. Together they mean that before you can learn anything from tactile data, you spend a lot of time getting it to show up correctly.

Synchronization is easy to get wrong. Tactile sensors, cameras, and robot state often run at different rates. If the tactile spike from a grasp lands a few frames away from the gripper-close event, the model learns the wrong cause and effect.

The FiftyOne LeRobot schema for tactile data

All 8 datasets are loaded into FiftyOne with fo.types.LeRobotDataset, the same way as in our 11-dataset LeRobot v3.0 roundup. Each one is a multimodal dataset where one sample equals one episode, with the same sample-level fields: media_reference, episode_index, task, tasks, length, duration, robot_type, and fps. Per-frame data stays in the LeRobot data/*.parquet and videos/*/*.mp4 files, and media_reference points into them.

What changes for touch is where each shape ends up:

Tactile shape	Example key	In FiftyOne
Gel-sensor video	`observation.images.left_tactile_left`	A video stream in the episode viewer, next to the RGB cameras
Force/torque vector	`observation.tactile` (SharpaDex), `observation.wrench` (Intern)	Per-dimension series in the viewer's Plot and Message tiles. Stays in `data/*.parquet`, not a sample field
Taxel grid	`observation.tactile.right_sensor_1` (Grabette)	Flattened to one series per taxel (36 for a 6×6 grid) in the Plot and Message tiles. Stays in `data/*.parquet`, not a sample field

Where touch lands when a LeRobot dataset is loaded into FiftyOne.

Where touch lands when a LeRobot dataset is loaded into FiftyOne.

Good to know. Can you filter on numeric tactile data? Not directly. The episode viewer reads every numeric feature from parquet, so you can plot any taxel or force axis over time and step through exact values. FiftyOne's queries and aggregations, though, run on sample fields in its database, and LeRobot per-frame data deliberately stays in the source files behind media_reference. To sort or filter episodes by touch, reduce each stream to a few per-episode numbers and write them back as fields (see the curation section below). A plot tile also shows a grid as 36 separate lines, not a heatmap, so render it yourself if you want to see the contact patch.

Loading any of the 8 takes two lines:

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

dataset = load_from_hub("Voxel51/xense-strap-tape-120ep")
session = fo.launch_app(dataset)

The 8 tactile datasets at a glance

All 8 are real-world data, and 7 are complete imports of their source repos. The exception is SharpaDex: the FiftyOne version is one collection season (19.6 GB) of a 3.4 TB release.

Dataset	Task	Tactile streams	Episodes/frames	Robot	License
Xense Earbud Case	Insert earbuds into cases, close the lid, and box them	4 gel videos (400×700)	834 / 1,213,518	Bimanual Flexiv Rizon 4	Apache-2.0
Xense Strap Tape	Grab the tape and strap it	4 gel videos (400×700)	120 / 191,261	Bimanual Flexiv Rizon 4	Apache-2.0
Xense Tron2 RT Pick and Place	Put the block into the box	4 gel videos (400×700)	115 / 48,999	`tron2_rt`	Apache-2.0
Intern Pour	Pour water from a cup into a beaker	2 touch-image videos (700×400) + 6-D wrench	51 / 44,042	7-DoF arm + Robotiq gripper, GELLO teleop	None declared
Intern Screw	Unscrew a cap from a bottle a person is holding	2 touch-image videos + 6-D wrench	50 / 31,960	Same	None declared
Intern Shiguan	Insert a test tube into a rack	2 touch-image videos + 6-D wrench	51 / 26,665	Same	None declared
SharpaDex Assemble Gears	Gear assembly on a base	Raw + deformation video, 60-D fingertip force/torque	410 / 302,020	Bimanual 7-DoF arms + 22-DoF hands	CC-BY-4.0
Grabette Tactile	Put the white cube in the correct cup	2 taxel grids (6×6, 4×8 int16)	48 / 19,353	Grabette handheld gripper	Apache-2.0

The 8 tactile LeRobot v3.0 datasets, with task, tactile streams, size, robot, and license.

The 8 tactile LeRobot v3.0 datasets, with task, tactile streams, size, robot, and license.

The 8 tactile datasets in detail

Xense: three bimanual sets, four tactile streams each

Xense Robotics builds visuo-tactile sensors and publishes datasets recorded with its own LeRobot fork, lerobot-xense, a LeRobot v5.1 fork that adds Flexiv Rizon4, Elite CS66, and ARX5 arms plus tactile grippers. All three sets share one layout: 3 RGB cameras, 4 tactile video streams (a left and right pad on each gripper, 400×700), and a 20-D state and action vector (left_tcp.{x,y,z,r1..r6}, right_tcp.{x,y,z,r1..r6}, and both gripper positions), all at 30 fps. Every stream is standard H.264 yuv420p, so the tactile video plays in a browser with no transcoding.

Keep it honest. None of the three cards name the tactile sensor or say what the tactile pixels physically mean.

Xense Earbud Case: 834 episodes of long-horizon insertion

An Xense Earbud Case episode in FiftyOne, with the RGB cameras and tactile streams playing in sync.

FiftyOne Format Dataset on Hugging Face: Voxel51/xense-earbud-case-834ep
Original Dataset: Xense/earbud_case_insertion_teleop_0515
License: Apache-2.0

This is the biggest dataset in the roundup by a wide margin: 834 teleoperated episodes and 1,213,518 frames on a bimanual Flexiv Rizon 4 (bi_flexiv_rizon4_rt). Each episode runs the whole sequence: pick up an earbud case from the stands, insert the matching earbuds, close the lid, and place the case in a box. Episodes run 32.3 to 78.1 seconds. During insertion, fingers hide the contact from the cameras, making this a good set for asking what the tactile streams add.

Fields you get: 3 RGB streams (head, left_wrist, right_wrist, 480×640), 4 tactile streams (left_tactile_left, left_tactile_right, right_tactile_left, right_tactile_right, 400×700), and the 20-D observation.state and action.

Known quirks. The source repo is still growing. An earlier snapshot listed 726 episodes and 1,065,060 frames, and this import has 834, so pin a revision if you need reproducible numbers. At 6.9 GB, it's also the largest download of the three Xense sets.

Xense Strap Tape: deformable tape, the longest episodes

Xense Strap Tape in FiftyOne, with gripper positions plotted next to the four tactile streams.

FiftyOne Format Dataset on Hugging Face: Voxel51/xense-strap-tape-120ep
Original Dataset: Xense/strap-tape-taccap-0929
License: Apache-2.0

120 episodes of "Grab the tape and strap the tape" on the same bimanual Flexiv rig, 191,261 frames in total. Tape is deformable, so the grip changes as it stretches and sticks. Episodes run 33.9 to 95.2 seconds, the longest in the roundup. At 0.84 GB, it's the smallest download with tactile video and the easiest place to start.

Fields you get: the same 3 RGB streams, 4 tactile streams, and 20-D state and action as the earbud set.

Known quirks. The taccap in the repo name probably refers to Xense's TacCap-Gripper, a wearable two-finger tactile data-collection device the company debuted at ICRA 2026, but the card doesn't say so. The episode count has also changed over time (an earlier snapshot showed 98).

Xense Tron2 RT Pick and Place: short episodes, a different robot

Xense Tron2 RT Pick and Place in FiftyOne, from the sample grid to a single episode.

FiftyOne Format Dataset on Hugging Face: Voxel51/xense-tron2rt-pnp-115ep
Original Dataset: Xense/tron2rt-pnp-0918
License: Apache-2.0

115 episodes of "Put the block into the box" on robot type tron2_rt, 48,999 frames in total. Episodes are short (10.7 to 22.2 seconds), so this is the quickest Xense set to scrub through when you want to see what a grasp looks like across all four tactile pads.

Fields you get: 3 RGB streams (top, left_wrist, right_wrist), 4 tactile streams, and the same 20-D tool center point (TCP) state and action layout.

Known quirks. The overhead camera is observation.images.top here and observation.images.head in the other two Xense sets, so code that hard-codes camera keys will break across them. Unlike its siblings, the card carries no code link or citation.

Intern: three GELLO-teleoperated tasks with touch images and wrench

The three Intern sets are community uploads by cloudfan that share one robot schema, intern_gello_7dof_robotiq: a 7-DoF arm with a Robotiq gripper, teleoperated with a GELLO leader arm. Each episode carries two RGB views (cam_high, cam_front), two tactile image streams (tactile_left_aug_diff, tactile_right_aug_diff, 700×400), a 6-D observation.wrench with per-axis observation.wrench_valid flags, an 8-D joint observation.state, an 8-D GELLO action, observation.eef_pose, and the original source_timestamp.

The source README says these are meant for training with the InternVLA A-series project. The robot config that ships with them maps only the two RGB views and describes the tactile and wrench channels as "preserved as auxiliary data, not automatically added to the RGB baseline." Out of the box, the touch data is along for the ride.

Keep it honest.

No license is declared on any of the three source repos, so you can't assume reuse rights.
No episode was filtered for success. The task string is the goal, not a guarantee that the demonstration achieved it.
The FiftyOne versions are lossy H.264 re-encodes (libx264, -crf 18) of the source MJPEG video. Frame counts and timestamps match the source, but the pixels don't match bit for bit.

Intern Pour: 51 pouring demonstrations, 3 with missing force

Intern Pour in FiftyOne, with the tactile streams and gripper command plotted over a pour.

FiftyOne Format Dataset on Hugging Face: Voxel51/intern-pour-lerobot-51ep
Original Dataset: cloudfan/intern-pour-lerobot
License: None declared in the source repo

51 demonstrations of "Pick up the small glass cup by its handle, pour the water into the large beaker, and place the cup back on the table," 44,042 frames at 30 fps, 21.6 to 39.3 seconds each. Pouring changes the weight in the gripper as the water leaves, which is a force signal no camera frame carries directly.

Fields you get: the shared Intern layout above.

Known quirks. Episodes 13, 34, and 42 have missing force readings that were filled with zeros, and observation.wrench_valid flags them. Filter on the validity flags, not on the zeros, or those frames will look like moments of no contact.

Intern Screw: unscrewing a cap from a bottle a person holds

Intern Screw in FiftyOne, with the gripper command plotted next to the tactile streams as the cap comes off.

FiftyOne Format Dataset on Hugging Face: Voxel51/intern-screw-lerobot-50ep
Original Dataset: cloudfan/intern-screw-lerobot
License: None declared in the source repo

50 demonstrations of "Unscrew the cap from the bottle held by the person and place the cap on the table," 31,960 frames, 17.6 to 43.5 seconds each. The bottle is held by a person, not clamped, so the counter-force comes from a moving human hand.

Fields you get: the shared Intern layout above.

Known quirks. None beyond the shared ones. Every frame has valid wrench readings.

Intern Shiguan: test-tube insertion

Intern Shiguan in FiftyOne, with wrench forces plotted during a test-tube insertion.

FiftyOne Format Dataset on Hugging Face: Voxel51/intern-shiguan-lerobot-51ep
Original Dataset: cloudfan/intern-shiguan-lerobot
License: None declared in the source repo

"Shiguan" is the pinyin for 试管, test tube. 51 demonstrations of "Pick up the test tube from the left side of the rack and insert it into the hole at the right end of the rack," 26,665 frames, 11.0 to 26.0 seconds each. It's the shortest of the three Intern tasks and the closest to a classic peg-in-hole insertion.

Fields you get: the shared Intern layout above.

Known quirks. None beyond the shared ones. Every frame has valid wrench readings.

SharpaDex Assemble Gears: dexterous hands, ten fingertip sensors

SharpaDex Assemble Gears in FiftyOne, with the raw and deformation tactile streams next to the RGB views.

FiftyOne Format Dataset on Hugging Face: Voxel51/sharpa-assemble-gears-410ep
Original Dataset: Sharpa-Robotics/SharpaDex-v1.0 (28,993 episodes, 59 tasks, 495 seasons, about 300 hours)
License: CC-BY-4.0

SharpaDex v1.0 is teleoperated bimanual dexterous manipulation on a rig with two 7-DoF arms, two 22-DoF hands, four RGB cameras, and tactile sensors on every fingertip. The full release covers assembly, tool use, deformable objects, cleaning, material transfer, and long-horizon tasks, exported as both LeRobot v3.0 and v2.1.

This FiftyOne version is one collection season (season_POC22027_2026_04_30_15_04_50_train) of the assemble_gears_on_base task: 410 episodes, 302,020 frames at 30 fps, 17.4 to 36.7 seconds each. It's a single contiguous season, not a random sample of the task or the release.

Fields you get: 4 RGB streams (head_left, head_right, wrist_left, wrist_right, 480×480), 2 tactile video streams (tactile_deform at 480×1200 and tactile_raw at 480×1600), a 65-D observation.state, action, and observation.state.joint_torque (left arm, left hand, right arm, right hand, and torso), a 24-D observation.state.tcp (pose and force/torque per arm), observed and commanded TCP poses, a per-frame subtask_index, and the 60-D observation.tactile: left and right thumb, index, middle, ring, and little fingertips, each with fx, fy, fz, tx, ty, tz. The task field holds structured text with Task, Instruction, Scene, and Success sections.

Known quirks.

Touch is concentrated in two fingers. With a contact threshold of 1.0 on per-fingertip force magnitude, the right index finger crosses it in 61% of frames and the right thumb in 42%. Left ring (9.5%) and left little (2.2%) come next, and the other five fingertips rarely cross it. That's consistent with a right-hand pinch doing the gear work, but check it against the tactile video before you trust the channel labels.
robot_type is the literal string robot in the source info.json.
The source README says the export doesn't fully declare TCP units, reference frames, or rotation convention, so don't infer Euler angles or quaternions from vector width alone.
subtask_index survives in the parquet, but the subtask text and skill labels it points to live in the source's meta/subtasks.parquet, which the FiftyOne LeRobot exporter doesn't carry.

Keep it honest. The T-Rex issue mentioned earlier, the one that found almost no offline benefit from real touch, trained on the full Sharpa dataset. A 410-episode slice can't settle that question, but it's a good place to look at what the tactile channels contain.

Grabette Tactile: a handheld gripper with two taxel pads

Grabette Tactile in FiftyOne. Taxel values plot as one series per cell, and episode 2 reads zero throughout.

FiftyOne Format Dataset on Hugging Face: Voxel51/grabette-tactile-full-synced-48ep
Original Dataset: CarolinePascal/grabette-tactile-full-synced
Additional Resources: Grabette release post · ACT policy trained on a derived version: CarolinePascal/act_tactile_grabette
License: Apache-2.0

Grabette is an open, low-cost handheld gripper from Pollen Robotics, released on the Hugging Face blog in July 2026 and inspired by Stanford's Universal Manipulation Interface (UMI). You hold it, perform the task by hand, and its cameras, inertial measurement unit (IMU), and gripper encoders record the demonstration. Simultaneous localization and mapping (SLAM) recovers the 6-DoF trajectory, and the output is a standard LeRobot dataset. A matching motorized gripper, Gripette, runs the learned policy on a robot arm.

This dataset adds two taxel sensors to the right gripper. It has 48 episodes of "Pick up the white cube and put it in the correct cup," 19,353 frames at 50 fps, 2.9 to 10.7 seconds each. It's the only dataset in the roundup that stores touch as a raw pressure grid.

Fields you get: 2 RGB streams (right_cam0, right_cam1, 960×720), an 8-D action (right_x, right_y, right_z, right_ax, right_ay, right_az, right_proximal, right_distal), an is_lost flag, and two int16 taxel grids, observation.tactile.right_sensor_1 (6×6) and observation.tactile.right_sensor_2 (4×8).

Known quirks.

The taxel grids show up in the episode viewer as one plottable series per taxel, but they aren't sample fields (see the schema section above). The arrays are in data/chunk-000/file-000.parquet.
Episode 2 has no tactile signal at all: all 68 taxels read 0 for all 145 frames. It's also the shortest episode, so it may be an aborted take.
The card doesn't state units, taxel layout, or the sensor model. Across all 19,353 frames, the raw values range from 0 to 1,774 on right_sensor_1 and 0 to 2,133 on right_sensor_2, and every taxel varies at some point, so no cells are dead.
is_lost is 0 in every frame. It likely flags SLAM tracking loss, which the Grabette pipeline checks for before conversion, but the card doesn't define it.
The release post describes Grabette as a robot-free, by-hand capture device and doesn't mention tactile sensing. The card doesn't say how they recorded these 48 episodes.

Curating tactile data: where the real gains are

This is the part I care most about, because it's where a small amount of effort pays off the most.

Six checks to run on any tactile dataset before you train on it.

Six things worth checking in any tactile dataset:

Dead or flatlined sensors. A taxel that reads zero for an entire episode, or a gel camera that's frozen, is pure noise. Flag those channels and drop them.
Saturation. If a sensor is pinned at its maximum, you've lost the information about how hard the contact actually was.
Drift across sessions. Compare at-rest baselines across collection days. If they wander, normalize per session, or at least know which episodes came from which sensor state.
Episodes where touch never happens. If the tactile signal barely moves during an episode, that episode teaches the model nothing about contact. It may be fine for vision training and useless for tactile learning.
Contact alignment. Check that tactile onset lines up with what the cameras and gripper state say is happening. A misalignment is a sync problem, not a learning problem.
The rare, valuable moments. Slips, regrasps, and near-failures are where touch matters most, and they're usually a small fraction of the data. Find them and make sure they're represented.

Split by episode, not by frame.

Frame-random splits leak near-identical frames into the test set. Holding out whole contact sequences cost RCT 17.7 points of Recall@1.

Consecutive tactile frames are nearly identical. The RCT authors measured what that does to evaluation: with the encoder held fixed, and only the test split changed, tactile-to-text Recall@1 was 80.0% under a frame-random split and 62.3% once whole contact sequences were held out, a 17.7-point drop. Random frame splits will make your tactile model look much better than it is.

Step 1: Turn numeric touch into per-episode fields

FiftyOne's queries and aggregations run on sample fields, and per-frame LeRobot data stays in parquet, so this step uses pandas to read the frames and FiftyOne to store, sort, and filter the results. Here's the pattern on SharpaDex: read observation.tactile, compute per-fingertip force magnitude, and write two per-episode fields back onto the samples.

import glob

import numpy as np
import pandas as pd
import fiftyone as fo

dataset = fo.load_dataset("sharpa_assemble_gears")
root = "sharpa-assemble-gears-410ep"  # local snapshot of the Hub repo

df = pd.concat(
    pd.read_parquet(f, columns=["episode_index", "observation.tactile"])
    for f in glob.glob(f"{root}/data/*/*.parquet")
)

# (frames, 10 fingertips, fx fy fz tx ty tz)
ft = np.stack(df["observation.tactile"].to_numpy()).reshape(-1, 10, 6)
force = np.linalg.norm(ft[:, :, :3], axis=-1)

df["peak_force"] = force.max(axis=1)
df["in_contact"] = (force > 1.0).any(axis=1)

stats = df.groupby("episode_index").agg(
    peak_force=("peak_force", "max"),
    contact_fraction=("in_contact", "mean"),
)

ep = dataset.values("episode_index")
dataset.set_values("peak_tactile_force", stats.loc[ep, "peak_force"].tolist())
dataset.set_values("contact_fraction", stats.loc[ep, "contact_fraction"].tolist())

session = fo.launch_app(dataset.sort_by("contact_fraction"))

The threshold of 1.0 is a starting point, not a calibrated contact definition, since the README doesn't declare units. Plot the distribution before you pick one.

Step 2: Look at the extremes

In this season, every one of the 410 episodes has contact for between 39% and 93% of its frames (median 64%). Per-episode peak force ranges from 10.9 to 48.9 in the sensor's raw units. Sorting by contact_fraction puts episodes 327, 318, and 326 at the top, all near 39%. Those are the first ones to open in the App and watch next to their tactile video. A low value could mean a fast, clean assembly or a fumbled grasp, and only watching tells you which.

The same pattern works for the other datasets. On the Intern sets, compute the fraction of frames where observation.wrench_valid is all ones. On Grabette, compute the per-episode peak and the per-taxel variance of each grid, so a dead cell shows up as a zero.

None of this is exotic. It's the same discipline that worked for image datasets: look at your data, find the problems, fix them, find the gaps, fill them. For touch, that means catching drift, dead sensors, and misaligned timestamps before they reach training.

Frequently asked questions

What is tactile data in robot learning?

Tactile data is anything a robot measures at the point of contact. In LeRobot datasets, it shows up as video from camera-based gel sensors (for example observation.images.left_tactile_left in the Xense sets), low-resolution pressure grids (observation.tactile.right_sensor_1 in Grabette), or force/torque vectors (observation.tactile in SharpaDex, observation.wrench in the Intern sets).

Which tactile datasets can I load directly into FiftyOne?

All 8 in this roundup load with load_from_hub("Voxel51/<repo>"). They are xense-earbud-case-834ep, xense-strap-tape-120ep, xense-tron2rt-pnp-115ep, intern-pour-lerobot-51ep, intern-screw-lerobot-50ep, intern-shiguan-lerobot-51ep, sharpa-assemble-gears-410ep, and grabette-tactile-full-synced-48ep. Each one is a multimodal FiftyOne dataset with one sample per episode.

Can I see Grabette's tactile grids in the FiftyOne App?

Yes, as time series. The episode viewer reads every numeric LeRobot feature, so observation.tactile.right_sensor_1 (6×6) and observation.tactile.right_sensor_2 (4×8) appear as one series per taxel in the Plot and Message tiles. They aren't sample fields, so to filter episodes by touch, compute per-episode statistics with pandas and write them back with set_values. For a spatial view of the contact patch, render the grids as heatmaps yourself.

Is observation.tactile the same thing across datasets?

No. In SharpaDex, observation.tactile is a 60-D vector: 10 fingertips × 6-axis force/torque. In Grabette, observation.tactile.* is the prefix for two int16 taxel grids. Check meta/info.json for shape and dtype before you assume what a tactile key holds.

What does aug_diff mean in the Intern tactile streams?

The cards don't define it. The source README calls tactile_left_aug_diff and tactile_right_aug_diff "700 x 400 RGB touch images." The name suggests a difference from a reference frame, which is a common way to show where a gel sensor deformed, but treat that as a guess until the uploader documents it.

Why were the Intern videos re-encoded?

The source stores video as MJPEG inside MP4, which browsers can't decode, so it wouldn't play in the FiftyOne App. All four streams in every episode were re-encoded to H.264 (libx264, -crf 18, yuv420p). Frame counts and timestamps match the source, but it's a lossy re-encode, so go back to the source repo if you need the original pixels.

Which of these datasets can I use commercially?

The three Xense sets and Grabette are Apache-2.0, and SharpaDex is CC-BY-4.0. Both licenses permit commercial use under their terms (attribution for CC-BY). The three Intern sets declare no license, so you can't assume any reuse rights.

Does adding tactile input always improve a robot policy?

No. LeFlexiTac (7/30 → 23/30 on an in-bag pen task) and FreeTacMan (21% → 71% average success across five tasks) report large gains on contact-rich tasks. An open issue on the T-Rex repo found that the offline benefit of real touch over neutral input on the full Sharpa data stayed within ±0.9%. Touch tends to help when the task depends on contact and when the tactile data is clean.

How to Find Edited Copies of Images: Image Copy Detection with pHash, CLIP, and FiftyOne

Harpreet Sahota — Thu, 29 Oct 2026 00:02:58 +0000

Exact-duplicate search misses a mirrored, cropped, or filtered copy of an image, so this post combines a perceptual hash and a CLIP embedding into one rule you control and shows the copies as a graph in FiftyOne.

Your model's test metrics look great. Then someone opens the test set and finds a mirrored copy of a training image. Then another one, with a filter on it. Then a third, cropped and covered in emoji.

You run a duplicate check. It finds nothing.

That's because duplicate checks look for the same bytes, and an edited copy has different bytes. A mirrored JPEG is a different file from the original. To a hash of the file contents, the two are strangers. To a person, they're the same picture.

The task is called image copy detection. Given a query image, decide whether it's an edited copy of an image in a reference collection, and say which one. This post walks through one practical approach: compute two signals per image, combine them with a rule you control, and read the result as a graph where every link comes with evidence. It's packaged as an open-source FiftyOne plugin called Image Link Lab, but the idea carries over to any dataset where copies hide behind edits.

Image Link Lab in FiftyOne. Tune the pHash and CLIP limits, and the copy graph redraws as you drag.

Key takeaways

Image copy detection asks whether a query image is an edited copy of a reference image, and which one. compute_exact_duplicates can't answer that, because edits change the bytes.
A perceptual hash (pHash) summarizes pixel layout in 64 bits. It tolerates recompression and small blurs and breaks on mirrors, rotations, and big crops.
A CLIP embedding (clip-vit-base32-torch from the FiftyOne model zoo) summarizes what the picture shows. It survives mirrors and crops and gets fooled by look-alikes.
Each signal's blind spot is the other's strength, so the plugin computes both and lets you choose how they combine: link a pair if either signal passes (combine: "any"), or only if both do ("all").
Moving a limit changes the answer. The Image Link Lab panel redraws the links as you drag the sliders and shows the numbers behind every line. Given a truth field, it scores each rule with FiftyOne's native evaluation so you can compare rules in Model Evaluation.
The output is ordinary FiftyOne data: a phash field, a copy_of Classification field, and copies_* evaluation runs. You can tag copies, save views, and filter a test split cleanly with the code you already use.

What is image copy detection?

Image copy detection is the task of deciding whether a query image is a modified copy of any image in a reference corpus. The edits can be mundane (crop, resize, or re-compress) or hostile (mirror, overlay text, blend with another image, or screenshot it on a different device).

The standard benchmark is DISC21, the dataset behind the Image Similarity Challenge at NeurIPS 2021. Its authors describe the goal as determining "whether a query image is a modified copy of any image in a reference corpus of size 1 million," with edits that include automated transformations, hand-crafted changes, and machine-learning-based manipulations. Both the query set and the reference set are mostly distractors that match nothing, a needle-in-a-haystack setting like real life. The paper is The 2021 Image Similarity Dataset and Challenge.

Copy detection is a different question from "which images look like this one?" A similarity search returns neighbors. Copy detection has to commit: this query is a copy of that original, or it isn't a copy of anything.

Five edited copies of one image are five different files, so exact-duplicate search finds none of them.

Where edited copies cost you

You don't need a content moderation team to care about this. Four situations come up again and again:

Train/test leakage. A mirrored copy of a training image in your test set inflates your metrics. You don't find out until the model meets real data.
Labeling cost. You pay to label the same picture several times, sometimes with conflicting answers.
Licensing and attribution. You need to know where an image came from, and a cropped copy hides its source.
Moderation and forensics. A known-bad image comes back filtered, or shows up on a second device.

The first two hit any training pipeline. The last two are what DISC21 was built around: its authors say the benchmark mimics social media problems such as misinformation and objectionable content.

Why exact duplicates and single embeddings fall short

FiftyOne already handles two nearby problems. compute_exact_duplicates finds byte-identical files. compute_near_duplicates, along with the Similarity Search panel, finds neighbors by a single embedding.

Neither is built for edited copies. Exact matching fails the moment a pixel changes. A single embedding gives you a ranked list of look-alikes, with one similarity score per pair and no way to say which kind of evidence you trust. If the score is wrong, you can't tell whether the embedding was fooled or the copy was over-edited.

What's missing is a way to combine two different kinds of similarity with a rule you control, and a way to see which image is linked to which, and why.

Two signals, two blind spots

The approach starts from a simple observation. Pixels and meaning fail on different edits.

A perceptual hash (pHash) summarizes an image's pixel layout in 64 bits. Two images whose hashes differ in only a few bits look the same at a glance. It's fast, needs no model, and shrugs off recompression and a small blur. It's defeated by anything that moves pixels around: a mirror, a rotation, or a big crop.

A CLIP embedding summarizes what the picture shows. Two images with a high cosine similarity are about the same thing. It survives mirrors and crops. It's fooled by look-alikes: two different photos of the same kind of scene can score higher than a heavily blurred copy of the same photo.

Pixels and meaning fail on different edits, so pHash and CLIP each catch copies the other misses.

The four pairs above show it. A mirrored copy is invisible to pHash and obvious to CLIP. A striped and blurred copy is the reverse. A look-alike that isn't a copy nearly gets linked by CLIP. And a flipped, blurred, filtered copy beats both.

Good to know. Why not train a copy-detection model? You can, and the DISC21 challenge exists to push that line of work. The two-signal approach is for the common case where you have a dataset on disk, no labeled copy pairs, and a question to answer this afternoon. Both signals work off the shelf.

Combine them with a rule you control

Each signal gets a limit. For pHash, it's the most bits two hashes may differ by. For CLIP, it's the least cosine similarity two images must reach. Then you choose how the signals combine:

Either signal is enough (any): link the pair if pHash or CLIP passes. You get more links, and more wrong ones.
Both must agree (all): link only if both pass. You get fewer links, and almost all of them right.

Moving a limit is the whole game. Loosen the pHash limit, and you catch more heavily edited copies and start linking images that only look alike. Tighten it to all, and the wrong links disappear along with most of the right ones. No setting gets everything.

That's why the plugin doesn't give you a single "is copy" score. It shows you where the trade-off sits for your data and lets you pick.

Keep it honest. There is no universal threshold. A limit that works on re-uploaded product photos will behave differently on a collection full of heavy crops and overlays. If you have even a small set of known copy pairs, use it to calibrate the rule.

Where FiftyOne and its plugin ecosystem fit

FiftyOne is an open-source tool for building high-quality datasets and visual AI models. It gives you the dataset layer: samples with fields you can filter and sort on, views, tags, the App for looking at your images, and native tools for similarity search and model evaluation.

It can't ship every check that depends on your data. That's the job of plugins.

A FiftyOne plugin is a folder FiftyOne discovers at runtime. It includes a manifest (fiftyone.yml), Python code, and optionally a React front end. You add a button, a form, a background job, or a whole panel to the App without forking FiftyOne or waiting for a release. Plugins come in two kinds:

Operators are actions. You trigger one, it runs, it finishes.
Panels are UI that stays open next to your data and reacts when you filter the grid.

Plugin code runs on the FiftyOne server in Python so that it can use NumPy, PyTorch, a GPU, or any library you can pip install. Long jobs can be delegated to a background worker so the App stays free. Every operator also has a URI, so anything that can call it, from a button to a notebook to an AI agent, runs the same code. For a deeper tour of how operators and panels work, see Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins.

Image Link Lab is built from standard parts: two operators (compute_signals, find_copies), one hybrid panel (image_link_lab), native similarity indexes, and native evaluation runs. The plugin adds only what FiftyOne lacks: a perceptual hash, a rule over two signals, and the graph.

The plugin adds three things FiftyOne lacks: a perceptual hash, a rule over two signals, and the graph. Everything else is native.

Step 1: Install the plugin and compute signals

Install the plugin and its requirements:

fiftyone plugins download https://github.com/harpreetsahota204/image_link_lab
fiftyone plugins requirements @harpreetsahota/image_link_lab --install

The plugin needs fiftyone>=1.22.1. CLIP runs on CPU but is much faster on a GPU.

Open any image dataset in the App, click + next to Samples, and add the Image Link Lab panel under Custom. Click Compute signals, keep both boxes ticked, and execute.

Computing the pHash and CLIP signals with the Image Link Lab operator.

The operator writes three things:

Where	What
`phash` field	the 64-bit pHash of each image, as hex
brain key `phash`	the hash bits as a native similarity index, so sort-by-similarity in the App works on pHash too
brain key `clip`	a native CLIP similarity index (`clip-vit-base32-torch` from the model zoo)

What the compute signals operator writes to your dataset: a phash field and two native similarity indexes.

It runs delegated by default, so start a worker with fiftyone delegated launch or choose "Execute now" in the form. pHash takes about a minute per 10,000 images, plus CLIP time. Signals that are already computed are left alone unless you tick them.

Step 2: Find copies

Click Find copies and fill in the form.

Filling in the find copies form: queries, candidate originals, an optional truth field, and the starting rule.

Query images are the ones you want explained. Each is linked to at most one original.
Candidate originals are where to look. Both can be the current view, the whole dataset, or a tag. They can overlap, and an image is never linked to itself.
ID field (optional) is what to call each image, if your dataset identifies images by something other than the sample ID.
Truth field (optional) holds, for each query, the ID of its known original. This is your ground truth, supplied by your dataset, not computed by the plugin. Set it, and you get scoring. Leave it empty, and the plugin works unsupervised: links are drawn, nothing is judged.
Rule sets the starting pHash limit, CLIP limit, and any or all. You'll change these in the panel, so this is just where the first result comes from.

Behind the form, the operator does three things:

Gathers candidates. For each query, it takes the 5 nearest originals by pHash and the 5 nearest by CLIP, plus the true original if a truth field is set. It stores both signals' values for every candidate. This is the slow step, about 4 seconds for 1,000 queries against 5,000 originals, and every later run reuses it.
Applies the rule and writes copy_of, a Classification per query whose label is the linked original's ID, or none. copy_of is a prediction from your rule, not a human label. When several candidates pass, the one passing the most signals wins, then the highest CLIP similarity, then the fewest differing bits.

Scores, if a truth field is set.

The slow step, gathering candidates, runs once. Every rule you try after that reuses them.

Step 3: Read the graph

The panel draws whatever queries are in the current grid view, up to 40 at a time. Filter the grid and the graph redraws.

Nodes are images. Copies have a gray border. Originals have an orange border and, when more than one copy points at them, a badge with the count.
Lines are candidate pairs the rule has something to say about. With a truth field set, green is a correct link, red is an incorrect link, and dashed amber is a missed original. Thick lines pass both signals, and thin lines pass one.
Images joined by lines form families, laid out as a star with the original in the middle. A loose rule joins families through wrong links, and you'll see red lines bridging two stars.
Queries with nothing to draw sit in a strip at the bottom. For a distractor with no original, that's the correct outcome.

Click an image and the grid filters to that image and everything it has a line to, including originals your current view doesn't contain. Click a line and an evidence pane opens with the copy and the original side by side, then one row per signal:

pHash   pixels    30 bits differ    limit ≤ 10    fails
CLIP    meaning   0.731 similar     limit ≥ 0.90  fails

Below the rows is a verdict in a sentence, such as "Not linked: pHash and CLIP are outside the limit. This is the true original, so the rule missed it." You can check every statement against the numbers above it. No model decides behind the scenes.

Move the sliders. The rule runs in the browser against the stored candidate values, so lines and counts update as you drag, with no server round trip.

Good to know. Does the graph work without ground truth? Yes. Without a truth field, every link is drawn green, and nothing is dashed. The colors mean "linked," not "right."

Step 4: Score the rule and compare alternatives

If you set a truth field, click Score this rule. The plugin runs the rule over every query, not just the 40 on screen, writes copy_of, and records a native FiftyOne evaluation named after the rule, such as copies_phash10_or_clip90. Then it opens Model Evaluation.

The evaluation is binary, per query. Ground truth is copy if the query has a known original and unique if it doesn't. The prediction is copy if the rule linked the query to its true original, and unique otherwise. FiftyOne computes precision, recall, F1, and a 2×2 confusion matrix from those two labels.

One definition matters: a copy linked to the wrong original counts as a miss, not a false positive. The evaluation asks whether you found the true original, and the answer is no.

The loop looks like this:

Score the starting rule and open its evaluation in Model Evaluation.
Click the false-negative cell. The grid filters to the misses, and the graph redraws them with dashed amber lines to their originals. Click a line to see which signal failed and by how much. That tells you which slider might help.
Change a limit, score again, and you get a second evaluation key.
Open one and compare it with the other.
Mark the rule you keep as Reviewed and delete the rest with dataset.delete_evaluation(key).

The false-positive cell is the other one worth clicking. Those are distractors the rule linked, usually look-alikes, which is CLIP's failure mode. Comparing a pHash-only run with a CLIP-only run shows which edits each signal survives.

Keep it honest. Model Evaluation scenarios, which break results down by edit type, are a Voxel51 feature. In open-source FiftyOne, filter the grid by an edit-type field and read the chips, or compute the numbers in Python.

Step 5: Act on the results

copy_of is an ordinary label field, so everything else in FiftyOne works on it:

from fiftyone import ViewField as F

copies = dataset.match(F("copy_of.label") != "none")
copies.tag_samples("copy")
dataset.save_view("copies", copies)

# Keep a test split clean
clean_test = dataset.match_tags("test").match(F("copy_of.label") == "none")

# Everything linked to one original
dataset.match(F("copy_of.label") == "R003513")

Typical setups differ only in what you pass as queries and originals:

Situation	Queries	Originals	Truth
Dedup a collection	whole dataset	whole dataset	none
Check a test split for leakage	tag `test`	tag `train`	none
Match seized images against a known-bad list	the seized images	the known-bad list	none, or a case label
Benchmark a rule	the edited images	the originals	the original's ID

Typical Image Link Lab setups, with what to pass as queries, originals, and truth for each.

Run the same workflow from Python

Everything the App does is an operator, so a notebook, a script, or an agent can run it:

import fiftyone as fo
import fiftyone.operators as foo

dataset = fo.load_dataset("disc21_link_lab")

# Step 1: signals (delegate=False runs it in this process)
foo.execute_operator(
    "@harpreetsahota/image_link_lab/compute_signals",
    {"dataset": dataset, "params": {"phash": True, "clip": True, "delegate": False}},
)

# Step 2: copies, with a rule, scored against a truth field
result = foo.execute_operator(
    "@harpreetsahota/image_link_lab/find_copies",
    {
        "dataset": dataset,
        "params": {
            "queries": "tag:final_query",      # or "view", "dataset", "tag:<tag>"
            "originals": "tag:reference",
            "id_field": "disc21_id",           # optional
            "truth_field": "reference_id",     # optional; turns on scoring
            "phash_max": 10,
            "clip_min": 0.90,
            "combine": "any",                  # "any" or "all"
            "delegate": False,
        },
    },
)
print(result.result)

# The results are ordinary FiftyOne fields and runs
dataset.load_evaluation_results("copies_phash10_or_clip90").print_report()

Run find_copies again with a different rule and the candidates are reused, so it takes about a second and adds another evaluation run to compare.

Every step is an operator with one URI, so the App, a notebook, and an AI agent all run the same code.

Both operators have typed inputs and show up through the FiftyOne Model Context Protocol (MCP) server (list_operators, get_operator_schema, execute_operator), so an AI agent can compute signals, try a rule, and read back precision and recall.

How it's built

The plugin is small on purpose. The Python operators and the pure-Python core (hashing, candidate gathering, the rule) are tested with pytest. The panel UI is built in React with VOODO, the component library the FiftyOne App itself uses.

The rule exists twice on purpose: in Python for the operators and in TypeScript so the sliders can run it in the browser. A shared fixture, tests/fixtures/rule_cases.json, feeds both test suites, so the two can't drift.

Four layers, each calling only the one below. A shared test fixture keeps the Python and TypeScript rules in sync.

Everything else is native FiftyOne. CLIP comes from compute_similarity, the pHash bits are registered as a second similarity index, results are plain Classification fields, scoring is evaluate_classifications, and rule history lives in evaluation runs.

Keep it honest. The demo dataset, disc21_link_lab, is a subset of DISC21: 1,000 edited queries and 5,000 originals, with a known original for 500 of the queries. The rest are distractors. The full benchmark has a reference corpus of 1 million images, so don't read results from this subset as benchmark numbers. DISC21's edits are also deliberately brutal (heavy crops, overlays, and blends), so recall will be low on it. On a collection of ordinary re-uploads, pHash alone will catch most copies.

Try it yourself

Frequently asked questions

What is image copy detection?

Image copy detection is the task of deciding whether a query image is an edited copy of an image in a reference collection, and identifying which one. It differs from similarity search, which returns neighbors without committing to a match.

How do I find edited or modified copies of an image in a dataset?

Compute more than one kind of similarity, since no single signal survives every edit. Image Link Lab computes a perceptual hash in the phash field and a CLIP similarity index, then links pairs with a rule over both. find_copies writes the result to a copy_of field.

Can perceptual hashing detect mirrored or cropped images?

Not reliably. A pHash summarizes how pixels are laid out, so anything that moves pixels around (a mirror, a rotation, or a big crop) defeats it. A CLIP embedding survives those edits, which is why the plugin uses both.

What's the difference between exact duplicates, near-duplicates, and copy detection in FiftyOne?

compute_exact_duplicates finds byte-identical files. compute_near_duplicates finds neighbors by a single embedding. Image Link Lab links each query to at most one original using two signals and a rule you set, and shows the evidence for every link.

How do I check a test split for train/test leakage?

Run find_copies with queries set to tag:test and originals set to tag:train. Then match copy_of.label != "none" on the test split to see which test images link to a training image.

Do I need ground truth labels?

No. Without a truth_field, the plugin draws links, and nothing is judged. With one, you get right, wrong, and missed colors, counts, and a native evaluation per rule.

How large a dataset can it handle?

Candidates are gathered with dense distance matrices, which works fine for tens of thousands of originals. Beyond that, narrow the originals to a tag or a view.

Can I use a different embedding than CLIP?

Not from the form. The plugin is deliberately opinionated: one signal for pixels, one for meaning. To swap models, change CLIP_MODEL in constants.py to any zoo model that produces embeddings.

Is CLIP slow?

It's the only model in the pipeline, and it runs once. On CPU, budget about a second per image. On a GPU, a few minutes for several thousand.

You Can’t Watch 10,000 Episodes. Measure These Instead: Curation Metrics for LeRobot Datasets

Harpreet Sahota — Thu, 29 Oct 2026 00:01:57 +0000

A field guide to the metrics worth computing on a LeRobot dataset before you train a policy or fine-tune a vision-language-action (VLA) model: what each one measures, when to trust it, where it misleads you, and how to run the whole workflow in FiftyOne.

You recorded 400 teleop episodes last week. You pulled another 6,000 from the Hugging Face Hub because the task looked close to yours. Now you want to fine-tune a VLA.

You can’t watch all of it. At 30 seconds per episode, 10,000 episodes is about 83 hours of video, and that’s before you open a second camera. So you scrub through a handful, they look fine, and you hit train.

Then the policy hesitates at the first grasp. Somewhere in there is an episode with a frozen wrist camera. Another has 11 seconds of dead time before the arm moves. A third has the task string "Hold". A fourth has a video track that runs 0.4 seconds behind the actions. None of these show up when you watch one episode at a time because none of them is visible in a single episode. They show up when you compare an episode to its peers.

That’s the gap this post is about. A viewer shows you data. Curation tells you which episode to open next.

This guide walks through the metrics that do that job for LeRobot and VLA datasets. For each one, it covers what it measures, why you’d compute it, when to use it, how to read it, and where it lies to you. It also shows where each metric lives in a FiftyOne panel, because a metric you can’t filter, tag, and export is just a number in a notebook.

The post isn’t about any one dataset. The metrics apply to anything in LeRobot v3 format.

Key takeaways

Curation metrics answer six different questions: is the episode broken (integrity), is the motion clean (smoothness), is it wasting frames (time), did the robot do what was commanded (tracking), can you trust the cameras (vision), and can a model learn from the instruction (language). Start with the cheapest.
Most of these metrics need no model and no GPU. Integrity checks read metadata and Parquet. Motion, time, tracking, gripper, and consistency metrics are signal math on action and observation.state. Only the camera metrics decode video, at roughly 0.4 s per camera per episode.
Compare episodes to their own task group, not to the whole dataset. A fold-a-shirt episode and a pick-a-cube episode have different “normal” motion, and a pooled score will rank one task as the problem.
Correlated metrics should share one vote. Five time metrics that all fire on the same pause are one problem, not five.
The published evidence is mixed. Spectral arc length (SPARC) smoothness filtering raised RoboMimic success by 16% using one-sixth of the data (RINSE, arXiv:2604.23000). A separate audit (arXiv:2606.10229) found 5 of 7 curation metrics exploited episode length, and that detection accuracy and policy success were uncorrelated. Use scores to order a review queue, and validate with a training run before you delete anything.
FiftyOne’s role is the workflow around the metrics: one sample per episode, a panel that re-ranks as you filter, click-to-filter charts, temporal tags on the episode timeline, and a lossless export of the kept subset as a LeRobot v3 dataset.

First, the LeRobot v3 format, in one paragraph

A LeRobot v3 dataset doesn’t store one file per episode. Frames from many episodes are concatenated into shared Parquet files (data/chunk-000/file-000.parquet), and each camera’s video is concatenated into shared MP4s. Metadata in meta/info.json holds the schema and fps, meta/stats.json holds normalization statistics, and meta/episodes/ holds one row per episode with its row range and, per camera, the video time window.

Two consequences matter for curation. First, “episode N” points into shared files, so bookkeeping can drift without anything crashing. Second, the timestamp column is written from the nominal fps, not from a logged clock, so classic sensor-health checks such as jitter and clock drift don’t apply. You check structure and alignment instead.

FiftyOne’s native fo.types.LeRobotDataset (FiftyOne 1.22 and later) imports each episode as one multimodal sample, with a synchronized episode viewer on top (API reference). It reads only the metadata up front and pulls Parquet rows and video frames on demand, which is what makes dataset-level scoring practical.

What you have to tell the tool

Nothing about your robot should be guessed. A 14-dimensional action vector could be two 7-degree-of-freedom (DoF) arms, or one arm plus a hand, or end-effector deltas. If a tool silently picks, every smoothness score downstream is wrong in a way you can’t see.

So the compute form asks for five things:

Which arrays are state and action. They default to observation.state and action when the dataset uses the LeRobot standard names, and stay unset otherwise.
What the action means. Is it absolute joint positions in the same space as the state, or something else (deltas, end-effector poses)? This is a yes/no question, and it only appears when the two arrays have the same size.
Robot and joints. Single or dual arm, whether there’s a multi-joint hand, and which joints are the arm and which are the gripper. You pick them by name, with each joint’s observed range beside it.
Gripper open direction. Does a high value mean open or closed?
Cameras. Which video cameras to score. None are selected by default, because decoding is the slow part.

Anything you leave unanswered switches off the metrics that need it, and the form lists exactly which. It never blocks the run.

The compute form asks what your arrays mean instead of guessing, and lists which metrics stay off until you answer.

Good to know. Why does “what the action means” matter so much? Tracking, acceleration-spike, and joint-limit metrics compare the commanded action to the achieved state. That comparison only means something if both live in the same joint space. If your action is end-effector deltas, a gap between the two is just a unit mismatch, so those metrics stay off.

The LeRobot curation metrics

Every metric below follows the same pattern: what it measures, why it matters, when to use it, how to read it, and the pitfall to watch for. The harder ones get an intuition-level explanation.

One convention first. Inside a score, every metric is oriented so that higher means worse. The tables and charts show each metric’s raw direction, and the scoring section explains how they combine.

1. Integrity: is this episode broken?

Integrity checks don’t say an episode is worse. They say it’s broken. They produce pass, warn, or fail flags and counts. They never enter the ranking score, and they get their own verdict. They’re also the cheapest checks you’ll run, since they read metadata and Parquet and decode nothing.

Run them first. There’s no point ranking the smoothness of an episode whose rows don’t match its video.

Figure 1. A v3 episode is a row range in a shared Parquet file plus a time window in a shared MP4 per camera. Each integrity check catches a case where a pointer stopped matching the data. Illustrative values.

Check	What it catches
`frame_gaps`	Non-contiguous or duplicated `frame_index` values. Any nonzero count means a broken episode.
`length_mismatch`	The episode’s declared `length` disagrees with the actual number of Parquet rows. Metadata and data disagree.
`video_window_mismatch`	Per camera, the video window duration disagrees with `length/fps`. Video and actions differ in length.
`nonfinite_values`	NaN or inf in `action` or `state`. Any nonzero count breaks training.
`too_short`	Episodes under a minimum duration or frame count. Catches one-frame and near-empty episodes.
`schema_mismatch`	When you merge sources: differences in feature shapes and names, `fps`, units, camera keys, or resolution. Flags a dataset that won’t collate cleanly.

The six integrity checks and the broken-episode case each one catches.

timestamp_dev (the largest deviation of consecutive timestamps from 1/fps) is reported as information only. Because LeRobot writes timestamps from fps, it’s usually about zero and doesn’t affect the verdict.

Why it matters. The Hugging Face team’s write-up on community datasets lists these defects as recurring: episodes with one or very few frames, Parquet files deleted without reindexing, and inconsistent action or state dimensions for the same robot. Each one fails silently until a dataloader hits it, or worse, doesn’t fail at all.

Pitfall. video_window_mismatch compares durations, so it can’t see a constant offset. An episode whose video starts 400 ms late passes. That’s what the camera lag metric later in this guide is for.

Integrity verdicts and task coverage in the LeRobot Curation panel. Click a row to open that episode in the inspector.

2. Motion: is the motion clean?

Motion metrics measure the shape of the movement an operator commanded. This is where most of the curation literature lives, and it’s also where the most care is needed, so the preprocessing matters as much as the metric.

What signal it reads. The action array over the joints you put in the Arm (and Hand) fields. action is what a policy learns to output. On leader-follower rigs such as SO-100, Koch, and ALOHA, it’s the operator’s hand, not the robot’s arm.

How the signal is prepared.

Each joint is range-normalized before speeds are combined, so a joint with a large range doesn’t dominate and units don’t matter.
Gripper and hand dimensions are excluded from arm speed. A 22-joint dexterous hand is scored as its own signal so it never swamps a 7-joint arm.
The low-pass cutoff is min(10 Hz, 0.4 × fps). At 30 fps that’s 10 Hz, well away from the Nyquist frequency of 15 Hz.

Metrics run on fixed-length windows because some are sensitive to episode duration. Each scored metric also keeps a worst-window value, so one bad stretch isn’t averaged into a fine episode.

`sparc`: how many corrections does the motion contain?

Measures. The spectral arc length of the arm’s speed profile.

Intuition. Take the arm's speed over time. A clean reach is one smooth bump: speed rises, peaks, falls. Take that curve’s frequency spectrum and trace along it. A single bump has a compact spectrum, so the traced path is short. Every extra correction, hesitation, or stop-and-go adds frequency content, and the path gets longer. SPARC is the negative of that path length, so smooth motion sits near 0, and fragmented motion goes very negative.

In its standard form, SPARC is the negative arc length of the normalized speed spectrum up to a cutoff frequency:

SPARC = -∫ sqrt( (1/ω_c)² + (dV̂(ω)/dω)² ) dω (0 to ω_c)

where V̂(ω) is the magnitude spectrum of the speed profile, normalized by its peak. You don’t need to compute that by hand to use the metric. The picture matters: more corrections mean a longer trace.

Why it matters. It’s the most direct validation in the literature. RINSE (arXiv:2604.23000) reports that SPARC-based filtering gave 16% higher RoboMimic success using one-sixth of the data. It also reports SPARC as about 10 times more noise-robust than log dimensionless jerk (LDLJ).

Use it when. You want one primary smoothness number per episode. It’s scale-invariant, so it works across arms of different sizes.

Read it. Closer to 0 is smoother. Very negative means fragmented, hesitant motion.

Pitfall. SPARC responds to stop-and-go motion, not to additive white noise. A jittery signal from a noisy encoder can still look smooth to SPARC. That’s what the next metric is for.

`ldlj`: how much jerk, once you account for size and speed?

Measures. Log dimensionless jerk of the speed profile. Jerk is the rate of change of acceleration. It’s integrated over the movement, made dimensionless using the movement’s duration and peak speed, then log-transformed.

Why it matters. It catches what SPARC doesn’t: high-frequency noise and abrupt changes.

Use it when. You suspect noisy teleop hardware or a bad leader-arm encoder, alongside sparc.

Read it. Closer to 0 is smoother.

Pitfall. It’s noisier than SPARC and duration-sensitive, which is why the windows are fixed-length and why it carries half weight in the score.

Figure 2. The same two reaches, moved three ways. SPARC: smooth -2.57, stop-and-go -5.25, jittery -2.56. LDLJ: smooth -8.87, stop-and-go -14.45, jittery -14.71. SPARC separates stop-and-go from the other two. LDLJ penalizes both stop-and-go and noise. Synthetic signals at 30 fps. Values computed from them.

`sparc_phase`: smoothness within each phase of the grasp (opt-in)

Measures. SPARC computed inside each gripper-delimited phase (approach, grasp, transport, and release), rolled up by median.

Why it matters. Contact is inherently jerky. When the gripper closes on an object, speed changes abruptly for good reasons. Scoring the whole episode as one signal counts those transitions as roughness. Scoring per phase stops that.

Use it when. Your task has clear grasp and release events, and you’ve declared gripper joints and the open direction.

Read it. Same as sparc.

Pitfall. It adapts the gripper-phase idea from RINSE and isn’t the paper’s exact method. It’s covered by synthetic tests only. It skips phases with too few samples.

Two more motion metrics exist for review only and are never scored. jerk_rms is the root mean square (RMS) jerk after a low-pass filter. It’s noise-sensitive and correlated with the other metrics. psd_lf_hf is the log ratio of low- to high-frequency power in the speed profile, and it’s unreliable at LeRobot frame rates, where tremor sits near Nyquist.

Keep it honest. RINSE assumed its episodes were already filtered for task success, and it tested behavior cloning with proprioceptive signals, not VLAs. Smoothness helps. It isn’t sufficient. A smooth episode can still show the wrong task.

Motion, time, tracking, and gripper metrics as click-to-filter histograms on the Motion & Action tab.

3. Time efficiency: is the episode wasting frames?

Time metrics measure dead time. They read the commanded motion (action), with an idle threshold set as a multiple of the episode’s own moving speed. That per-episode threshold means a slow, careful operator isn’t flagged just for being slow.

Metric	Measures	Read it
`idle_lead_s`	Seconds of stillness at the start	High means dead time before the task begins. Also stores a suggested trim start (`keep_from_s`).
`idle_trail_s`	Seconds of stillness at the end	High means a long dead tail. Stores a suggested trim end (`keep_to_s`).
`longest_pause_s`	Longest idle run that isn’t at either end	High means a mid-task stall or hesitation.
`end_motion_ratio`	Speed in the last second relative to typical moving speed	High means the episode probably ended mid-motion.
`length_z`	Robust z of episode duration within its task group	Scored as `abs(z)`. Positive means unusually long (struggle). Negative means unusually short (abandoned).

Time efficiency metrics, what each one measures, and how to read a high value.

Figure 3. Left: one episode's arm speed with the three idle stretches the time metrics measure, and the keep window they suggest. Right: length_z places an episode's duration against its task group. Synthetic signals.

Why it matters. Two published pipelines treat this as signal. OpenVLA removed all-zero no-op actions from its training data. SCIZOR’s removed samples included “manipulation failures, slow motion, and pauses.” Both pipelines treated dead frames as removable, and either way they cost training compute.

Use it when. Your data comes from human teleop, where operators pause to think, reposition, or wait for a timer.

Pitfalls.

Trailing idle and end_motion_ratio get half weight, because fixed-duration recording setups produce a dead tail from the timer itself.
A pause next to a regrasp counts as recovery and is excluded from longest_pause_s. A hesitation that precedes a retry is the operator fixing a problem, not wasting time.
idle_frac (the overall fraction of idle frames) is shown for context and never scored. A high value can be a legitimate pause. path_length_z (opt-in) is also never scored, since it correlates with length_z.
length_z, path_length_z, and idle_frac depend on episode length by construction. Only length_z is scored, and it shares one vote with the idle metrics. That design decision comes straight from the length-confound audit discussed later.

Trimming is a suggestion. keep_from_s and keep_to_s store a proposed range. Nothing is written back to the dataset.

4. Tracking and contact: did the robot do what was commanded?

On a leader-follower rig, action is where the operator’s leader arm is, and observation.state is where the follower robot actually is. When the two agree, the robot is doing what it was told. When they diverge, something physical happened: a collision, a stall, a slipping joint, or a following arm that couldn’t keep up.

These metrics read the state, so they run only when you’ve declared the action to be absolute joint positions in the state space.

Figure 4. The follower trails the leader by a few frames (right), then stalls while the command keeps moving. The residual after lag alignment lights up exactly where the stall is, and the catch-up produces an acceleration spike. Synthetic signals.

`track_resid`: how far is the follower from the command?

Measures. The per-joint, normalized gap between action and state after lag alignment, with each joint’s median offset removed first. Arm joints only.

Why it matters. Calibration offsets are constant and harmless. You want the part of the gap that varies, where contact and slippage live.

Read it. High means the robot isn’t reaching the commanded position. It also raises interval flags that become spans on the timeline.

`track_lag_ms`: how late is the follower?

Measures. The delay between command and response, estimated by sliding one signal against the other and picking the offset with the highest cross-correlation:

lag = argmax_k corr( action[t], state[t + k] )

Why it matters. Mostly it’s a hardware property. A servo with a slow control loop lags every episode by about the same amount.

Read it. It’s summarized per dataset or session, and only episodes far from the dataset median are flagged. It never enters a score.

`accel_spike_frac`: sudden jolts

Measures. The fraction of frames where any joint’s state acceleration exceeds median + k × MAD (median absolute deviation), with an absolute floor.

Why it matters. It’s a proxy for collisions and contact. The absolute floor stops idle joints from turning every frame into a spike.

`joint_limit_frac (opt-in)`: operating near the edge

Measures. The share of (frame, joint) pairs within a margin (default 2% of range) of a joint limit.

Pitfall. It uses your limits if you give them, and otherwise falls back to meta/stats.json min and max. That’s a weak proxy, because observed extremes always sit at the edge of the observed range.

Keep it honest. These metrics have no published downstream validation. Treat them as sanity checks. Related work (RoboDrop, arXiv:2609.10021) shows observation-action temporal misalignment is a major corruption that action-only methods handle poorly, and a teleoperation paper (arXiv:2605.26349) flags stalls and operation near joint limits, but its user study had 3 operators. That’s weak evidence, not a validated benchmark.

5. Gripper: is the operator fighting the grasp?

Gripper metrics read the gripper dimension (averaged into one signal if you picked several), and they need the open direction you declared.

Metric	Measures	Read it
`gripper_flips_per_s`	Open/close transitions per second, with hysteresis so noise near the threshold doesn’t count	High means chatter or a hesitant operator.
`missed_grasp_frac` (opt-in)	Fraction of close commands where the follower closes fully with no stall residual, meaning nothing was held	High means empty grasps. Heuristic.
`recovery_count` (never scored)	Number of detected regrasps: open, then close again near the same place	Neutral or positive. Writes a `recovery` tag

Gripper metrics for chatter, empty grasps, and regrasp recovery.

Why it matters. A gripper that flips ten times in four seconds is either a noisy sensor or an operator who can’t get a grip. Both are worth a look.

Figure 5. Hysteresis keeps noise near the threshold from counting as a transition. Chatter scores 0.88 flips per second against 0.25 for a clean grasp. The regrasp has more transitions than the clean grasp and is tagged as a recovery instead of being penalized. Synthetic signals.

The design choice worth knowing. Several clean open-close cycles are often a recovery, an operator correcting a failed grasp. That’s behavior you may want a policy to learn. So recovery_count is never penalized, and a pause next to a regrasp isn’t counted against the time metrics either.

Pitfall. The gripper and tracking metrics have no published downstream validation. missed_grasp_frac and recovery_count are covered by synthetic tests only.

6. Consistency: does this demonstration agree with its peers?

`action_divergence`: same situation, different action

Measures. For frames in this episode, find the nearest state neighbors in other episodes of the same task group, and measure the variance of their action.

Intuition. If five demonstrations are in nearly the same state and four push left while one pushes right, the policy sees contradictory labels for the same input. Belkhale et al. (NeurIPS 2023) call this action divergence and report that “making actions more consistent tends to increase policy success.”

Use it when. You have enough episodes per task group for neighbors to exist (about 20).

Read it. High means this demonstration takes a different action from peers in a similar state.

Pitfall. Similar states can legitimately need different actions. A task with two valid strategies produces divergence that isn’t a defect. Treat it as a review signal.

Figure 6. Each arrow is the action another episode took from a state near this episode's. Variance of those actions: 0.00 on the left, 0.59 on the right. Schematic, in a 2D stand-in for the state space.

`stats_leverage` (never scored): how much one episode bends the normalization

Measures. How many feature dimensions this episode alone stretches beyond the rest of the dataset’s meta/stats.json bounds.

Why it matters. Training normally normalizes with stats.json. One episode with a sensor glitch that reports a joint value of 400 where everything else lives in -1 to 1 distorts that normalization for every other episode, and you’d never see it in a single-episode viewer.

7. Camera: can you trust the pixels?

Camera metrics are opt-in because they decode video. They’re pixel and signal arithmetic on a few decoded frames, with no model, so there’s nothing to download.

We store every value per camera and compare it with the same camera in other episodes. A wrist camera is never judged against a top camera. An episode’s camera score is its worst camera.

Metric	Measures	Read it
`blur`	The 10th percentile of Laplacian variance over 12 sampled frames, at a fixed size (longer side 480 px), reported as `log10(1 + variance)`	Lower is blurrier. The log scale exists because raw variance is heavy-tailed.
`exposure_err`	Distance of mean luminance from mid-gray, from 0 (mid-gray) to 1 (black or white). Median over sampled frames	High means the image is too dark or too bright.
`clipped_frac`	Share of pixels at pure black or pure white	High means blown highlights or crushed shadows.
`frozen_frac`	Of 6 short bursts, the share where frames are nearly identical while the state shows the robot moving	High means a frozen or dropped feed.
`video_action_lag_ms`	Absolute offset between motion in the video and arm speed in the action, from cross-correlation over the central 30 s	Near 0 is in sync.

Camera metrics, computed from a few decoded frames and compared with the same camera in other episodes.

Figure 7. The frame metrics on a synthetic scene, with values computed at the tile's resolution. The plugin measures at a fixed size (longer side 480 px) and compares each camera with the same camera in other episodes. A live camera on a static scene still shows frame-to-frame change (about 0.8 gray levels here). A frozen feed shows none.

Why frozen_frac is built the way it is. A still scene with a still robot isn’t a frozen camera. So a burst counts only when the robot is moving, and the pixels aren’t. A live camera on a static scene reads about 0.3 gray levels of frame-to-frame change or more, and the threshold (0.05) sits far below that.

Why video_action_lag_ms is the one to take seriously. If your video runs 400 ms behind your actions, the policy is trained to predict the action from an observation that’s already stale. It’s the failure that integrity checks can’t see, since durations match and every file is intact. The metric is relative to peers: a lag shared by every episode sets the baseline, and only episodes that differ from it stand out. Nothing is reported when the video doesn’t track the action (correlation under 0.3), for example when the robot is out of view.

Figure 8. Video motion energy trails the arm speed by 400 ms here. Cross-correlation finds the offset. Across episodes of one camera, the lag everyone shares is the baseline, and the episodes that differ from it stand out. Synthetic signals.

Use them when. You merge data from several sessions, several rigs, or the Hub. A camera problem that hits one session or one rig is easy to miss when you sample a few episodes, and obvious once you rank them all.

Pitfalls.

A scene that’s dark by design reads as high exposure_err, and black borders count as clipped. The per-camera comparison absorbs both, but a view that pools many robots will flag more episodes than a single-task dataset.
Cameras stored as images rather than video can’t be read, and aren’t offered.

Cost. Frame metrics share one decode of 12 frames, so enabling all three costs the same as enabling one. On the development set, a video takes about 0.05 s for the frame metrics and about 0.9 s per episode (all cameras) for the lag. Budget about 0.4 s per camera per episode when you run it.

The Vision tab compares each camera with the same camera in other episodes.

8. Language: can a VLA learn from this instruction?

For a VLA, the task string is the conditioning input. It’s what the model sees alongside the images. A clean trajectory with the instruction "Hold" teaches the model very little about language.

Metric	Measures
`task_missing`	Empty, whitespace-only, or null task string
`task_generic`	Fewer than 3 tokens, a placeholder (`"task desc"`, `"Hold"`), or no action verb

The two language checks run on each episode’s task string.

Task strings are canonicalized (case, whitespace, and punctuation) before grouping, so "Pick up the cube." and "pick up the cube" land in one group.

Why it’s separate. Language flags never touch the score, the verdict, or the flag count. They get their own verdict and their own column. A VLA user reads it next to the score. A policy trained without language conditioning can ignore it.

Keep it honest. Whether generic instructions hurt VLA language grounding hasn’t been measured in anything we reviewed. task_generic rests on the Hugging Face community’s list of these strings as real defects, which they documented while curating data for SmolVLA. Static checks also can’t tell you that a perfectly grammatical instruction describes the wrong video. That’s a model-based check, covered below.

Language verdicts flag missing or generic task strings without touching the score.

9. Outliers: unusual is not the same as bad

Outlier metrics are never scored.

Metric	Measures
`iforest_score`	Isolation-forest anomaly score over the episode’s metric vector, fit within its task group
`novelty_knn`	Mean distance to the nearest same-group episodes in metric-vector space
`is_outlier`	Either score at z ≥ 2

The three outlier metrics, fit within each task group and never scored.

Use them for. A second look at episodes that no single metric flags but that sit far from everything else in the group.

Read them. novelty_knn is two-sided. Very high means unlike the rest. Very low means near-duplicate.

Why they never score. The length-confound audit (arXiv:2606.10229) found that isolation-forest curation matched no curation at all, and that outlier detectors can rank defective episodes as normal. It’s a single-author preprint on 80 scripted demonstrations, so read it as a caution rather than a verdict. The practical takeaway is this: a “weird” episode might be your most valuable edge case.

Outliers get their own scatter on the Overview tab, for information only. They never enter the score.

Master reference table

Metric	Reads	Direction	In score?	Use it when
`frame_gaps`, `length_mismatch`, `nonfinite_values`, `too_short`	metadata, Parquet	any nonzero is bad	Own verdict	Always, first
`video_window_mismatch`, `schema_mismatch`	metadata, video windows	any mismatch is bad	Own verdict	Merging sources, any camera data
`sparc`	`action`	closer to 0 is smoother	Yes	Always
`ldlj`	`action`	closer to 0 is smoother	Yes, half weight	Noisy hardware
`sparc_phase`	`action` + gripper	closer to 0 is smoother	Yes (opt-in)	Grasp-and-release tasks
`jerk_rms`, `psd_lf_hf`	`action`	lower / higher is smoother	Never	Review only
`idle_lead_s`, `longest_pause_s`	`action`	lower is better	Yes	Human teleop
`idle_trail_s`, `end_motion_ratio`	`action`	lower is better	Yes, half weight	Human teleop
`length_z`	metadata	abs(z), lower is better	Yes	Mixed-quality sources
`idle_frac`, `path_length_z`	`action`	context	Never	Review only
`track_resid`	`action` + `state`	lower is better	Yes	Leader-follower rigs
`accel_spike_frac`, `joint_limit_frac`	`state`	lower is better	Yes	Contact-heavy tasks
`track_lag_ms`	`action` + `state`	far from median is flagged	Never	Comparing rigs and sessions
`gripper_flips_per_s`, `missed_grasp_frac`	gripper	lower is better	Yes	Grasp tasks
`recovery_count`	gripper	neutral	Never	Understanding recovery behavior
`action_divergence`	`state` + `action`	lower is better	Yes	20+ episodes per task
`stats_leverage`	`meta/stats.json`	lower is better	Never	Before training
`blur`, `exposure_err`, `clipped_frac`	video frames	see above	Yes (camera group)	Multi-session data
`frozen_frac`	video + `state`	lower is better	Yes (camera group)	Always, if you can afford to decode
`video_action_lag_ms`	video + `action`	near 0 is in sync	Yes (camera group)	Multi-rig data
`task_missing`, `task_generic`	task string	flag	Own verdict	Fine-tuning a VLA
`iforest_score`, `novelty_knn`	metric vector	context	Never	Second-look queue

Every curation metric in this guide: what it reads, which direction is worse, whether it enters the score, and when to use it.

Beyond heuristics: the model-based metrics

Everything above runs without a model. That’s useful: nothing to download, nothing to host, and cheap, repeatable results. It also means certain questions stay unanswered. A heuristic can’t tell you that an instruction describes the wrong video, that two episodes are near-duplicates in what they show, or whether the robot actually completed the task.

The research literature has methods for each. They cost more, and their evidence is a mix of strong and thin.

One thing to be clear about: the plugin this post demonstrates computes none of these. It’s deliberately heuristic-only. FiftyOne provides the infrastructure for the model-based tier. It has embeddings and similarity through the FiftyOne Brain, a model zoo (including remote zoo models) for running vision-language models (VLMs), and the same panel and tagging workflow to review whatever those produce. You’d build these as additional operators on the same pattern.

1. Instruction-video agreement (VLM). Sample frames, ask a vision-language model whether the video shows the instruction, and store the answer, a confidence score, and a proposed rewrite. SmolVLA’s data pipeline did a version of this, using Qwen2.5-VL-3B to rewrite task strings from sample frames plus the original label, with a prompt asking for under 30 characters starting with an action verb (“Pick,” “Place,” “Open”). Watch out: VLM judges are noisy, and OpenGVL’s benchmark found open VLMs lag proprietary ones. Keep the result as its own verdict, never folded into a score.

2. Embedding-based duplicates and novelty. Embed sampled frames and the action trajectory, then find near-duplicates by cosine similarity. SCIZOR does this at the transition level, using a joint state-action embedding, and reports that its two thresholds (εs = 0.58, εd = 0.99) transferred across RoboMimic, Open X-Embodiment, and real data. Watch out: near-duplicates in a table-top task might be exactly the repetition that robust policies need. Duplication needs a human decision about how much is too much.

3. Task progress. Ask a model to order shuffled frames by progress, then measure how well that order matches the true frame order (Value-Order Correlation, from GVL). OpenGVL applied this to LeRobot Hub data and found task-definition problems, labeling ambiguity, and failed or out-of-distribution episodes. TOPReward replaces generated numbers with token probabilities and reports VOC of 0.874 with Qwen3-VL-8B against 0.218 for GVL with the same model on a filtered Open X-Embodiment subset. Watch out: these are model-dependent, and the numbers come from the authors’ own benchmarks.

4. Information-based scoring. DemInf (Hejna et al., RSS 2025) estimates each demonstration’s contribution to state-action mutual information with k-nearest neighbor (kNN) estimators over learned embeddings. Its abstract reports a 5-10% improvement on RoboMimic and better performance on real ALOHA and Franka setups. The authors note it works best with relative actions.

5. Policy-in-the-loop. CUPID uses influence functions to estimate each demonstration’s effect on expected return from evaluation rollouts, and reports that under 33% of curated data yields state-of-the-art RoboMimic diffusion policies. Demo-SCORE trains a classifier on successful versus failed policy rollouts and filters demonstrations that resemble the failures, and its abstract reports over 15-35% higher absolute success. These need a policy and rollouts, so they’re the highest-fidelity and the most expensive tier.

The CUPID finding to remember. In CUPID’s results, DemInf curated the “highest overall quality,” yet CUPID-curated policies matched or beat it. The authors write that “human perception of demonstration quality does not necessarily correspond to data that maximizes downstream policy success.” That’s a direct warning about every heuristic score in this guide, including the ones that look best.

Dataset-level metrics

Some questions aren’t about any one episode.

Task coverage. Episodes per canonical task. A dataset with 1,900 episodes of one task and 12 of another isn’t balanced, no matter what the total says.
Source balance. Episodes per source repo when you’ve merged several.
Diversity. How spread out trajectories are within a task. Belkhale et al. also found “state diversity is not always beneficial,” so more spread isn’t automatically better.
Mixture weights. If you’re combining datasets, the weights matter as much as the filtering. Re-Mix (Hejna et al., CoRL 2024) optimizes domain weights and reports beating uniform weights by 38%. Octo and OpenVLA both curated mixtures from Open X-Embodiment by hand. Octo dropped datasets without images or delta end-effector control, plus ones “too repetitive, low image resolution, or excessively niche.” OpenVLA kept single-arm, third-person-camera, end-effector datasets.

The panel reports coverage as a count on the Integrity & Coverage tab. It’s a count, not a pick, since the right balance depends on what you’re training for.

From metrics to a ranking

Dozens of metrics don’t help you decide where to look. One ranking does. Getting there takes two decisions: compared to what, and how to combine them.

Compared to what: normalization

Task strings are canonicalized, then episodes are grouped by task (an episode with several tasks is grouped by its sorted set). Groups with at least 20 episodes are scored against themselves. A smaller group is pooled across views, and a view with fewer than 20 episodes overall is pooled and marked low-confidence in the panel.

The scoring uses robust z-scores with a floored scale. Zero-inflated metrics (such as idle_lead_s and frame_gaps, where most episodes score exactly zero) use percentile ranks or a tail-based scale, because a standard deviation computed over mostly zeros is meaningless.

Why per task. Without it, you rank tasks against each other. When we ran an earlier version of this scoring on a multi-task robot dataset, two tasks topped the ranking as a group. That’s the cross-task effect: a batch that mixes tasks produces misleading outliers, because “normal” differs by task. Score episodes of the same task together.

Figure 9. A slower, smaller task (B) pooled with a larger one (A). Pooled, the warn line sits near A's distribution and flags 26 of B's 30 episodes for being task B. Per task, each group is flagged against its own distribution: 2 of 120 in A and 1 of 30 in B. Synthetic data, robust z with a median and MAD scale.

Combined how: groups, and the max

Correlated metrics share a group, and a group counts as one vote:

Group	Scored members
`motion`	`sparc`, `ldlj`, `sparc_phase`
`time`	`idle_lead_s`, `idle_trail_s`, `longest_pause_s`, `end_motion_ratio`, `length_z`
`tracking`	`track_resid`, `accel_spike_frac`, `joint_limit_frac`
`gripper`	`gripper_flips_per_s`, `missed_grasp_frac`
`consistency`	`action_divergence`
`camera` (opt-in)	`blur`, `exposure_err`, `clipped_frac`, `frozen_frac`, `video_action_lag_ms`

The scored members of each metric group. Each group counts as one vote in the episode score.

Inside a group, the value is the maximum of the weighted, oriented member z-scores. Across groups, the episode’s score is the maximum group value, with the weighted mean as a tie-breaker. Weights are 1.0 except ldlj, idle_trail_s, and end_motion_ratio at 0.5.

The effect is that five fine ones can’t average away one severe problem in one dimension. An episode with a frozen camera and perfect motion still ranks near the top. Each episode also carries n_flags (how many groups are flagged) and driver (which group is behind its score), so you know why it ranks where it does.

Figure 10. An example episode with made-up z-scores. Every group is fine except camera, where frozen_frac is at 4.3. The score is the maximum group value, n_flags counts groups at or above the warn line (z = 2), and driver names the group behind the score.

The worst-first ranking shows each episode's score, flag count, and the group driving it.

The workflow, step by step

A metric earns its place by changing what you do next. Here’s the loop.

Step 1: Import the dataset

One sample is one episode. Set VFF_MULTIMODAL=1 before importing FiftyOne in every process that touches a LeRobot dataset, since it enables the multimodal episode viewer. Install the plugin with the plugin manager or link a checkout into your plugins folder.

import fiftyone as fo

dataset = fo.Dataset.from_dir(
    dataset_dir="/data/lerobot/my_dataset",
    dataset_type=fo.types.LeRobotDataset,
)

# Merge another source if you want to curate across datasets
dataset.add_dir(
    dataset_dir="/data/lerobot/other_dataset",
    dataset_type=fo.types.LeRobotDataset,
)

session = fo.launch_app(dataset)

See the App guide for sidebar filters and Spaces, which is where the curation panel opens.

Step 2: Compute quality

Open the operator browser and run LeRobot curation: compute quality. It’s delegated by default, so it runs in the background. The form has four tabs: Data (what you tell the tool), Metrics (a checkbox per metric, grouped by family), Camera, and Normalization (the smallest task group scored on its own, default 20).

Your picks are remembered, so re-running doesn’t start from scratch.

Running LeRobot curation: compute quality from the operator browser. It runs delegated, in the background.

Step 3: Read the Overview

Open a panel and choose LeRobot Curation. The Overview tab has the score histogram, verdict counts for Integrity and Language, episodes per task, the outlier scatter, and the worst-first ranking. The panel follows the view: filter the grid and it re-ranks.

Step 4: Drill into the family that drives the problem

Use the driver column to pick a tab. Motion & Action shows one histogram per metric. Vision shows the five camera metrics with camera chips. Click any bar and the samples grid filters to those episodes.

Step 5: Open the inspector

Click any row. The inspector shows every metric with its per-arm or per-camera breakdown, joint traces (action solid, state dashed), the speed profile with the idle threshold, the gripper timeline, and a few frames from the cameras, picked around the flagged spans.

Flagged spans (idle stretches, the longest pause, the roughest smoothness window, and acceleration spikes) are also written as temporal tags on the episode’s timeline once their metric reaches warn so that you can scrub straight to them. Regrasp recoveries are tagged too, as information. A re-run replaces the plugin’s tags and leaves any tag you drew yourself alone.

The inspector shows every metric, the joint traces, and the speed profile, and writes flagged spans as temporal tags on the episode timeline.

Step 6: Tag, don’t delete

The footer has three tag buttons: review, exclude-candidate, and relabel. They tag the selection, or everything in view if nothing is selected. Nothing is deleted or hidden. These are ordinary FiftyOne sample tags, so you can filter on them with view stages, save a view, or act on them from code.

Marking an episode exclude-candidate is always a separate human action. The scores never do it for you.

Step 7: Export the kept subset

keep = dataset.match_tags("exclude-candidate", bool=False)

keep.export(
    export_dir="/data/lerobot/my_dataset_curated",
    dataset_type=fo.types.LeRobotDataset,
    export_media=True,
)

The native exporter (API reference) writes a self-contained LeRobot v3 dataset. export_media=True is required, episodes are renumbered contiguously, and task and frame indexes are rebuilt. From there, you can push it to the Hub as you would any LeRobot dataset.

If you’d rather edit in place, send the excluded episode_index values to lerobot-edit-dataset with the delete_episodes operation. Task string rewrites are a separate matter: as of the sources we reviewed, lerobot-edit-dataset can’t yet edit task strings, so accepted rewrites need to be patched into meta/tasks.parquet and the per-frame task_index after export.

Keep it honest

Scores are triage, not verdicts. Smoothness and timing say nothing about whether the demonstration did the right thing. The plugin’s own documentation puts it this way: use it to decide where to look first, and don’t use it as an automatic accept/reject gate.

The evidence is genuinely mixed.

Smoothness filtering helped in RINSE, but not in behavior cloning with proprioceptive signals on episodes already filtered for success.
The curation-metrics audit found 5 of 7 metrics exploited episode length, and detection accuracy didn’t predict policy success. That’s one preprint on 80 scripted demonstrations. It’s also exactly why outlier metrics aren’t scored and why the length check exists.
CUPID found that data humans would call high quality can underperform data selected by influence on the policy.
The gripper and tracking metrics have no published downstream validation.

The thresholds are starting points. Defaults such as the 2% joint-limit margin, the warn line at z ≥ 2, the 20-episode group size, and the camera thresholds were calibrated on a 102-episode development set. They aren’t yet adjustable in the form. Check them on your own data.

How the metrics are checked. Each scored metric has a corruption it must catch: added hesitation, an inserted idle run, a duplicated frame, a blanked task, a stalled joint, and a blurred or frozen video. The harness applies the corruption to a real episode and asserts that the corrupted copy ranks worse than the original. It also re-scores every episode truncated to a common length, to find metrics that only measure duration. On 102 episodes, it takes about 5 minutes with camera metrics included. That tells you a metric detects what it claims to detect. It can’t tell you the detection predicts policy quality.

So validate. Before you drop data, train on the curated subset and the full set, then compare. A curation run you haven’t validated with a training run is a hypothesis.

Where FiftyOne fits

Rerun and Foxglove are excellent at what they’re built for: looking closely at a recording. If you want a synchronized view of cameras, joint traces, and 3D for one episode, they’re good tools.

Curation asks a different question, which is about a collection. Which of these 10,000 episodes should I look at first? Of those, which ones have a camera problem? Show me only the ones from this session. Tag these 40. Give me everything else as a dataset I can train on.

That needs a dataset model rather than a viewer: episodes as queryable samples, fields you can filter on, charts that filter the grid when you click them, tags that persist, saved views, and an export that round-trips to the training format. FiftyOne is built around that model, and that’s why the workflow above works the way it does.

The part that matters most if you build your own is how little of this is fixed. FiftyOne’s plugin framework is Python operators plus React panels, and you can mix the two in hybrid panels. The panel in this guide is a React app backed by operators. One data operator returns the whole payload (rows, histogram bins, thresholds, and verdict counts) and re-fires whenever the view changes. A second operator returns a single episode’s arrays and frames on demand for the inspector. The five tabs are handwritten React. Adding a metric is one Python function and one dictionary entry, and charting it means copying a documented chart template.

If your rig has a metric nobody else computes, such as force-torque spikes, a custom contact signal, or a per-operator score, you add it, and it gets the same filter, tag, inspector, and export workflow as everything else.

Nothing in this post requires this exact plugin. It’s one implementation of the pattern.

Try it yourself

LeRobot curation plugin: https://github.com/harpreetsahota204/lerobot_data_curation
FiftyOne multimodal and LeRobot docs: https://docs.voxel51.com/user_guide/multimodal.html
Build your own plugin: https://docs.voxel51.com/plugins/developing_plugins.html
FiftyOne Brain (embeddings, similarity): https://docs.voxel51.com/brain/index.html
Annotating LeRobot subtasks and dialogue in FiftyOne: https://docs.voxel51.com/plugins/plugins_ecosystem/fo_annotate_lerobot.html
LeRobot dataset v3 format: https://huggingface.co/docs/lerobot/en/lerobot-dataset-v3
Hugging Face on community dataset defects: https://huggingface.co/blog/lerobot-datasets
RINSE (smoothness metrics): https://arxiv.org/abs/2604.23000
Data Quality in Imitation Learning (action divergence): https://arxiv.org/abs/2306.02437
SCIZOR: https://arxiv.org/abs/2505.22626
DemInf: https://arxiv.org/abs/2502.08623
CUPID: https://arxiv.org/abs/2506.19121

Frequently asked questions

What metrics should I compute first on a LeRobot dataset?

Start with the integrity checks (frame_gaps, length_mismatch, nonfinite_values, too_short), because they’re free and tell you whether the rest is worth running. Then add sparc for motion, idle_lead_s and longest_pause_s for dead time, and frozen_frac if you can afford video decode.

What’s the difference between SPARC and LDLJ?

Both measure smoothness of the speed profile. SPARC measures how many corrections the motion contains, via its frequency spectrum, and is scale-invariant. It responds to stop-and-go motion but not to additive white noise. LDLJ is log dimensionless jerk. It does respond to white noise, but it’s noisier and duration-sensitive, so it carries half weight in the score.

Why are scores compared per task instead of across the whole dataset?

Different tasks have different normal motion and timing. A pooled score ranks the slowest, most correction-heavy task as the problem. Per-task normalization compares each episode to its own peers. It needs about 20 episodes per task. With fewer, the panel pools the view and shows a low-confidence banner.

Why isn’t track_lag_ms part of the score?

It’s mostly a hardware property. A slow servo loop delays every episode by roughly the same amount. The metric is useful for comparing rigs and sessions, so we summarize it per dataset and flag only episodes far from the median.

Does a generic task string such as “Hold” hurt a VLA?

Nobody has measured it in anything we reviewed. The task_generic check rests on the Hugging Face community’s documented list of such strings as real defects. A VLA uses the instruction as conditioning input, so a weak string is a reasonable thing to review. It has its own verdict and never enters the score.

Can I use the scores to drop episodes automatically?

We recommend against it. The published evidence is mixed: smoothness filtering helped in one study, while another found most curation metrics exploited episode length and that detection accuracy didn’t predict policy success. Use the ranking to decide what to review, tag exclude-candidate yourself, and validate with a training run.

Does the plugin use any models?

No. Every metric is signal or pixel arithmetic, so there’s nothing to download. Model-based checks such as VLM instruction verification and embedding-based duplicate detection are deliberately left out. FiftyOne provides the embeddings and similarity and model zoo infrastructure to build them as extra operators.

How do I export only the curated episodes?

Filter out the exclude-candidate tag with dataset.match_tags("exclude-candidate", bool=False) and call .export(..., dataset_type=fo.types.LeRobotDataset, export_media=True). The exporter writes a self-contained v3 dataset with episodes renumbered contiguously.

Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins

Harpreet Sahota — Wed, 28 Oct 2026 23:55:51 +0000

How the FiftyOne plugin framework turns your MCAP and LeRobot curation checks into buttons, panels, and background jobs that run on every episode, and why that works across a whole dataset rather than one recording.

You have 400 teleoperation episodes and a policy to train by Friday.

Some of those episodes are bad. The operator hesitated for eight seconds before the grasp. The wrist camera froze halfway through. The gripper opened and closed three times because the first grasp slipped. The task string says "pick up" and nothing else.

A policy will learn from all of it.

You could open each episode in a log viewer and scrub through it. At two minutes per episode, that's more than 13 hours, and you'll stop paying attention long before the end. What you want is a ranked list, worst first, with every flag one click away from the moment it happened.

No viewer ships that list, because the checks depend on your robot. A dual-arm rig with multi-joint hands needs different smoothness metrics than a wheeled rover with an inertial measurement unit (IMU) and GPS. The tool that curates your data has to be one you can extend in Python, on your schema, across the whole dataset.

That's what the FiftyOne plugin framework is for.

Key takeaways

A FiftyOne plugin is a folder with two files, fiftyone.yml and __init__.py. FiftyOne discovers it at runtime, so you never fork FiftyOne or wait for a release to add a button, form, panel, or background job.
FiftyOne reads .mcap, .bag, and .rrd files as media_type="multimodal" episodes and imports LeRobot v3 with fo.types.LeRobotDataset, so a plugin can score every episode in a dataset with the same code.
FiftyOne plugin code runs on the FiftyOne server in Python so that it can use NumPy, SciPy, PyTorch, a GPU, or a secret API key. Long jobs run in the background through delegated execution.
Most viewer extensions run in the browser against one recording, and labeling-platform hooks act on one workflow task. FiftyOne plugins run server-side Python with custom UI across a whole dataset.
Two open-source plugins show the pattern for physical AI: lerobot_data_curation ranks LeRobot episodes worst-first, and demo_quality_scorer does the same for MCAP recordings.

Why physical AI data needs custom tooling

Image datasets are forgiving. Every sample is a picture, every label is a box or a mask, and the same quality checks work on most of them.

Robot data has no such standard shape. One MCAP file from an autonomous vehicle might hold six cameras, two LIDARs, an IMU, GPS fixes, and a /tf tree. One LeRobot episode from a tabletop arm holds two camera streams, a 14-dimensional observation.state, a matching action array, and a task string. An egocentric dataset adds audio and hand poses.

The questions you need answered are specific to that shape:

Is the motion in this episode smooth, or full of corrections?
Did any sensor drop frames, drift its clock, or clip its values?
Does the action track the state, or is there lag between video and action?
Is the gripper behavior consistent with the other episodes of the same task?
Is the language instruction usable for fine-tuning a vision-language-action (VLA) model?

A generic viewer can't answer those. It can only show you the data and wait for you to notice.

FiftyOne gives you the dataset layer: episodes as samples, fields you can filter and sort on, views, tags, and an App with a multimodal viewer that plays MCAP and LeRobot streams in lockstep. Plugins give you the part that's specific to your robot.

Diagram of FiftyOne at the center of an ML stack, with plugin operators and panels connecting it to labeling tools, vector search backends, model inference, and automation.

What is a FiftyOne plugin?

A plugin is a folder FiftyOne finds. If the folder contains a fiftyone.yml, FiftyOne loads it at runtime and adds whatever it declares to the App, the SDK, and the CLI.

Plugins add two kinds of things:

Operators are actions. You trigger one, it runs, it finishes. "Compute episode quality" is an operator.
Panels are UI that stays open next to your data and reacts to what you do. A worst-first ranking that re-sorts when you filter the grid is a panel.

Under the hood, a panel is a special kind of operator, so everything below about operators applies to panels too.

	Notebook script	FiftyOne plugin operator
Who can run it	You, in your kernel	Anyone with the App open, no code
Inputs	Variables you edit by hand	A typed form, validated before anything runs
What it acts on	The dataset you loaded	The view or selection the user has open
Long jobs	Your kernel is busy until it finishes	Delegated to a worker, so the App stays free
Reuse	Copy and paste	One URI: buttons, panels, pipelines, notebooks, agents

What changes when you promote a notebook script to a FiftyOne plugin operator.

Good to know. Why not keep it in a notebook? Start in a notebook. Promote the code to an operator when someone else needs to run it, or when you catch yourself running it a second time. The figure below shows what changes.

Comparison of a notebook script and a FiftyOne plugin operator across who can run it, inputs, what it acts on, long jobs, and reuse.

The row that matters most for robot data is "what it acts on". A script scores whatever dataset you loaded. An operator scores the view the user has open right now, so the same code works on one task, one robot, one week of collection, or the episodes someone tagged for review.

The anatomy of a FiftyOne plugin

Here's the folder for a plugin with a Python backend and a custom React panel:

FiftyOne plugin folder with required fiftyone.yml and init.py files and optional requirements.txt and JavaScript files, showing what loads on the Python server and what loads in the App.

fiftyone.yml is the manifest. It holds the plugin name in @org/name form, the operators and panels it exposes, and any secrets it needs.
__init__.py holds your classes and a register(p) function.
requirements.txt lists pip dependencies. For MCAP work, that's usually mcap, mcap-protobuf-support and friends.
Everything below the dashed line is optional JavaScript for custom UI.

A Python-only plugin is two files. Teal in every figure in this post means the FiftyOne server, which runs Python. Orange means the App in your browser.

Keep it honest. Every operator name listed in fiftyone.yml must match the name in that operator's config. If you register a class but forget to list it, the operator doesn't show up, and you get no error. It's the most common "where did my operator go" bug.

A complete operator in 25 lines

This operator tags samples whose detections fall below a confidence threshold:

import fiftyone as fo
import fiftyone.operators as foo
import fiftyone.operators.types as types
from fiftyone import ViewField as F


class TagLowConf(foo.Operator):
    @property
    def config(self):
        return foo.OperatorConfig(name="tag_low_conf", label="Tag low-confidence")

    def resolve_placement(self, ctx):
        return types.Placement(
            types.Places.SAMPLES_GRID_SECONDARY_ACTIONS,
            types.Button(label="Tag low-conf"),
        )

    def resolve_input(self, ctx):
        inputs = types.Object()
        fields = list(ctx.dataset.get_field_schema(embedded_doc_type=fo.Detections))
        inputs.enum("field", fields, required=True, label="Label field")
        inputs.float("thresh", default=0.3, label="Confidence threshold")
        return types.Property(inputs)

    def execute(self, ctx):
        field, thresh = ctx.params["field"], ctx.params["thresh"]
        view = ctx.target_view().filter_labels(field, F("confidence") < thresh)
        view.tag_samples("low_conf")
        ctx.ops.notify(f"Tagged {len(view)} samples")


def register(p):
    p.register(TagLowConf)

And here's what the user sees:

TagLowConf operator code color-coded to the button, input form, and notification it creates in the FiftyOne App.

Four methods, four jobs:

config names the operator. name is its ID, and label becomes the form's heading.
resolve_placement puts a "Tag low-conf" button in the grid's action bar.
resolve_input declares a dropdown and a number. The App builds the form, with defaults and validation, from those two lines.
execute does the work on the target view and sends a notification.

You wrote no UI code. You described the inputs and the App built the interface. Swap the body of execute for spectral arc length (SPARC) smoothness over an /odom channel, and you have the skeleton of an MCAP quality scorer.

How an operator call works

Every operator call is two round trips between the App and the FiftyOne server:

Sequence diagram of a FiftyOne operator call, with one round trip that resolves inputs into a form and a second that executes and returns UI operations.

Resolve inputs. You click the button. The App sends the current context (dataset, view, selection) to the server, which calls your resolve_input and returns a schema. The App turns it into a form. If the operator is dynamic, this loop re-runs on every input change, which is how demo_quality_scorer shows a signal picker only after you choose a telemetry channel.
Execute. You hit Execute. The server validates the parameters against your schema, then calls execute. You return a result plus any UI operations, and the App applies them.

Your code runs only on the server. The browser draws what the server tells it to. That's why an operator can decode protobuf MCAP messages with mcap-protobuf-support, run SciPy's Welch power spectral density (PSD) on an IMU stream, or call a model on a GPU, without shipping any of that to the browser.

When something breaks, ask which arrow failed. A wrong form means resolve_input. A fine form that does nothing means execute. An execute that ran but didn't change the App means the UI operations you returned.

Long jobs run in the background with delegated execution

Scoring 400 episodes is not a click-and-wait job. Decoding video alone costs lerobot_data_curation about 0.4 seconds per camera per episode.

Flowchart comparing immediate execution in the FiftyOne server with delegated execution, where jobs queue in MongoDB and an orchestrator runs them.

Set allow_delegated_execution=True in the operator's config, and the user can choose. Immediate execution runs inside the server while the App waits, with a live progress bar. Delegated execution writes the job to a queue in MongoDB, and a separate worker runs the same execute method. You start a worker with fiftyone delegated launch and track jobs with fiftyone delegated list.

Both physical AI plugins below default to delegated execution, because a real corpus takes minutes.

Keep it honest. If no worker is running, delegated jobs sit in the queue forever. And a delegated job has no browser attached, so write results to the dataset rather than trying to change the user's view.

Panels: UI that stays open

A curation tool needs more than a form. It needs a place to see the ranking, click into an episode, and tag it. That's a panel.

FiftyOne panel lifecycle from on_load to render to event handlers, next to four places to keep state: panel state, panel data, execution store, and dataset fields.

A panel's on_load runs when it opens, render returns a layout, and events call your Python handlers with a fresh context. The event that matters most for curation is on_change_view: when the user filters the grid, your panel re-computes for the new view.

Here's a complete panel that shows class counts and follows the grid:

class ClassCounts(foo.Panel):
    @property
    def config(self):
        return foo.PanelConfig(name="class_counts", label="Class counts")

    def on_load(self, ctx):
        self.refresh(ctx)

    def on_change_view(self, ctx):
        self.refresh(ctx)

    def refresh(self, ctx):
        counts = ctx.view.count_values("ground_truth.detections.label")
        ctx.panel.set_state("n", len(counts))
        ctx.panel.set_data("counts", [{"type": "bar", "x": list(counts), "y": list(counts.values())}])

    def render(self, ctx):
        panel = types.Object()
        panel.md(f"**{ctx.panel.get_state('n')} classes**")
        panel.plot("counts")
        return types.Property(panel)

Small values go in panel state, which your Python handlers can read. Large payloads like chart data go through set_data, which is write-only from Python. Settings you want to survive a reload go in the execution store, ctx.store. Results that belong to the data go in dataset fields, where you can filter and sort on them like any other field.

When the built-in widgets run out, you have two options. A hybrid panel keeps state and compute in Python and renders a React component you write. A JS panel puts React in charge of the UI and calls your Python operators whenever it needs the server. Both plugins below are JS panels backed by Python operators, which suits dense, interactive dashboards. If you build a React UI, VOODO, the component library the FiftyOne App itself uses, makes it look native.

Comparison of Python, hybrid, and JS FiftyOne panels showing what runs in the browser, what runs on the server, and which need a Node build.

Operators compose

Every operator has a URI, like @voxel51/brain/compute_similarity, and a typed input schema. Anything that can call an operator can call yours.

One FiftyOne operator called from eight places: an App button, the operator browser, another operator, a pipeline stage, a notebook, AI agents through the MCP server, a background worker, and teammates.

That includes the FiftyOne Model Context Protocol (MCP) server, which exposes list_operators, get_operator_schema, and execute_operator to AI agents. An agent can find your quality scorer, read its inputs, run it on the episodes tagged review, and report back which ones failed sensor health. The schema validates the agent's JSON the same way it validates a person filling in the form.

For a physical AI team, that means the curation logic you write once runs from a button during review, from a notebook in your training pipeline, from a background worker overnight, and from an agent.

How FiftyOne plugins differ from viewer extensions

Most robotics viewers have extension points, and they're usually scoped to the recording you have open: a custom panel, a message converter, or a new file loader that runs in the browser or desktop app. Labeling platforms add hooks that act on one task in a workflow stage.

FiftyOne plugins run on the server, in Python, against any view of a dataset. A single plugin can add a form, run a job across every episode in the background, render a custom panel with the results, and expose all of it to agents through the MCP server, without leaving the language your robotics stack is written in.

For live debugging of a single recording, a dedicated log viewer is the right tool, and FiftyOne doesn't try to replace it. FiftyOne reads .rrd files and MCAP files, so a common setup is to debug individual logs in a viewer and curate the dataset in FiftyOne.

Two FiftyOne plugins for MCAP and LeRobot curation

Both plugins below were built using only the pieces covered above. Both follow the same rule, stated in their READMEs: the scores order your review queue, and a person makes the call. A perfectly smooth demonstration can still show the wrong task.

LeRobot Data Curation

lerobot_data_curation ranks the episodes of a LeRobot v3 dataset worst-first, so you know which to look at before training a policy or fine-tuning a VLA.

It scores each episode on motion smoothness, time efficiency, action-state tracking, gripper behavior, consistency, integrity, and language, and optionally on camera quality: blur, exposure, clipping, frozen feeds, and video-action lag. It has no model-based metrics, so there's no model to download.

How it maps to the framework:

A typed form with tabs. The compute operator's form has Data, Metrics, Camera, and Normalization tabs. You pick the observation.state and action arrays, and assign arm and gripper joints by name. Nothing is guessed from the data, and your picks are remembered for the next run.
Delegated by default. The operator sets allow_delegated_execution=True and default_choice_to_delegated=True, so scoring runs in the background. Camera decoding is the slow part.
A JS panel backed by Python operators. The LeRobot Curation panel is a React component with Overview, Motion & Action, Integrity & Coverage, Vision and Language tabs. Click a histogram bar to filter the grid to those episodes. Click a row to open an inspector with joint traces, the speed profile, the gripper timeline, and frames picked around the flagged spans.
It follows the view. The panel refreshes when the view changes, so filtering the grid re-ranks it.
Tags on the timeline. Flagged spans such as idle stretches, the longest pause, and acceleration spikes are written as temporal tags on each episode so that you can scrub straight to them in the multimodal viewer. Footer buttons apply review, exclude-candidate, and relabel sample tags.

import fiftyone as fo

dataset = fo.Dataset.from_dir(
    dataset_dir="/data/lerobot/my_dataset",
    dataset_type=fo.types.LeRobotDataset,
)
session = fo.launch_app(dataset)
# then run "LeRobot curation: compute quality" from the operator browser

Keep it honest. Scores are relative to the batch you scored. Per-task normalization needs about 20 episodes per task, and the default thresholds were calibrated on a 102-episode development set. Check them on your own data. Cloud-hosted LeRobot sources aren't supported yet.

Demo Quality Scorer

demo_quality_scorer does the same job for multimodal MCAP datasets: teleoperations, autonomous vehicles, drones, rovers, and egocentric capture.

Demo Quality Scorer plugin in the FiftyOne App ranking robot episodes from a multimodal MCAP dataset and opening one in the multimodal viewer.

It computes three metric families and ranks episodes worst-first:

Motion smoothness: SPARC, log dimensionless jerk (LDLJ), jerk RMS, and a low-to-high frequency power ratio, computed over windows of a speed profile you choose from position or velocity signals such as /odom -> twist_linear.
Sensor health: dropout, rate stability, clock drift, clipping, and cross-channel desync, all from message timestamps and raw values, so it works on any channel the plugin can decode.
Outliers: models that flag episodes whose motion and health features don't look like the rest of the batch.

How it maps to the framework:

A dynamic form. The form inspects the first sample's channels. If no telemetry channel carries a numeric signal, the motion family switches itself off and says why. If the server lacks a decoder, the form names the package to install.
Immediate or delegated, your choice. Run immediately with an in-app progress bar, or schedule it in the background (default).
A JS panel. The Episode Quality panel is a React component that has Motion, Health, and Outliers tabs. Click a histogram bar to filter the grid. Click a table row to open that episode in the multimodal viewer, with a toast naming the timecode of its worst flagged interval.
Results as fields. Scores land in quality.* fields such as quality.sparc and quality.ldlj, so you can sort and filter on them outside the panel too.

pip install "fiftyone[multimodal]>=1.22.0" mcap mcap-protobuf-support \
    mcap-ros1-support mcap-ros2-support numpy scipy scikit-learn
fiftyone plugins download https://github.com/harpreetsahota204/demo_quality_scorer

Keep it honest. Structural errors, such as the wrong action at a key moment, are invisible to every metric here. Re-running over a different set of episodes re-fits normalization, so expect scores to move.

Build your own in six steps

Your robot has checks neither plugin covers. Here's the loop for writing them.

Six steps to build a FiftyOne plugin: scaffold, write, register, run, iterate, and ship, with a loop from iterate back to write.

Step 1: Scaffold the plugin

Run fiftyone plugins create or make a folder in your plugins directory with a fiftyone.yml:

name: "@your-org/gripper_checks"
description: Flags episodes with repeated grasp attempts
operators:
  - score_grasps

Step 2: Write the operator

One class per operator or panel, like the examples above. Keep the scoring logic in a plain Python function so a notebook, a test, and the operator can all call it.

Step 3: Register it

Call p.register(ScoreGrasps) inside register(p), and make sure the name matches fiftyone.yml.

Step 4: Run the App in debug mode

export VFF_MULTIMODAL=1
fiftyone app debug my_lerobot_dataset

fiftyone app debug prints server logs and tracebacks to your terminal. VFF_MULTIMODAL=1 turns on the multimodal viewer in the App; set it before fiftyone is imported in every process that touches a multimodal dataset.

Step 5: Iterate

Edit, refresh the App, repeat. Python changes are picked up on the next request without a server restart. If you have a React panel, keep npm run dev rebuilding it.

Step 6: Ship it

Push to GitHub. Anyone installs it with:

fiftyone plugins download https://github.com/your-org/gripper_checks
fiftyone plugins requirements @your-org/gripper_checks --install

Good to know. How do I test an operator without the App? Operators are callable from Python with foo.execute_operator, so you can test them headless in pytest.

Try it yourself

Plugin developer guide
FiftyOne multimodal user guide for MCAP and LeRobot
voxel51/fiftyone-plugins, the official plugins and panel examples
lerobot_data_curation
demo_quality_scorer
FiftyOne MCP server
VOODO, the React design system for plugin panels

Frequently Asked Questions

How do I curate a LeRobot dataset before training a policy?

Import it with fo.Dataset.from_dir(dataset_dir=..., dataset_type=fo.types.LeRobotDataset), where each sample is one episode. Then install lerobot_data_curation and run its compute operator to rank episodes worst-first on motion, gripper, camera, and language metrics. Review the top of the list and tag episodes as exclude-candidate or relabel.

Can FiftyOne open MCAP files?

Yes. FiftyOne treats .mcap, .bag, and .rrd files as media_type="multimodal" samples, one episode per file, and plays their streams in a synchronized multimodal viewer. Set VFF_MULTIMODAL=1 before importing fiftyone to turn the viewer on.

How do I find bad demonstrations in robot teleoperation data?

Score every episode on motion smoothness (SPARC, LDLJ, jerk) and sensor health (dropout, rate stability, clock drift), then sort worst-first. demo_quality_scorer does this for MCAP datasets. Treat the scores as a review queue, because a smooth demonstration can still show the wrong task.

Do FiftyOne plugins need JavaScript?

No. A Python-only plugin is fiftyone.yml plus __init__.py, and the App builds forms and panels from Python declarations. Add React only when you need custom interactivity, such as the dashboards in lerobot_data_curation and demo_quality_scorer.

Where does FiftyOne plugin code run?

It runs on the FiftyOne server, in Python, or on a background worker for delegated operations. It never runs in the browser, so it can use any Python library, a GPU, or secrets without exposing them.

50 Embodiments, 497 Episodes, One Indexed Dataset: Curating the LeRobot Community Dataset with FiftyOne

Harpreet Sahota — Wed, 28 Oct 2026 23:54:50 +0000

A single prompt to an AI agent running FiftyOne Skills pulled 10 episodes from every robot embodiment in the LeRobot community dataset, computed video embeddings with Qwen3-VL, and built a queryable similarity index. Here's the code and the thinking behind it.

The LeRobot community dataset is one of the largest community-built pools of cross-embodiment robot data: 235 contributors, 50 robot types, ~900 GB. Hugging Face's dataset card lists 46 robot types across 791 cleaned datasets. The 50 here come from scanning robot_type in all 1,755 contributor folders, so the two counts cover different sets of datasets. It's organized as 1,755 contributor datasets, one per folder, so picking a balanced slice starts with reading each one's metadata. If you're training a policy that has to generalize across robots, you need episodes from every embodiment and a way to see which clips are actually worth keeping.

Now that FiftyOne natively supports LeRobot, I figured I’d use the FiftyOne skill to import LeRobot datasets and check it out. All it took was a one-sentence prompt:

"i want to have a little bit from every embodiment, i have enough disk space for the whole dataset... but let's take 10 episodes from each embodiment to start"

That sentence kicked off a session that read every contributor dataset's metadata, found 50 unique robot types, downloaded the right shard files for each, imported 497 episodes into FiftyOne, embedded every clip with Qwen3-VL, computed a similarity index, and saved 75 queryable views. All without writing a data loader, a Parquet parser, or a video decode loop.

One sentence was enough because the agent had FiftyOne Skills installed. Skills are packaged workflows that teach AI assistants how to do complex FiftyOne tasks: importing LeRobot episodes, running zoo models, computing embeddings, finding duplicates. The agent didn't have to guess at the LeRobot v3 layout or the FiftyOne API. It followed the skill. Install them in Cursor, Claude Code, Codex, or Gemini CLI with one command:

curl -sL skil.sh | sh -s -- voxel51/fiftyone-skills

This is what FiftyOne is for. Not visualization. Dataset curation at the scale that robot learning actually requires.

New here? FiftyOne is an open-source toolkit for loading, querying, visualizing, and curating ML datasets — images, videos, and multimodal data. LeRobot is Hugging Face's framework for robot learning: it defines a standard v3 dataset format (Parquet + MP4 shards with episode metadata) that the community uses to share teleoperation recordings.

Key takeaways

lerobot/community_dataset_v3 is a collection of 1,755 contributor datasets across 50 robot embodiments, each a complete LeRobot v3 dataset with its own meta/info.json.
Using the FiftyOne Skills dataset-import workflow, the agent scanned every */*/meta/info.json, grouped by robot_type, picked the smallest source dataset per embodiment with ≥10 episodes, and downloaded only the needed shard-0 files.
Each FiftyOne sample is one episode — not a frame, not a video file — with fields robot_type, task, duration, fps, dataset_name, and a media_reference pointer into the local LeRobot directory tree.
Qwen3-VL-Embedding-2B turns each episode (up to 32 frames from its time window) into one 2048-dimensional video embedding. The similarity index, Uniform Manifold Approximation and Projection (UMAP), uniqueness, and representativeness all run from that one field in under a minute.
The same model embeds text into the same space, so the index answers text queries: type "folding a cloth" and get the same bimanual cloth-folding episodes back, with no labels involved.
uniqueness scores range from 0.013 to 1.0 across the 497 episodes. 292 clips score below 0.1 because each embodiment's 10 clips come from one recording session. The two highest scores, the only ones above 0.7, both belong to single-frame episodes (0.02–0.03 s long): uniqueness flags broken recordings as readily as rare behavior.
75 saved views — 50 by robot type plus analytical cuts by frames per second (FPS), duration, robot category, task keyword, and curation intent — are stored in the dataset and load from the App's Views dropdown with no code.

What you need

To load and query the pre-built dataset (no GPU required):

pip install "fiftyone>=1.22.0" "huggingface_hub[cli]" transformers qwen-vl-utils torch torchvision

To re-run UMAP visualization yourself (Step 3):

pip install umap-learn

FiftyOne 1.22.0 is the first release with fo.types.LeRobotDataset. The dataset loads from the Hub with embeddings, similarity index, UMAP layout, and curation scores already computed. Steps 1–2, Step 4, and episode-to-episode search in Step 5 run entirely on the pre-built data, with no GPU or model weights. Text search in Step 5 downloads Qwen3-VL-Embedding-2B (about 4 GB) to embed your query; it runs on CPU and is faster on a GPU. Step 3 is for readers building their own dataset who want to replicate the embedding pipeline.

How the LeRobot community dataset is organized

FiftyOne App grid of LeRobot community dataset episodes with bar charts of robot type and an episode duration histogram, then one episode opened with its camera views and state and action values

lerobot/community_dataset_v3 is a collection of contributor datasets. Each one is a complete LeRobot v3 dataset in its own <contributor>/<dataset>/ folder:

community_dataset_v3/
  kuehnrobin/
    g1_stack_cube_left/
      meta/info.json
      data/
      videos/
  villekuosmanen/
    pick_coffee_capsule_under_dome/
      meta/info.json
      ...

1,755 directories. 17,617 files. 897.5 GB total. Each sub-dataset carries its own meta/info.json with its robot type, episode count, and FPS.

fo.types.LeRobotDataset reads one LeRobot v3 dataset at a time, so the agent started by downloading all */*/meta/info.json files — a 56-second fetch — and then scanning them to build the plan:

# Counts across all 1,755 sub-datasets
Robot types (50 unique):
  so100                   899 datasets   34801 total_episodes
  so101                   312 datasets    7490 total_episodes
  so101_follower          121 datasets    4414 total_episodes
  so100_follower          120 datasets    3065 total_episodes
  arx5                     43 datasets    2108 total_episodes
  Unitree_G1_inspire        1 dataset        7 total_episodes  # only 7 available
  so100MovellaDot          75 datasets      157 total_episodes  # max 5 per source
  ...

All 1,755 sub-datasets were already at codebase_version: v3.0. No conversion needed.

Left: the contributor/dataset folder layout, with the per-dataset meta/info.json files the agent scans first. Middle: the four-step plan — scan, group by robot_type, pick one source dataset per embodiment, download. Right: inside one source dataset, only file-000.parquet and file-000.mp4 (the first ~10 episodes) are fetched; the other shards are skipped.

Good to know. What's a shard-0 set? LeRobot v3 packs many episodes into shared MP4 and Parquet shards. file-000.parquet and file-000.mp4 hold the first batch of episodes for each dataset. Downloading only those files makes a 10-episode subset cheap, even from a 50-GB source dataset.

Step 1: load the dataset

The agent session did the heavy lifting — reading every contributor dataset's metadata, picking the right shard files, importing 497 episodes, computing embeddings, and pushing the result to the Hub. The import step is covered in How to Import LeRobot Datasets into FiftyOne Using Agent Skills.

The dataset is at Voxel51/community_v3_10per_embodiment. Load it:

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

ds = fouh.load_from_hub("Voxel51/community_v3_10per_embodiment")

print(len(ds))                    # 497
print(ds.distinct("robot_type"))  # 50 embodiments
print(ds.get_field_schema())      # episode_index, robot_type, task, duration, fps,
                                  # dataset_name, media_reference,
                                  # qwen3vl_embedding, qwen3vl_embedding_camera,
                                  # uniqueness, representativeness, ...

From here on, the source videos live in the Hub repo alongside the metadata. FiftyOne reads through the media_reference pointer — nothing to download separately. And because the unit is an episode rather than a file, every query you run is a dataset-level question, not a file-browser search.

What "one sample = one episode" actually means

A single LeRobot episode opened in the FiftyOne App, showing synced top and wrist camera views, a joint state and action table, a joint position plot, and the sample's fields, including media_reference

Most tools treat robot data as video files or frame sequences. FiftyOne takes a different unit: the episode.

Every sample in this dataset carries:

sample.episode_index   # int — 0..9 within the source dataset
sample.robot_type      # str — e.g. "arx5", "so100", "Unitree_G1_Dex3"
sample.dataset_name    # str — e.g. "villekuosmanen/pick_coffee_capsule_under_dome"
sample.task            # str — e.g. "Pick up the coffee capsule"
sample.duration        # float — seconds (0.02 to 148.6 in this dataset)
sample.fps             # float — 10, 15, 20, 25, 30, or 50
sample.media_reference # LeRobotEpisodeReference — a FiftyOne type with two fields:
                       #   .data   = [chunk_index, file_index, first_row, last_row]  (Parquet)
                       #   .videos = {camera: [chunk_index, file_index, from_ts, to_ts]}  (MP4)

Think of a FiftyOne sample as a row in a table: the left side has filterable scalar fields (robot_type, task, duration, fps) and the right side has the media_reference pointer. The pointer's two values — data[2:4] and videos[top][2:4] — map to the actual Parquet rows and MP4 time window in the LeRobot folder on disk.

A FiftyOne sample (left) with filterable scalars above the line and the media_reference pointer below it. Arrows connect the pointer's two values — data[2:4] and videos[top][2:4] — to the actual Parquet rows and MP4 time window in the LeRobot folder on disk.

Three roads lead from a FiftyOne view back to frame data:

Read in place via sample.media_reference.data — no copy, your code, one episode at a time.
Torch dataset via view.values("episode_index") passed to LeRobotDataset(..., episodes=...) — no copy, for training loops. episode_index restarts at 0 in every source dataset, so filter the view to one dataset_name first; for a mix of sources, export instead
Export via view.export(dataset_type=fo.types.LeRobotDataset) — copies data, produces a standalone v3 repo. Each export must come from a single source dataset, so filter by dataset_name first.

The rule: in-process → don't copy. Out-of-process → copy.

Path	Call	Copies data?	Use it when
Read in place	`sample.media_reference.data`	No	Your own code needs one episode's rows, such as for a plot, a metric, or a policy
Torch dataset	`LeRobotDataset(..., episodes=view.values("episode_index"))`	No	A training loop reads batches from a curated subset filtered to one `dataset_name`
Export	`view.export(dataset_type=fo.types.LeRobotDataset)`	Yes	Something outside the Python process needs a standalone LeRobot v3 repo, one source per export

Three ways to get frames from a FiftyOne view of LeRobot episodes, and when each one copies data

From inspection to curation

Filtering LeRobot episodes by task in the FiftyOne App sidebar and lasso-selecting points in the Embeddings panel to show the matching episodes in the grid

Timeline viewers are built for inspecting one recording at a time: open a log, scrub the timeline, and watch camera, inertial measurement unit (IMU), and joint streams in sync. That's the right tool when you're debugging one specific run.

FiftyOne starts where that ends. Once a run is recorded and you have hundreds of episodes, the question isn't "what happened in this episode?" It's "across all 497 episodes from 50 different robots, which clips are visually unique, which are near-duplicates, which ones actually belong in my training set?"

A timeline scrubber can't answer that. A queryable, indexed dataset can.

The rest of this post shows exactly what that looks like — every step from here on is something you can't do in a timeline scrubber.

Step 2: what's already on the dataset

The Hub dataset ships with a video embedding for every episode, computed with Qwen/Qwen3-VL-Embedding-2B from the qwen3vl_embeddings remote zoo source. Each episode's time window is decoded with PyAV; up to 32 evenly spaced frames go to the model as one video clip, producing a 2048-dimensional qwen3vl_embedding field. qwen3vl_embedding_camera records which camera stream the frames came from, since camera names and layouts differ across the 50 embodiments.

What Qwen3-VL-Embedding actually produces, and why it matters

Qwen3-VL is an open vision-language model family from Qwen/Alibaba. It comes in two variants: Qwen3-VL-Instruct (generates text — captions, bounding boxes, answers) and Qwen3-VL-Embedding (trained for retrieval, so it produces dense vectors directly). The embedding model reads the clip, and the hidden state of the last token becomes the vector, L2-normalized:

# what produced qwen3vl_embedding (inside the qwen3vl_embeddings zoo model)
outputs = model(**inputs)                                        # frames + instruction
embedding = outputs.last_hidden_state[row, last_token_position]  # one 2048-d vector per clip
embedding = F.normalize(embedding, p=2, dim=-1)

Top lane: an episode's time window is decoded with PyAV, up to 32 evenly spaced frames are tokenized by the vision encoder and run through the language model, and the last token's hidden state becomes one 2048-d vector stored as qwen3vl_embedding. Bottom lane: at query time, a text query goes through the same model and lands in the same space, which is what makes text-to-video search work.

This model is useful for robotics data because video and text map into the same vector space. The similarity index qwen3vl_embedding_sim keeps the model name, so it accepts text queries as well as episodes: ds.sort_by_similarity("folding a cloth", k=10, brain_key="qwen3vl_embedding_sim") — no labels, no metadata matching, just the model's understanding of what was in the clip.

For Vision-Language-Action (VLA) model development, this is specifically useful at the data curation stage. A VLA policy needs to generalize across scenes, embodiments, and task variations. But you can only curate what you can measure. Video embeddings give you a numeric handle on task semantics and visual appearance simultaneously, which makes it possible to compute:

Similarity — which episodes, across all 50 embodiments, are doing visually similar things?
Uniqueness — which demonstrations cover behavior your dataset hasn't seen much of?
Representativeness — which episodes best stand in for entire clusters of similar behavior?

You can't extract these signals from episode metadata alone. They require a model that actually watched the clips.

You also get uniqueness and representativeness scores out of the box. More on those below.

If you want to try a different model, such as Qwen/Qwen3-VL-Embedding-8B, write its vectors to a new field; Step 5 shows the full embedding loop for LeRobot episodes.

Step 3: Compute similarity, uniqueness, and representativeness

FiftyOne App grid of LeRobot episodes tagged with uniqueness and representativeness scores, next to the Embeddings panel and a score histogram

Skip the code if you loaded from the Hub. The pre-built dataset already includes qwen3vl_embedding_sim, qwen3vl_embedding_umap, uniqueness, and representativeness. The code block below is for readers building their own LeRobot dataset — it requires umap-learn and an existing embedding field. The explanations after it apply either way.

If you're building your own dataset, compute the qwen3vl_embedding field first (the loop is in Step 5), then run the four calls:

import fiftyone.brain as fob

EMBED = "qwen3vl_embedding"

# Similarity index — powers Sort by Similarity in the App. Passing the model name
# lets the index embed text queries (register the qwen3vl_embeddings source first)
fob.compute_similarity(
    ds, embeddings=EMBED, model="Qwen/Qwen3-VL-Embedding-2B",
    brain_key="qwen3vl_embedding_sim",
)

# UMAP — 2D layout for the Embeddings panel
fob.compute_visualization(
    ds, embeddings=EMBED, method="umap",
    brain_key="qwen3vl_embedding_umap", num_dims=2, seed=51,
)

# Uniqueness — how different each episode is from its nearest neighbors
fob.compute_uniqueness(
    ds, embeddings=EMBED,
    uniqueness_field="uniqueness",
    similarity_index="qwen3vl_embedding_sim",
)

# Representativeness — how close each episode is to a cluster center
fob.compute_representativeness(
    ds, embeddings=EMBED,
    representativeness_field="representativeness",
    similarity_index="qwen3vl_embedding_sim",
)

Why UMAP for video embeddings?

Each episode is a 2048-dimensional vector. You can't look at that. Dimensionality reduction maps it to 2D so you can see structure in the Embeddings panel.

The three common choices are principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), and UMAP:

PCA is linear. It finds the axes of maximum variance, which is fast but can't capture the curved manifold structure that emerges when you embed semantically diverse video. A pick-and-place cluster and a folding cluster may sit on a low-variance direction PCA ignores entirely.
t-SNE is non-linear and good at revealing local clusters, but it destroys global structure — the distance between clusters in the 2D plot is meaningless, and it's slow on datasets larger than a few thousand samples.
UMAP is non-linear, keeps local cluster structure, and usually keeps more of the arrangement between clusters than t-SNE does. Don't over-read it, though: distances between far-apart clusters are still only roughly meaningful. It runs on 500 episodes in seconds. FiftyOne's UMAP backend uses Euclidean distance with min_dist=0.1 by default, which gives tight clusters.

The same 497 Qwen3-VL-Embedding vectors reduced three ways, colored by robot category. PCA's two axes hold 20% of the variance, and the categories overlap; t-SNE separates local knots but inter-knot distances are arbitrary; UMAP is the layout the Embeddings panel shows. In every map, the 10 clips from one source dataset collapse into a single knot—same scene, same camera, same task.

For a first look at a cross-embodiment dataset, UMAP is the practical choice: you get clusters you can click into, and their rough arrangement is meaningful.

The map also shows what the embeddings respond to most: the recording session. For 495 of the 497 episodes, the nearest neighbor comes from the same source dataset. Step past a clip's own siblings, though, and the next matches in the full 2048-d space often share the task on a different robot. The nearest non-sibling to the so100 box-in-bin episodes is a so100MovellaDot ball-in-box episode (cosine distance 0.157), and the nearest non-sibling to the arx5 dome-lifting episodes is a trossen_ai_stationary cube-in-box episode (0.170).

What uniqueness and representativeness actually compute

Both metrics are grounded in the embedding space. The FiftyOne Brain docs cover uniqueness and representativeness; here's what each one computes:

Uniqueness — K-nearest-neighbor distance, K=3. For each episode, FiftyOne finds its 3 nearest neighbors in embedding space and computes a weighted mean of those distances:

uniqueness = weighted_mean(distances_to_3_nearest_neighbors)
             weights = [0.6, 0.3, 0.1]   # closer neighbors count more
             normalized to [0, 1] by dividing by the dataset maximum

Interpretation: uniqueness=1.0 means this episode is farther from its nearest neighbors than any other episode in the set — it occupies a region of embedding space that no other clip is near. uniqueness=0.013 means the episode is almost on top of another one — effectively a near-duplicate.

Left: a toy embedding space. Episode A is far from its three nearest neighbors (line width = weight), so it scores 1.0; episode B sits in a knot of near-identical clips and scores ≈ 0.01. Right: the real distribution over all 497 episodes on a log scale — 292 score below 0.1, two score above 0.7, and one arx5 clip (a single-frame episode) sets the maximum at 1.0.

Representativeness — K-Means distance to cluster center, K=20 clusters. For each episode, FiftyOne finds the nearest cluster center and scores it as:

representativeness = 1 / (1 + distance_to_nearest_cluster_center)
                     normalized locally within each cluster

Interpretation: representativeness=1.0 means this episode is at the heart of its cluster — it's the most "typical" example of that group of behaviors. Low representativeness means the episode sits at the edge of a cluster, more idiosyncratic than central.

Left: K-Means centers (×) with rings of equal distance. Episode P sits on its center and scores ≈ 1.0; episode Q is assigned to the blue cluster but sits far out and scores ≈ 0.3. Right: the real distribution — 198 of 497 episodes score above 0.8, and the floor of 0.33 means no episode is a pure outlier.

Why these two metrics are the right pair for VLA data curation

A Vision-Language-Action model needs two things from its training data:

Coverage — enough behavioral diversity for the policy to generalize to new scenes and task variations. Low-uniqueness episodes (near-duplicates) don't add coverage; they just add training compute and risk overfitting to that one scene configuration.
Signal density — enough repetition of each important behavior for the policy to learn it. Keeping only high-uniqueness episodes would discard the repetitions that make a skill learnable.

Uniqueness and representativeness together let you navigate that tradeoff explicitly:

Goal	Query
Seed set: diverse AND learnable	High uniqueness + high representativeness
Redundancy audit: what can be cut?	Low uniqueness floor (near-duplicates)
Canonical demos per task	Sort by representativeness within each task view
Rare demonstrations (or broken ones)	Uniqueness > 0.7, then check duration

Four VLA data curation goals and the uniqueness and representativeness queries that answer each one.

Every episode is placed by its two scores (uniqueness on a symlog axis, colored by robot category). The shaded regions are the queries from the table: the redundancy audit on the left, canonical demos along the top, high-uniqueness outliers on the right, and the 50 orange-ringed points are curate_diverse_seeds — the top-50 by uniqueness with representativeness above 0.5.

In this dataset, uniqueness ranges from 0.013 to 1.0. The floor shows near-identical clips in the set — 292 of 497 score below 0.1, and the bottom six are four moss cube-stacking episodes and two sam_bimanual cloth-folding episodes, each from a single recording session. At the top, only two episodes score above 0.7: an arx5 clip at 1.0 and an xarm6 clip at 0.86. Both are single-frame episodes, 0.02–0.03 seconds long. representativeness ranges from 0.33 → 1.0 — no episode is a pure outlier; even the most peripheral clips are loosely connected to a cluster.

Keep it honest. Both ends of the uniqueness ranking come from how we built this subset, not from rare behavior or wasted data. Each embodiment contributes 10 consecutive episodes from one recording session, so every clip has nine siblings sitting right next to it in embedding space. On this dataset, every one of the 50 episodes in curate_redundant_candidates has a sibling as its nearest neighbor, sampled on purpose. The top of the ranking has the opposite story: the five single-frame episodes look unlike everything else because there's almost nothing in them. They take the top two spots, and all five land in curate_diverse_seeds. Uniqueness did its job by surfacing recordings worth a second look. Those clips belong in the cut pile, though, not the seed set, so pair uniqueness with a duration filter.

These two numbers let you make dataset decisions — what to keep, what to cut, what's overrepresented — without watching a single video. A log viewer shows you one episode at a time. FiftyOne gives you a signal across all 497 at once.

Step 4: Save views that answer real questions

Switching between the category_single_arm and category_bimanual saved views from the Views dropdown in the FiftyOne App

Views in FiftyOne are lazy queries — no data is copied, they resolve on load. Once saved, they appear in the App's Views dropdown.

The pre-built dataset already has 75 views saved. Check what's there:

import fiftyone as fo

print(ds.list_saved_views())
# ['so100', 'so101', ..., 'curate_diverse_seeds', 'curate_redundant_candidates',
#  'category_humanoid', 'task_pick_place', 'fps_30', ...]

To build them yourself on your own dataset:

import fiftyone as fo

# Curation: 50 most visually distinct AND cluster-representative episodes
ds.save_view(
    "curate_diverse_seeds",
    ds.sort_by("uniqueness", reverse=True)
      .match(fo.ViewField("representativeness") > 0.5)
      .limit(50),
)

# Redundancy candidates: bottom 50 by uniqueness
ds.save_view(
    "curate_redundant_candidates",
    ds.sort_by("uniqueness").limit(50),
)

# Robot category views
HUMANOID = ["Unitree_G1_Dex3", "Unitree_G1_Gripper",
            "Unitree_G1_Inspire", "Unitree_G1_inspire", "g1"]
ds.save_view("category_humanoid",
    ds.match(fo.ViewField("robot_type").is_in(HUMANOID)))

# Task keyword views
ds.save_view("task_pick_place",
    ds.match(fo.ViewField("task").re_match("pick|place|grasp|grab", options="i")))

# FPS views — reveal recording-condition variation across embodiments
for fps_val in [10.0, 15.0, 20.0, 25.0, 30.0, 50.0]:
    view = ds.match(fo.ViewField("fps") == fps_val)
    if len(view) > 0:
        ds.save_view(f"fps_{int(fps_val)}", view)

75 views total. fps_30 covers 387 of the 497 episodes. fps_10 covers 30 — all from three stationary arms: widowx, ur5e_gello, and mycobot320. task_pick_place returns 367 episodes; task_fold_pour returns 70.

Each is a reusable, shareable slice of the dataset that resolves in milliseconds. No script, no re-download, no custom data loader — just a name in a dropdown.

Step 5: similarity search by episode and by text

Running a text similarity search for "robot arm picking up lego pieces" against the qwen3vl_embedding_sim index in the FiftyOne App's Similarity Search panel

Search by episode

The qwen3vl_embedding_sim index powers episode-to-episode search directly from the App or from Python. Pass any sample's ID, and FiftyOne returns the nearest neighbors by video embedding distance. Because every clip's closest matches are its own recording session, filter out the query's dataset_name to see what other robots are doing something similar:

from fiftyone import ViewField as F

# one sam_bimanual cloth-folding episode as the query
query = ds.match(
    (F("dataset_name") == "girardijp/sam_fold_cloth_single")
    & (F("episode_index") == 0)
).first()

# unfiltered: the query and its 9 siblings from the same session
same_session = ds.sort_by_similarity(query.id, k=10, brain_key="qwen3vl_embedding_sim")

# filtered: the 10 nearest episodes from other source datasets
others = (
    ds.sort_by_similarity(query.id, k=100, brain_key="qwen3vl_embedding_sim")
    .match(F("dataset_name") != query.dataset_name)
    .limit(10)
)
for s in others:
    print(f"[{s.robot_type}] {s.task}")
# [so100_bimanual] playing with folding      (all 10 hits)

Left: the query (star) on the UMAP, with its session's knot and the cross-source neighbors. Right: the 10 nearest episodes from other source datasets, ranked by cosine distance. The query's 9 siblings all sit within 0.015; past them, every hit is a so100_bimanual "playing with folding" episode, 0.19–0.31 away.

Selecting LeRobot episodes in the FiftyOne App and running an image similarity search from the Similarity Search panel

A sam_bimanual rig folding a cloth and a pair of SO-100 arms folding cloth look nothing alike as robots, and the model still puts them next to each other. That's the cross-embodiment signal. Some sessions pull in a neighboring robot type instead: the 10 Unitree_G1_Dex3 cube-stacking clips sit within 0.030 of each other, and the first other episode, at rank 10, is a Unitree_Z1_Dual red-cup pick (0.142).

Search by text

Reopening saved similarity searches in the FiftyOne App, such as "robot arm picking up lego pieces" and "blue robotic arm," to reload their matching LeRobot episodes

The same index takes text. The dataset stores the episode vectors, but FiftyOne still needs the model to embed your query, so register the qwen3vl_embeddings remote zoo source first:

import fiftyone.zoo as foz

# declare which models the remote source provides
foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/qwen3vl_embeddings",
    overwrite=True,
)

# optional: the first text query downloads the weights automatically
foz.download_zoo_model("Qwen/Qwen3-VL-Embedding-2B")

for query in ["folding a cloth", "pouring water into a cup", "humanoid robot picking up a can"]:
    view = ds.sort_by_similarity(query, k=3, brain_key="qwen3vl_embedding_sim")
    print(query, "->", [(s.robot_type, s.task) for s in view])
# summarised output:
# folding a cloth -> sam_bimanual, "Fold a cloth" (x3)
# pouring water into a cup -> bi_so100_follower, "bi soarm100 pour water" (x3)
# humanoid robot picking up a can -> Unitree_G1_Inspire, "Pick up the cans into the box on the table." (x3)

The same text box works in the App's similarity panel once qwen3vl_embedding_sim is selected. Not every query lands: "robot stacking a cube" returns a Unitree_Z1_Single red-cup pick rather than the Unitree_G1_Dex3 cube-stacking clips. Treat text search as a fast way to find candidates, then check them.

Build the embedding and index on your own dataset

LeRobot samples have no filepath, so compute_embeddings can't read the clips for you: decode each episode's time window with PyAV and pass the frames to the model as one video. This is the code behind qwen3vl_embedding (set ROOT to the folder that holds your <contributor>/<dataset>/ directories):

from pathlib import Path

import av
import fiftyone.brain as fob
import fiftyone.zoo as foz

ROOT = Path("/path/to/community_dataset_v3")


def episode_frames(video_path, from_ts, to_ts, max_frames=32):
    frames = []
    with av.open(str(video_path)) as container:
        stream = container.streams.video[0]
        fps = float(stream.average_rate or 30)
        step = max(1, int((to_ts - from_ts) * fps) // max_frames)
        container.seek(int(from_ts * 1_000_000), any_frame=False, backward=True)
        i = 0
        for frame in container.decode(stream):
            ts = float(frame.pts * stream.time_base)
            if ts < from_ts:
                continue
            if ts > to_ts or len(frames) >= max_frames:
                break
            if i % step == 0:
                frames.append(frame.to_image())
            i += 1
    return frames


foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/qwen3vl_embeddings", overwrite=True
)
model = foz.load_zoo_model("Qwen/Qwen3-VL-Embedding-2B")  # or Qwen/Qwen3-VL-Embedding-8B
if model._embedder is None:
    model._load_model()

for sample in ds.iter_samples(autosave=True, progress=True):
    cam, (chunk, file, from_ts, to_ts) = next(iter(sample.media_reference.videos.items()))
    path = ROOT / sample.dataset_name / "videos" / cam / f"chunk-{chunk:03d}" / f"file-{file:03d}.mp4"
    frames = episode_frames(path, from_ts, to_ts)
    # model.embed() expects a file path; the underlying embedder accepts a frame list as a video
    sample["qwen3vl_embedding"] = model._embedder.process([{"video": frames}])[0].cpu().float().numpy()
    sample["qwen3vl_embedding_camera"] = cam

fob.compute_similarity(
    ds,
    embeddings="qwen3vl_embedding",
    model="Qwen/Qwen3-VL-Embedding-2B",
    brain_key="qwen3vl_embedding_sim",
)

Passing the model name to compute_similarity lets the index embed text at query time; an index built without it accepts only episodes as queries. On a GPU, the 497 episodes take about 20 minutes. Then run the UMAP, uniqueness, and representativeness calls from Step 3.

Putting it together: export a curated LeRobot v3 training split

Once loaded, querying across 50 embodiments looks like this:

# 50 most unique AND representative episodes, minus the single-frame recordings
seed_view = ds.load_saved_view("curate_diverse_seeds").match(fo.ViewField("duration") > 1)

# Episodes from other robots that look like a given clip
query = ds.match(fo.ViewField("robot_type") == "sam_bimanual").first()
similar = (
    ds.sort_by_similarity(query.id, k=100, brain_key="qwen3vl_embedding_sim")
    .match(fo.ViewField("dataset_name") != query.dataset_name)
    .limit(10)
)

# Episodes matching a description (register the qwen3vl_embeddings source first)
cloth = ds.sort_by_similarity("folding a cloth", k=10, brain_key="qwen3vl_embedding_sim")

# Export the curated subset as standalone LeRobot v3 repos, one per source dataset
for name in seed_view.distinct("dataset_name"):
    seed_view.match(fo.ViewField("dataset_name") == name).export(
        export_dir=f"./my_training_subset/{name.replace('/', '__')}",
        dataset_type=fo.types.LeRobotDataset,
    )

The export loops over dataset_name because a LeRobot v3 repo has one feature schema, and the 50 embodiments here have different cameras and action dimensions. The 45 seed episodes come from 16 source datasets, so you get 16 repos.

That last step is the point. You went from a 900 GB community dataset to a curated, re-indexed LeRobot v3 training split — filtered by visual uniqueness, robot category, task type, or any combination — without writing a parser, a data loader, or a custom video decode loop. A viewer shows you what the robot did. A curated dataset decides what goes into the model.

Frequently Asked Questions

What does fo.types.LeRobotDataset actually store in FiftyOne's database?

Episode-level metadata only: episode_index, task, duration, fps, robot_type, and a media_reference that points into the local LeRobot directory. No pixel data, no Parquet rows. The source files stay where they are; FiftyOne reads through the pointer.

Can I use apply_model on a LeRobot dataset in FiftyOne?

Not for a standard zoo model — those expect sample.filepath pointing to a video or image file, and LeRobot samples use media_reference instead. The embedding in this post was computed by working around that: PyAV extracts frames from the episode's time window, and the frames go to the model as a list of PIL images. Step 5 shows the full loop.

Why does Qwen3-VL-Embedding use last-token pooling?

That's how Qwen3-VL-Embedding was trained. It's a decoder-only model with no CLS token, and its retrieval training teaches the final token's hidden state to summarize the whole input, whether it's a video clip or a sentence. Using the same pooling for both is what puts episodes and text queries in one space. Mean pooling over an instruction-tuned model's hidden states gives usable vectors for comparing clips, but not for matching them against text.

How do saved views work after load_from_hub?

They're stored in the FiftyOne dataset document alongside the samples. ds.list_saved_views() returns all 75 names; ds.load_saved_view("curate_diverse_seeds") returns the 50 episodes matching that query, resolved against whatever embeddings and fields exist locally.

How do I export a curated FiftyOne view to LeRobot v3 for policy training?

Export a curated view as standalone LeRobot v3 repos, one per source dataset:

for name in view.distinct("dataset_name"):
    view.match(fo.ViewField("dataset_name") == name).export(
        export_dir=f"./my_curated_subset/{name.replace('/', '__')}",
        dataset_type=fo.types.LeRobotDataset,
    )

The exporter refuses to mix episodes from different sources in one repo, since each embodiment has its own cameras and action layout. For each source, it re-indexes episodes from 0, recomputes stats, and writes a self-contained v3 structure that any LeRobot-compatible trainer can read directly.

Real vs. Synthetic LeRobot Datasets: 5 Sets and the Sim-to-Real Evidence

Harpreet Sahota — Wed, 28 Oct 2026 23:48:43 +0000

A field guide to 5 manipulation datasets in LeRobot v3.0, 2 real and 3 synthetic, with their exact fields and three strategies for generating robot data when real teleoperation can't scale.

Large language models train on trillions of tokens. Vision-language models train on billions of images. The largest dataset in this post has 637,498 trajectories, all synthetic. The two real, teleoperated datasets top out at 4,258 and 1,804 episodes. Every real demonstration requires a human at a controller, a physical robot, and a prepared scene. That collection process doesn't scale like web data.

We pulled 5 manipulation datasets into LeRobot v3.0 and FiftyOne: two real, teleoperated single-arm datasets and three simulated datasets spanning single-arm and bimanual setups. This guide examines the data bottleneck, compares three synthetic-generation strategies, and links each dataset to its live FiftyOne Space, paper, and license.

Key takeaways

5 datasets: 2 real teleoperation (fmb_multi on a Franka Panda, rh20t_cfg1 on a Flexiv arm), 3 synthetic (robocasa-MG_100, robotwin_unified, InternData-A1).
Scarcity starts at the source: fmb_multi has 1,804 episodes and rh20t_cfg1 has 4,258. Their FiftyOne exports contain 302 and 28 episodes, respectively. InternData-A1's full synthetic release has 637,498 trajectories, roughly 350× the size of fmb_multi.
The 3 synthetic datasets use different generation strategies: MimicGen decomposes and re-stitches human demonstrations (robocasa-MG_100); an LLM code generator and VLM observer create and repair task programs without seed demonstrations (robotwin_unified); and a compositional pipeline scripts skill policies from an asset library without demonstrations (InternData-A1).
Only the InternData-A1 paper reports sim-to-real transfer: zero-shot transfer after pretraining exclusively on synthetic data. The robocasa-MG_100 and robotwin_unified papers don't make that claim for the exports in this post.
Both real datasets here report fps as nominal, not measured — their source .npy/session data carries no per-frame timestamps, so frames map onto a synthetic grid rather than a recorded rate. Worth knowing before treating fps as ground truth for either.
Licensing: CC-BY-4.0 (fmb_multi), mixed CC-BY-SA-4.0/CC-BY-NC-4.0 by episode (rh20t_cfg1), Apache-2.0 (robocasa-MG_100, robotwin_unified), CC-BY-NC-SA-4.0 (InternData-A1).

Why real robot data is scarce, concretely

Human teleoperation requires trained operators, specialized hardware, real objects, and safety controls.

Architecture can't compensate for narrow data from one robot in one environment. Researchers use simulation, human video, and robot self-exploration to expand the training set beyond what teleoperators can collect.

Scarcity isn't the only problem.

Robot data consists of sequences, and a policy must learn how an action at time T changes the state at T+1. When fps is nominal rather than measured from timestamps, as in both real datasets in this post, the timing is an assumption rather than a recorded fact.

The SmolVLA paper reports state-of-the-art results with an order of magnitude less data than competing approaches and attributes the result to curation rather than collection volume alone. Each synthetic dataset below makes a different choice about how much real data stays in its generation loop.

Three ways to generate synthetic manipulation data

Dataset	Generation strategy	Seed data	Scale achieved	Sim-to-real evidence
robocasa-MG_100	MimicGen: decompose human demos into object-centric segments, re-stitch into new scene/object configurations, keep only successful rollouts (rejection sampling)	1,250 human teleop demos (4 operators, 3D SpaceMouse)	72,000 trajectories in the full Generated-3000 release; this export is the Generated-100 1/30 subsample, 2,400 episodes	Paper reports BC-Transformer results on the generated data; no sim-to-real transfer claim for this export
robotwin_unified	An LLM code-generation agent writes task programs from natural-language instructions, executes them in a SAPIEN-based simulator, and a VLM observer inspects execution to detect and localize failures for iterative repair	None — fully automated, no human demonstrations at any stage	100,000+ trajectories across 50 tasks and 5 robot embodiments in the source corpus; this export is the full aloha-embodiment slice, 27,500 episodes	Domain randomization (textures, lighting, clutter) is the stated sim-to-real strategy; the source paper cautions against treating benchmark performance as a real-world deployment proxy
InternData-A1	Fully decoupled compositional pipeline: asset specification, scripted skill policies, task composition, and rendering as independent stages — no demonstrations of any kind	None	637,498 trajectories, 401.4M frames, across 4 embodiments and 70 tasks	Paper reports zero-shot sim-to-real transfer on several tasks after pretraining a policy exclusively on this synthetic corpus

Three synthetic data generation strategies compared by seed data requirement, scale achieved, and published sim-to-real evidence.

MimicGen augments 1,250 real seed demonstrations. RoboTwin needs no seed demonstrations, but its program-synthesis loop uses a VLM observer to detect failures. InternData-A1 also skips demonstrations and composes hand-scripted skill policies across an asset library. Only InternData-A1 reports zero-shot transfer, but one result across three different task suites can't establish which strategy generalizes best. The sim-to-real claim has not been independently reproduced.

The 5 LeRobot datasets at a glance

"Episodes" reflects what's in each live FiftyOne export, not the full upstream source. robocasa-MG_100 and robotwin_unified are complete imports; fmb_multi, rh20t_cfg1, and InternData-A1 are disk-budget-constrained subsets (InternData-A1's export also covers a single task/embodiment out of 1,488 possible task/embodiment directories in its source repo).

Dataset	Description	License	Real or Sim
fmb_multi	Contact-rich Franka Panda assembly demonstrations across three boards, phase-labeled by skill primitive	CC-BY-4.0	Real
rh20t_cfg1	Real Flexiv-arm teleoperation with ten synchronized camera views, part of RH20T's one-shot skill-transfer dataset	Mixed CC-BY-SA-4.0 / CC-BY-NC-4.0	Real
robocasa-MG_100	Synthetic kitchen manipulation generated via MimicGen augmentation of real human demonstrations	Apache-2.0	Sim
robotwin_unified	Bimanual ALOHA-embodiment simulation generated by an LLM code-gen agent and VLM observer, with heavy domain randomization	Apache-2.0	Sim
InternData-A1	A single Franka articulation task ("close the laptop") from a fully synthetic, zero-demonstration compositional simulation pipeline	CC-BY-NC-SA-4.0	Sim

The 5 LeRobot v3.0 manipulation datasets in this post, with descriptions, license terms, and whether the data is real or simulated.

The 5 LeRobot datasets in detail

fmb_multi — Contact-rich Franka assembly, phase-labeled

FiftyOne playing an fmb_multi episode with four synchronized Franka Panda camera streams, joint position and end-effector pose plots, and a 28-dimensional observation state panel.

FiftyOne Format Dataset on Hugging Face: Voxel51/fmb-multi-302ep
Try it in a Hugging Face Space: harpreetsahota/fmb-multi-302ep
Original Dataset: robot-lev/fmb_multi (1,804 episodes)
Additional Resources: Paper (arXiv:2401.08553) · Project page
License: CC-BY-4.0

Designed to evaluate generalization in contact-rich assembly tasks, the Functional Manipulation Benchmark focuses on complex insertion-and-regrasp chains where force sensing plays a critical role, rather than basic pick-and-place routines.

fmb_multi serves as its multi-object subset: across three distinct assembly boards, a Franka Panda robot arm performs a structured sequence of skill primitives (approach, grasp, insert, place, regrasp). Throughout these trials, the system collects 6-axis end-effector force/torque readings along with visual and proprioceptive inputs. The benchmark prioritizes cross-object and board adaptability over massive scale, capping the entire suite at under 2,000 trajectories.

In this export, 302 demonstrations are captured (distributed almost evenly across the three boards at 100, 100, and 102 episodes), drawn from a full source collection of 1,804 episodes—showing inherent data sparsity before any FiftyOne filtering.

Available fields:

Camera Streams: 4 video feeds (side_1, side_2, wrist_1, wrist_2; 256×256 AV1 format, converted from original BGR)
Proprioception & State: observation.state (28-dimensional vector covering joint position, velocity, end-effector pose, end-effector velocity, and gripper state), alongside dedicated observation.state.joint_position, .ee_pose, and .gripper fields
Force & Kinematics: observation.force and observation.torque (measured in the end-effector frame), plus a flattened 6×7 observation.jacobian matrix
Control Action: A 7-dimensional Cartesian action vector

rh20t_cfg1 — Flexiv arm, ten camera views, license split by episode

FiftyOne playing an rh20t_cfg1 episode with five of the ten synchronized camera views of a Flexiv arm, a 15-dimensional observation state, and force and torque plots.

FiftyOne Format Dataset on Hugging Face: Voxel51/rh20t-cfg1-28ep
Try it in a Hugging Face Space: harpreetsahota/rh20t-cfg1-28ep
Original Dataset: robot-lev/rh20t_cfg1 (4,258 episodes; RH20T's full dataset spans 110,000+ sequences across many robot configs, of which cfg1/Flexiv is one)
Additional Resources: Paper (arXiv:2307.00595) · Project page
License: mixed, 16 episodes CC-BY-SA-4.0 / 12 episodes CC-BY-NC-4.0 in this export

Designed to test one-shot skill transfer—specifically whether a robot can master a new task from a single demonstration—RH20T collected an extensive library of over 110,000 manipulation sequences across various robotic platforms and camera setups.

The rh20t_cfg1 subset focuses on a Flexiv arm captured by ten synchronized camera views per episode. It preserves RH20T's scene-based licensing model, dividing scenes roughly equally between permissive and non-commercial terms.

You can verify the scene-to-license mapping for each episode via the folder attribute in meta/rh20t_episodes.json, with the full 28-row breakdown detailed on the dataset card.

Dataset Overview & Key Features

Visual Streams: Includes 10 multi-angle camera feeds (cam_<serial>, 360×640 AV1) recorded from fixed RealSense-style positions.
State & Kinematics: Features a 15-dimensional observation.state vector combining end-effector pose (7-dim), joint positions (7-dim), and gripper status (1-dim). Note that joint values are zeroed out if the source has_joint flag was false.
Force/Torque & Actions: Provides wrist-based observation.force/torque measurements alongside a dedicated 6-axis arm-mounted observation.robot_ft sensor, an 8-dimensional control action (target pose + gripper state), and original meta.rating quality annotations.

robocasa-MG_100 — Full synthetic kitchen dataset via MimicGen

FiftyOne playing a robocasa-MG_100 "open the microwave door" episode with one wrist view and two workspace views, a 16-dimensional state, and a 12-dimensional action.

FiftyOne Format Dataset on Hugging Face: Voxel51/robocasa-MG_100
Try it in a Hugging Face Space: harpreetsahota/robocasa-MG_100
Original Dataset: DAVIAN-Robotics/robocasa-MG_100 — all 2,400 episodes, full dataset, no subsetting
Additional Resources: Paper (arXiv:2406.02523) · MimicGen paper (arXiv:2310.17596) · Homepage
License: Apache-2.0

Designed to assess how synthetic data scaling can build upon a limited pool of human demonstrations, RoboCasa features procedurally generated kitchen settings covering 120 distinct configurations across 10 layouts and 12 visual styles.

The MG_100 release aligns with the paper's Generated-100 experimental tier—a 1/30 subset of MimicGen rollouts comprising 24 atomic tasks with 100 trajectories each (totalling 688,750 frames, with episode lengths spanning 89 to 838 frames). MimicGen segments human teleoperation recordings into object-focused primitives, recombines them across varied scene geometries and asset configurations, and validates success through rejection sampling. Because this system multiplies a foundational seed dataset of 1,250 human demonstrations gathered from 4 operators, it relies far more heavily on initial human inputs than the other synthetic benchmarks.

Available Fields:

3 camera streams: 1 wrist-mounted view and 2 static workspace angles (formatted as 128×128 AV1 video).
State Representation: A 16-dimensional observation.state array capturing gripper joint states, base pose, and base-to-end-effector position/quaternion.
Action Space: A 12-dimensional delta-pose action array incorporating base and gripper commands.
Status Flag: A boolean next.done flag.

robotwin_unified — 27,500-episode bimanual simulation with domain randomization

FiftyOne grid view of the robotwin_unified dataset filtered to the ALOHA embodiment, with a dashboard charting the 50 most frequent robot tasks and episode duration.

FiftyOne Format Dataset on Hugging Face: Voxel51/robotwin-unified-27500ep
Try it in a Hugging Face Space: harpreetsahota/robotwin-unified-27500ep
Original Dataset: lerobot/robotwin_unified — this export is all 27,500 episodes, full aloha-embodiment portion, no subsetting
Additional Resources: Paper (arXiv:2506.18088) · Simulator/generator code (MIT)
License: Apache-2.0

Bypassing human demonstration requirements entirely, RoboTwin 2.0 advances data generation beyond traditional MimicGen-style augmentation.

The process begins with an LLM writing executable task scripts from natural-language instructions, then running them inside a SAPIEN-based simulation environment. A vision-language model evaluates execution frame by frame, pinpointing failures to guide the code-generation agent in self-correcting errors through iterative refinement until high reliability is achieved. robotwin_unified represents the bimanual ALOHA dataset produced by this automated setup: 27,500 episodes comprising 50 dual-arm tasks and 6,075,103 frames. The corpus features extensive domain randomization—spanning 11,000 synthetic background textures, lighting, tabletop elevations, and instruction variants across a 731-object asset library.

With 23,559 unique task strings, the dataset maintains a near 1:1 mapping between instructions and episodes.

Available Fields:

Visual Feeds: 3 camera streams (an overhead cam_high plus dual wrist views cam_left_wrist and cam_right_wrist, 480×640 AV1)
State Representation: A 14-DoF observation.state capturing waist, shoulder, elbow, forearm_roll, wrist_angle, wrist_rotate, and gripper parameters for both left and right arms
Action Space: A 14-dimensional action vector aligned with the state dimensions.
Task strings: With 23,559 distinct task strings across 27,500 episodes, grouping by task isn't practical. Free-text search and filtering are the intended way to find episodes.

InternData-A1 — One task, one embodiment, out of 1,488 possible exports

FiftyOne playing an InternData-A1 "close the laptop" episode with head and hand camera views, a 49-dimensional observation state, and gripper pose plots.

FiftyOne Format Dataset on Hugging Face: Voxel51/interndata-a1-close-the-laptop-369ep
Try it in a Hugging Face Space: harpreetsahota/interndata-a1-close-the-laptop-369ep
Original Dataset: InternRobotics/InternData-A1 (637,498 trajectories, 401.4M frames, 4 embodiments, 70 tasks)
Additional Resources: Paper (arXiv:2511.16651) · Homepage
License: CC-BY-NC-SA-4.0

Among all synthetic datasets highlighted here, InternData-A1 represents the most radical departure from traditional workflows, bypassing human demonstrations entirely—even as an initial seed.

Instead, it relies on a decoupled simulation architecture that synthesizes environment scenes, object assets, and programmatic skill policies. This approach unlocks massive scaling (401.4M frames across 637,498 trajectories) that would be nearly impossible to gather via manual teleoperation. Beyond sheer volume, its most notable milestone is empirical: a VLA policy pretrained solely on this synthetic collection demonstrates zero-shot transfer across multiple physical tasks—making it the only synthetic dataset in this group with published sim-to-real transfer validation.

The upstream repository hosts 1,488 standalone LeRobot v3.0 datasets structured under <embodiment>/<task_category>/<task> directories, each following an independent schema. This specific export isolates franka-1/articulation_tasks/close_the_laptop, featuring 369 out of the 578 episodes available for that task, all guided by the uniform prompt "Close the laptop."

Dataset Features & Structure

Visual Inputs: Two video perspectives (images.rgb.head at 360×640 and images.rgb.hand at 480×640) alongside camera intrinsic and extrinsic parameters.
Kinematics & Control: Granular state and action values, including states.joint.position, actions.gripper.pose, and leader-arm teleoperation signals under master_actions.*.
Derived Convenience Aggregations: Includes a 49-dimensional observation state vector and a 43-dimensional action vector, engineered post-import to align with FiftyOne's State & Action visualizer natively.

More LeRobot datasets. These five come from the 11 LeRobot v3.0 datasets for VLA training roundup. Five humanoid and bimanual datasets are compared in 5 humanoid and bimanual LeRobot datasets.

Frequently Asked Questions

Why is real robot data so much scarcer than simulated data?

Every real demonstration requires a human operator, a physical robot, a real environment with real objects, and safety overhead around all of it — none of which parallelizes the way web scraping does. The two real datasets in this post (fmb_multi, rh20t_cfg1) top out at 1,804 and 4,258 episodes at the source; the largest synthetic dataset here (InternData-A1) is 637,498 trajectories, because simulation removes the human-in-the-loop requirement for every single trajectory.

Which of these synthetic datasets has actual sim-to-real evidence?

Only InternData-A1 — its paper reports zero-shot sim-to-real transfer on several tasks after pretraining a policy exclusively on the synthetic corpus. robocasa-MG_100's paper reports simulation-only BC-Transformer results, and robotwin_unified's paper explicitly cautions against treating its benchmark performance as a real-world deployment proxy without separate validation.

What does "fps is nominal" mean and why does it matter?

It means the frame rate in the dataset's metadata wasn't measured from actual per-frame timestamps—it's a value assumed for a synthetic, evenly spaced frame grid. Both fmb_multi (source .npy files carry no timestamps) and rh20t_cfg1 (cameras were downsampled from variable native capture rates) report nominal fps. If your training pipeline assumes fps reflects real elapsed time between frames, that assumption doesn't hold for either dataset as tightly as it does for datasets with measured timestamps.

Can synthetic data actually replace real teleoperation data?

The evidence is suggestive but not settled. InternData-A1 reports zero-shot transfer on one task suite, and the result has not been independently reproduced. The SmolVLA result also suggests that data quality and diversity can matter more than raw scale. Treat each sim-to-real claim as one data point, not a general result.

Which licenses permit commercial use?

Apache-2.0 (robocasa-MG_100, robotwin_unified) permits commercial use subject to its terms. CC-BY-4.0 (fmb_multi) permits commercial use with attribution. CC-BY-NC-SA-4.0 (InternData-A1) prohibits commercial use. rh20t_cfg1's 28-episode export mixes CC-BY-SA-4.0 and CC-BY-NC-4.0, so check the per-episode license table before using it commercially.

5 Humanoid and Bimanual LeRobot Datasets, Compared by State and Action Space

Harpreet Sahota — Wed, 28 Oct 2026 23:47:42 +0000

Source: Generated using Nano Banana 2

A field guide to 5 humanoid and bimanual manipulation datasets in LeRobot v3.0 — with the exact state/action dimensionality, fields, and research behind each one, and why "humanoid dataset" hides a 7x spread in how these robots actually represent themselves.

Two robots doing near-identical tasks—reach into clutter, pick something up, place it somewhere—can record their own state as a 29-dimensional vector or a 216-dimensional one, depending entirely on how many joints, cameras, and collection-process signals the platform records. Neither number is wrong. They're just different robots, and a policy trained on one has no native way to use data from the other.

That's embodiment heterogeneity, one of the five reasons robot data is fundamentally harder to organize than the image and text data that trains typical vision-language models.

We pulled 5 humanoid and bimanual manipulation datasets — real teleoperation and pure simulation, a Unitree G1 to a pair of matched humanoid torsos — into LeRobot v3.0 and FiftyOne, and the spread in how they represent the same broad category of task is the throughline of this post. Below: what embodiment heterogeneity actually costs in practice, a side-by-side of state/action dimensionality across all 5, and a tour of each with links to the live FiftyOne Space, the paper, and the license.

Key takeaways

5 datasets spanning a dual-arm mobile manipulator (AgiBotWorld2026), a full humanoid on legs (HIW-500-LeRobot, Unitree G1), two matched simulated humanoid torsos (tavis-head-gr1t2, tavis-head-reachy2), and a stationary bimanual dexterous-hand rig (trex_dataset).
observation.state dimensionality ranges from 29 (HIW-500-LeRobot's joint vector, which sits alongside a separate 23-dim whole-body-control vector) up to 216 (AgiBotWorld2026, which also carries RL collection signals), roughly a 7x spread across datasets that all fall under "humanoid/bimanual manipulation."
tavis-head-gr1t2 and tavis-head-reachy2 run the same 5-task benchmark on two different humanoid torsos — the cleanest pair in this post for seeing embodiment heterogeneity in isolation, since the task and camera setup are held constant and only the robot's own DoF count changes (44-dim vs. 36-dim state).
3 of 5 are real-world teleoperation (AgiBotWorld2026, HIW-500-LeRobot, trex_dataset); 2 are simulation (tavis-head-gr1t2, tavis-head-reachy2, both built in NVIDIA IsaacLab).
Licensing: MIT (trex_dataset), CC-BY-4.0 (HIW-500-LeRobot, both tavis-head-*), CC-BY-NC-SA-4.0 (AgiBotWorld2026, non-commercial only).

What is embodiment heterogeneity?

Source: arXiv 2510.24795

Different robots have different action spaces (a 7-DoF single arm vs. a 14-DoF bimanual rig vs. a humanoid's whole-body state), different proprioceptive observation spaces (varying joint counts and sensor configurations), and different sensor modalities (RGB, depth, multiple viewpoints, tactile).

The same action vector induces different physical motion on different robots because end-effector coordinates translate differently depending on kinematic structure—and camera observations vary by mount position, resolution, and field of view on top of that. Coordinate frames aren't even guaranteed to agree: "forward" for one robot's base frame can be "left" for another's.

The field's general answer is coarse alignment.

For example, Open X-Embodiment normalizes everything to a unified 7-DoF end-effector representation (position x/y/z, orientation roll/pitch/yaw, gripper state), then de-normalizes it per embodiment at training time. That's a real compromise: the same action values might mean an absolute position for one robot and a relative velocity for another, and camera observations still don't line up. But that's what makes cross-embodiment learning possible at all, and models trained on that coarsely aligned diversity have been shown to outperform narrowly targeted single-embodiment counterparts, even when evaluated on a single embodiment.

None of the 5 datasets below have had that coarse alignment applied.

They're raw, as each source team designed them. That's deliberate for this post: seeing the unaligned heterogeneity first is what makes clear why Open X-Embodiment-style alignment work exists in the first place.

The embodiment spread

Dataset	Robot	observation.state dim	action dim	Camera streams	Distinguishing sensing
HIW-500-LeRobot	Unitree G1 (real)	29 (+ separate 23-dim observation.state.wbc)	23	3	Whole-body-control vector (pivot velocity/pose + dual end-effector pose)
tavis-head-reachy2	Reachy2 (sim)	36	19	4	Active-vision head (pan/tilt neck)
tavis-head-gr1t2	GR1T2 (sim)	44	19	4	Active-vision head (pan/tilt neck)
trex_dataset	Dexmate Vega-1 + Sharpa Wave hands (real)	58	58	3 RGB + 20 tactile	Per-fingertip 6-axis tactile wrench
AgiBotWorld2026	AgiBot G2 (real)	216	44	7	RL collection signals (reward/done/succeed/intervened)

State and action dimensionality, camera count, and distinguishing sensing across all 5 humanoid LeRobot datasets, ordered by observation.state size.

Reading this table left to right isn't just "more DoF, bigger number."

Each jump reflects a different design decision about what the platform needs to represent: HIW-500-LeRobot's vector is almost entirely joint angles because it's a legged, whole-body-controlled humanoid where balance and locomotion state matter as much as arm position. trex_dataset's state and action vectors are identical in size and layout ([L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22]) because every one of its 58 dimensions is a commanded joint position — dexterous hands alone eat 44 of those 58 dimensions. AgiBotWorld2026's 216-dim vector is the largest here not because the robot has more joints than the others, but because it's the only dataset in this post collected via reinforcement-learning-style rollout, and RL collection signals get concatenated in alongside camera extrinsics and full-body state.

The tavis-head-gr1t2/tavis-head-reachy2 pair is the cleanest illustration of pure embodiment heterogeneity in this table, because it's the only case here where the task, camera rig, and collection method are held constant and only the robot changes. Same 5 tasks (ClutterPickCubeTask, ClutterPickLiftTask, ConditionalPickTask, MultiShelfScanTask, WaitThenActTask), same 4-camera setup, same 60fps, same VR teleoperation pipeline in IsaacLab — and the state vector still differs by 8 dimensions (44 vs. 36) because GR1T2 and Reachy2 simply have different kinematic structures.

If you want one policy that works on both, you'd need exactly the kind of per-embodiment de-normalization Open X-Embodiment does at the field level, in miniature, for these two robots alone.

The 5 humanoid LeRobot datasets at a glance

Keep it honest. "Episodes" reflects what's in each live FiftyOne export, not the full upstream source. tavis-head-gr1t2 and tavis-head-reachy2 are complete (800/800 each); the other 3 are disk-budget-constrained subsets of much larger real-world sources.

Dataset	Description	License	Real or Sim
AgiBotWorld2026	Real-world dual-arm humanoid manipulation collected via RL-style human-intervention rollout on the AgiBot G2 platform	CC-BY-NC-SA-4.0	Real
HIW-500-LeRobot	Whole-body Unitree G1 humanoid teleoperation performing household chores across real homes in Southeast Asia	CC-BY-4.0	Real
tavis-head-gr1t2	Simulated active-vision humanoid benchmark (GR1T2) comparing head-mounted vs. fixed cameras on identical manipulation tasks	CC-BY-4.0	Sim
tavis-head-reachy2	The same active-vision benchmark as tavis-head-gr1t2, run on a Reachy2 humanoid instead	CC-BY-4.0	Sim
trex_dataset	Bimanual dexterous manipulation with real per-fingertip tactile sensing on a Dexmate Vega-1 robot	MIT	Real

The 5 datasets with a short description, license, and whether the data is real or simulated.

The 5 humanoid LeRobot datasets

AgiBotWorld2026 — Real-world dual-arm manipulation with RL collection signals

FiftyOne App playing an AgiBotWorld2026 episode with head fisheye, wrist, and top head camera panels alongside a reward stream and a 216-dimensional observation state panel.

FiftyOne Format Dataset on Hugging Face: Voxel51/AgiBotWorld2026-sample-10ep
Try it in a Hugging Face Space: harpreetsahota/AgiBotWorld2026-sample-10ep
Original Dataset: agibot-world/AgiBotWorld2026 (102,078 episodes, 13.6TB)
Additional Resources: Project homepage: agibot-world.com
License: CC-BY-NC-SA-4.0

As one of the most expansive public initiatives in real-world humanoid manipulation, AgiBot World uses the dual-arm AgiBot G2 platform to collect long-horizon tasks in real-world offices, homes, and retail environments rather than controlled laboratory environments. The 2026 dataset release structures this data into three primary collection modalities: standard imitation learning, RL-based rollout featuring human intervention (HG-DAgger), and multi-signal interaction logs. This FiftyOne export focuses exclusively on the HG-DAgger reinforcement learning split, capturing a policy mid-training alongside reward metrics and human-correction markers rather than polished expert trajectories. All 10 exported episodes are derived from a single task archive (ReinforcementLearning/Home/task_12192/HG-DAgger, "Insert the loose change into the coin slot of the piggy bank dagger").

Data Breakdown & Features

RL Collection Signals: Unique to the RL framework, each frame includes reward, done, succeed, intervened (indicating active human correction), truncated, and action_prob.
Camera Streams: 7 visual channels including top_head, hand_left, hand_right, two head fisheyes, a rear fisheye, and a head_depth stream (configured with dtype: video and gray16be pixel format, allowing depth information to be ingested like standard RGB).
State & Action Vectors: A 216-dimensional observation.state vector and a 44-dimensional action vector.

HIW-500-LeRobot — Humanoid teleoperation in real homes, sampled across all 11 tasks

FiftyOne App playing a HIW-500-LeRobot kitchen organization episode from a Unitree G1, with head camera video, pivot velocity plots, and the 29-dimensional observation state next to the 23-dimensional action vector.

FiftyOne Format Dataset on Hugging Face: Voxel51/HIW-500-LeRobot-48ep
Try it in a Hugging Face Space: harpreetsahota/HIW-500-LeRobot-48ep
Original Dataset: BitRobot/HIW-500-LeRobot (23,743 episodes, ~2TB); raw ROS bag / MCAP recordings of this same data are separately published at BitRobot/HIW-500
Additional Resources: Project page
License: CC-BY-4.0

The Humanoids In-the-Wild dataset delivers precisely what its title promises: hundreds of hours capturing a Unitree G1 humanoid carrying out everyday domestic chores across authentic Southeast Asian residences. It avoids controlled motion-capture setups and uniform apartment stages; instead, each episode introduces shifting floor plans, variable lighting conditions, physical clutter, and distinct operator teleoperation techniques. To ensure broad coverage rather than pulling an isolated single-task shard, we gathered 48 episodes across 6 distinct shard groups, capturing all 11 task categories—ranging from "setting the table" (15 episodes) to "hang hanger" (1 episode).

Available fields:

Video Streams: 3 streams in AV1 format (480×1280 head camera, 480×640 left_wrist, and 480×640 right_wrist views).
Observation State: A 29-DoF observation.state covering hips, knees, ankles, waist, shoulders, elbows, and wrists.
Whole-Body Control: A 23-dimensional observation.state.wbc vector covering pivot pose/velocity, dual end-effector poses, and trigger/squeeze inputs.
Action Vector: A corresponding 23-dimensional action vector.

Unique Structure: The only dataset among the five highlighted where locomotion/stability controls (hip, knee, and ankle joint data) share a vector with manipulation attributes due to its full legged humanoid design.

tavis-head-gr1t2 and tavis-head-reachy2 — Same benchmark, two robots, different state size

FiftyOne App playing a tavis-head-reachy2 ClutterPickCubeTask episode in simulation with head, fixed, and both wrist camera views beside the 36-dimensional state and 19-dimensional action panels.

FiftyOne Format Dataset on Hugging Face: Voxel51/tavis-head-gr1t2-800ep · Voxel51/tavis-head-reachy2-800ep
Try it in a Hugging Face Space: harpreetsahota/tavis-head-gr1t2-800ep · harpreetsahota/tavis-head-reachy2-800ep
Original Dataset: tavis-benchmark org (both are complete, 800/800 episodes each)
Additional Resources: Paper (arXiv:2605.07943) · Code · Paired π₀ checkpoints for head-camera vs. fixed-camera comparison, published under the same tavis-benchmark org
License: CC-BY-4.0

The TAVIS benchmark isolates a critical question often overlooked by robot-learning datasets: does active head and camera movement directly improve manipulation performance, and can this advantage be quantified? To evaluate this, the benchmark records identical tasks using matching four-camera setups across two distinct humanoid torsos: Fourier's GR1T2 and Pollen's Reachy2. Every episode captures parallel streams from both a head-mounted active vision camera and a fixed baseline camera, enabling direct policy comparisons with head-camera integration enabled or disabled using the same demonstration data.

Both platforms evaluate five shared task classes (split into 250/250/100/100/100 episodes):

ClutterPickCubeTask
ClutterPickLiftTask
ConditionalPickTask
MultiShelfScanTask
WaitThenActTask

All demonstrations were gathered in NVIDIA IsaacLab using VR teleoperation recorded at 60fps across identical four-camera configurations, making this dataset the premier benchmark for studying embodiment heterogeneity under controlled conditions.

Dataset Specifications & Structure

Visual Observations: Four 480×640 AV1 camera streams captured at 60fps (OBS_HEAD, OBS_FIXED, OBS_WRIST_LEFT, and OBS_WRIST_RIGHT).
Action Space: A unified 19-dimensional action vector governing bimanual arm motion, neck movement, and gripper control across both robots.
State Representation: Proprioceptive state (observation.state) varies by kinematic structure—44 dimensions for GR1T2 versus 36 dimensions for Reachy2—meaning these vectors cannot be used interchangeably despite the shared action space.
Pose & Language Annotations: Includes dedicated left/right end-effector position and orientation attributes, alongside per-frame language_instruction prompts (featuring 33 distinct instruction strings for Reachy2).

To support head-camera versus fixed-camera evaluation, the official tavis-benchmark organization provides paired π₀ baseline checkpoints.

trex_dataset — Bimanual dexterous manipulation with fingertip tactile sensing

FiftyOne App playing a trex_dataset episode from a Dexmate Vega-1 rig, showing head and wrist cameras, a tactile sensor stream, and per-fingertip tactile force plots next to the 58-dimensional state and action panels.

FiftyOne Format Dataset on Hugging Face: Voxel51/trex-dataset-23ep
Try it in a Hugging Face Space: harpreetsahota/trex-dataset-23ep
Original Dataset: zekaiwang/trex_dataset (5,464 episodes, 1.5TB)
Additional Resources: Paper (arXiv:2606.17055) · Project page
License: MIT

The T-Rex dataset highlights an exceptionally dense sensing modality rarely captured in robotic manipulation: authentic per-fingertip tactile feedback. Built using a stationary bimanual Dexmate Vega-1 setup equipped with Sharpa Wave dexterous hands, each hand features five dedicated tactile sensors. Every one of its 58 state values corresponds strictly to arm or hand joint positions, as the robot's base, torso, and head remained stationary during collection—distinguishing it from the mobile or legged systems featured elsewhere. Teleoperated with Manus gloves and VIVE trackers across 207 distinct objects and 22 motor primitives, this 23-episode slice represents the initial contiguous release of the larger dataset.

Key Data Fields:

Visual Streams: 3 RGB feeds (head_left, left_wrist, right_wrist at 640×360).
Tactile Video Feeds: 20 distinct streams—providing a raw sensor image alongside an estimated deformation map for every fingertip (2 hands × 5 fingers × 2 stream types).
Kinematic Vectors: 58-dimensional observation.state and matching action vectors structured as [L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22] (both share identical dimensions because both record joint positions).
Tactile Force Measurements: A 60-dimensional observation.tactile_force vector capturing a 6-axis wrench per fingertip—a rich modality unique to this dataset among all five showcased.

Decoding Technicalities: To preserve exact physical measurements, the 20 tactile video channels are compressed losslessly via libx264 -qp 0. This enforces the H.264 High 4:4:4 Predictive profile, which standard web browsers cannot natively render in an HTML <video> tag, so they lack video thumbnails in standard web interfaces. Standard decoding workflows in FiftyOne, ffmpeg, PyAV, or torchcodec handle these streams smoothly, while the 3 primary RGB channels use conventional yuv420p encoding and preview seamlessly everywhere.

More LeRobot datasets. These five come from the 11 LeRobot v3.0 datasets for VLA training roundup. Five more are compared in real vs. synthetic LeRobot datasets.

Frequently Asked Questions

What is embodiment heterogeneity and why does it matter for VLA training?

It's the fact that different robots have different action spaces, proprioceptive observation spaces, and sensor modalities — a 7-DoF arm, a 14-DoF bimanual rig, and a legged humanoid's whole-body state simply don't share a representation. It matters because you can't concatenate datasets across embodiments without some form of alignment; a policy trained on one robot's 216-dim state vector has no native way to consume another robot's 36-dim vector, even if both robots are "doing the same task."

Can I combine these 5 datasets into one training set?

Not directly — none of them share a common action or state representation. tavis-head-gr1t2 and tavis-head-reachy2 are the closest thing to compatible here (identical 19-dim action space, same task definitions), but even they differ in observation.state size (44 vs. 36). Combining any of these 5 for cross-embodiment training would require Open X-Embodiment-style coarse alignment — normalizing to a shared representation (e.g. a unified end-effector pose vector) and accepting that the same action values mean physically different things per robot.

Why do tavis-head-gr1t2 and tavis-head-reachy2 have different state dimensions for the same tasks?

Because GR1T2 and Reachy2 are physically different robots with different degrees of freedom, even though the TAVIS benchmark holds the task, camera rig, fps, and collection method constant across both. The 44-vs-36 dimension gap in observation.state is purely a function of each robot's kinematic structure—nothing about the task changed.

Which of these could I combine most easily?

tavis-head-gr1t2 and tavis-head-reachy2 share an identical 19-dim action space and the same 5 task definitions, so a policy that only consumes actions (not full proprioceptive state) could plausibly train across both with minimal remapping. Every other pair in this post differs in both state and action dimensionality, camera count, and often collection method.

Which of these datasets are real vs. simulated?

Real: AgiBotWorld2026, HIW-500-LeRobot, trex_dataset. Simulated (NVIDIA IsaacLab): tavis-head-gr1t2, tavis-head-reachy2.

Which datasets are safe for commercial use?

MIT (trex_dataset) is unrestricted. CC-BY-4.0 (HIW-500-LeRobot, tavis-head-gr1t2, tavis-head-reachy2) requires attribution but allows commercial use. CC-BY-NC-SA-4.0 (AgiBotWorld2026) is non-commercial only.

How to Import LeRobot Datasets into FiftyOne Using Agent Skills

Harpreet Sahota — Wed, 28 Oct 2026 23:41:35 +0000

FiftyOne App showing a LeRobot dataset of robot arm episodes, with the sidebar filtering 800 samples down to 100 by task.

A standardized dataset format solved robot learning’s data fragmentation problem. Here’s how to get one of those datasets into FiftyOne without writing a parser, and what happens when the version doesn’t match what you expect.

Robot learning needs demonstration data the way language models need text, but until recently every lab and vendor recorded it differently: proprietary formats per robot, inconsistent schemas, no shared way to index or search across datasets. Hugging Face built LeRobotDataset to fix that: one standardized format (Parquet for state and action, MP4 for camera feeds) across radically different embodiments, from SO-100 arms to bimanual ALOHA rigs to industrial Kuka arms.

The standardization paid off measurably. SmolVLA, a 450M-parameter policy, was pretrained only on community-uploaded lerobot-tagged datasets on the Hub, and that alone lifted SO-100 task success from 51.7% to 78.3% (source). The format itself had to evolve under that success: the original v2.x layout stored one Parquet file and one MP4 per episode, which started hitting filesystem limits once community datasets reached millions of episodes. lerobot v0.4.0 shipped v3.0 in response, packing many episodes into shared chunked files and supporting streaming without a full download first.

FiftyOne ships fo.types.LeRobotDataset as a first-class import path, built around one idea: a sample is an episode, not a file. The fiftyone-lerobot-import agent skill wraps that native support with the parts a script usually skips: detecting whether a Hub dataset is still on v2.x, running the documented conversion before import, and refusing to create a dataset or install a package without asking first.

Key takeaways

The fiftyone-lerobot-import agent skill imports LeRobotDataset v3.0 recordings into FiftyOne, where one sample is one episode rather than one file.
The skill auto-detects LeRobot's older v2.x layout (meta/episodes.jsonl, meta/tasks.jsonl) and runs the documented conversion to v3.0 before importing, instead of failing on an unsupported-version error.
Two explicit stop gates hold the run, one before installing any package (pyarrow, huggingface_hub, lerobot) and one before creating the FiftyOne dataset.
Tested end to end against lerobot/aloha_sim_insertion_human, a real v2.1-tagged Hub dataset, which converted to v3.0 and imported as 50 episodes of a bimanual ALOHA arm with zero episodes skipped.
The skill is a thin layer over the open-source FiftyOne SDK (fo.Dataset.from_dir, dataset.add_dir), so the same import runs identically from a chat prompt or a checked-in script.

Part one: what is an agent skill, actually?

Strip away the buzzwords and a skill is a folder. Inside it is a markdown file that tells an AI coding agent, step by step, how an expert would do a specific job: which files to check first, which command to run, which traps to avoid, and how to tell success from failure. The agent reads that folder like a new hire reading a training guide, then does the work itself in your terminal, on your data.

Voxel51 packages this kind of runbook for FiftyOne into an open-source repository, installed with one line:

curl -sL skil.sh | sh -s -- voxel51/fiftyone-skills

The installer clones the repo, discovers every skill inside it, and lets you pick which ones your agent should have access to going forward.

This post is about fiftyone-lerobot-import. Point it at a local LeRobot directory or a Hugging Face Hub repo id, and it detects the version, converts if needed, and builds the FiftyOne dataset. That’s a different problem from the one fiftyone-dataset-import’s MCAP path solves. MCAP’s question is a schema scan: what kind of sensor data is this? LeRobot’s question is about versioning: what version of this format is this, and does it need conversion first?

Part Two: A Real LeRobot Dataset, End to End

A skill only proves itself on data that actually exercises its hardest path. A dataset already sitting on v3.0 would import cleanly and tell you nothing about the conversion logic, so we looked for a dataset still tagged v2.1 on the Hub.

The dataset: lerobot/aloha_sim_insertion_human

lerobot/aloha_sim_insertion_human is one of LeRobot’s official example repos: a bimanual ALOHA arm running a simulated peg-insertion task, one task label (“Insert the peg into the socket.”), 50 episodes, 50 fps, 25,000 total frames.

Detail	Value
Repo	lerobot/aloha_sim_insertion_human
Revision pulled	v2.1
Episodes	50
FPS	50
robot_type	aloha
Camera	observation.images.top (video, av1 codec, 480×640)
observation.state / action dims	14 / 14 (bimanual: waist, shoulder, elbow, forearm roll, wrist angle, wrist rotate, gripper, same layout on both arms)
Task	“Insert the peg into the socket.”

What lerobot/aloha_sim_insertion_human contains, pulled at the v2.1 tag.

We pulled the Hub’s v2.1 tag specifically, not main (already v3.0), so the conversion path would get exercised for real instead of simulated.

The prompt and the version-detection moment

The prompt to the agent was one sentence: import lerobot/aloha_sim_insertion_human into FiftyOne.

The skill’s inspection step reads meta/info.json and finds codebase_version: v2.1, along with layout markers like meta/episodes.jsonl and meta/tasks.jsonl. Instead of failing on unsupported versions, the skill detects the older format, hits the packages stop gate, and runs the conversion script (python-m lerobot. scripts. convert_dataset_v21_to_v30) to update the dataset to v3.0, ensuring reliable import even with legacy data.

Hits the packages stop gate and asks before installing lerobot (needed only for the conversion).
Runs the conversion: python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id lerobot/aloha_sim_insertion_human --root <path> --push-to-hub=false.
Keep it honest. The installed lerobot release (v0.6.2) didn’t expose the converter at the documented lerobot.datasets.v30.* module path; it was still at the older lerobot.scripts.* path. The skill’s own troubleshooting notes already anticipate this exact fallback, and it’s what happened on the first real run, not a hypothetical edge case.
Re-checks meta/info.json, confirms codebase_version: v3.0, and the new layout: meta/episodes/chunk-000, meta/tasks.parquet, meta/stats.json.
Hits the creation stop gate and presents the full import plan before touching anything:

LeRobot Import Plan for lerobot/aloha_sim_insertion_human:

Source:
  - codebase_version: v3.0 (converted from v2.1: yes)
  - 50 episodes, 50 fps, robot_type: aloha
  - camera: observation.images.top (video, av1)
  - observation.state: 14 dims, action: 14 dims
  - tasks: 1 ("Insert the peg into the socket.")

Import:
  - Dataset name: lerobot-aloha-sim-insertion-human
  - Episodes: all
  - Persistent: yes

Proceed with import? (yes/no)

The payoff: 50 episodes, zero skipped

dataset.media_type comes back "multimodal". All 50 episodes import; dataset.info["lerobot"]["skipped_episodes"] is an empty list: a clean import, worth stating plainly rather than manufacturing a gap that isn’t there. Every episode runs exactly 10 seconds (dataset.bounds("duration") returns (10.0, 10.0)), which checks out at 50 fps × 500 frames/episode.

Opening episode 0 in the App shows the observation.images.top tile: two robot arms working over a blue and red object, with the episode header reading “Insert the peg into the socket. · 10s.” That’s what “one sample equals one episode” looks like in the viewer, not an abstraction: a single click surfaces the whole 10-second demonstration, not one video frame.

Part Three: The Same LeRobot Import, in Code

A chat prompt is a good way to explore a dataset you haven’t seen before, but a pipeline that runs on every new upload needs code, not a conversation. Everything the skill did above sits on top of the open-source FiftyOne SDK. Here’s the full pipeline, runnable from a fresh environment against the same Hub dataset used in Part Two.

Step 1: Install the dependencies

pip install "pyarrow>=10.0.0" huggingface_hub lerobot

lerobot is only needed for the v2.1 → v3.0 conversion below. A dataset already on codebase_version: v3.0 only needs pyarrow and huggingface_hub.

Step 2: Download from the Hub

fo.Dataset.from_dir reads from a local directory, not directly from the Hub, so pull the files down first:
from huggingface_hub import snapshot_download

local_dir = snapshot_download(
    repo_id="lerobot/aloha_sim_insertion_human",
    repo_type="dataset",
    revision="v2.1",
    local_dir="/abs/path/lerobot/aloha_sim_insertion_human",
)

Step 3: Convert v2.1 to v3.0

python -m lerobot.scripts.convert_dataset_v21_to_v30 \
    --repo-id lerobot/aloha_sim_insertion_human \
    --root /abs/path/lerobot/aloha_sim_insertion_human \
    --push-to-hub=false

Skip this step for any dataset already tagged v3.0 on the Hub; for example, pull revision="v3.0" in Step 2 instead.

Step 4: Import into FiftyOne

import fiftyone as fo

dataset = fo.Dataset.from_dir(
    dataset_dir=local_dir,
    dataset_type=fo.types.LeRobotDataset,
    name="lerobot-aloha-sim-insertion-human",
    persistent=True,
)

assert dataset.media_type == "multimodal"
fo.launch_app(dataset)

Adding a second recording session to the same dataset is the same four steps repeated against the new source directory, then dataset.add_dir(dataset_dir=..., dataset_type=fo.types.LeRobotDataset) instead of from_dir.

No Parquet reader, no video decode loop, no custom viewer. The importer reads meta/info.json and meta/episodes/*.parquet, nothing in the video files, so it stays fast even on large sources. The skill’s stop gates are a conversation; the SDK calls underneath are the same ones either way.

Where This Leaves You

Every team re-solves “is my data in the format my tools expect” by hand, one script at a time, and LeRobot is just the latest format to need that check. fiftyone-lerobot-import turns that check into something an agent already knows how to run correctly against your files, every time.

Dataset: lerobot/aloha_sim_insertion_human, part of LeRobot’s official example datasets, Hugging Face
Skill repo: voxel51/fiftyone-skills, installed the same one-line way as Part One

Frequently Asked Questions

How do you import a LeRobot dataset into FiftyOne?

Two ways: ask an agent with the fiftyone-lerobot-import skill, or call fo.Dataset.from_dir(..., dataset_type=fo.types.LeRobotDataset) directly. Both require the dataset to be on codebase_version: v3.0.

What’s the difference between LeRobot v2.x and v3.0?

v2.x stores one Parquet file and one MP4 per episode. v3.0 packs many episodes into shared chunked files and supports streaming. fo.types.LeRobotDataset only supports v3.0. v2.x sources need conversion first.

Do I need the lerobot package to import into FiftyOne?

Only for converting a v2.x source to v3.0. A straight v3.0 import needs only pyarrow>=10.0.0.

How does FiftyOne represent a LeRobot dataset?

One sample per episode, not per file, with media_type == "multimodal" and fields like episode_index, task, duration, and fps.

What happens if some episodes in the source are malformed?

They’re reported under dataset.info["lerobot"]["skipped_episodes"] instead of being silently dropped or silently accepted.

From Franka Arms to Humanoids: 11 LeRobot v3.0 Datasets for VLA Training in 2026

Harpreet Sahota — Wed, 28 Oct 2026 23:40:28 +0000

A field guide to 11 robot-learning datasets, from Franka arms to full humanoids and real teleoperation to simulation, reformatted into LeRobotDataset v3.0 and loaded into FiftyOne with their exact fields and per-frame streams.

A vision-language-action (VLA) model doesn't learn from a folder of images.

It learns from episodes: a camera stream, a robot's joint state, and the action it took, all stamped to the same clock and lined up frame by frame. Get that alignment wrong, and the model can't tell which arm motion caused which visual change. The training signal disappears.

This makes data engineering a prerequisite for modeling.

We pulled together 11 datasets in LeRobot v3.0, from a Franka arm to a Unitree G1 humanoid and a fully synthetic manipulation suite. We reformatted or subsetted each dataset and loaded it into FiftyOne, where you can inspect video streams, state/action vectors, and task labels before downloading.

This guide explains the data problem, compares LeRobot with the raw-sensor-log alternative MCAP, breaks down the FiftyOne schema, and links each dataset to its live FiftyOne Space, paper, and license.

Key takeaways

11 datasets, all loadable today with fiftyone.utils.huggingface.load_from_hub, spanning single Franka arms to full humanoids (HIW-500-LeRobot on Unitree G1, tavis-head-gr1t2, tavis-head-reachy2) to a bimanual dexterous-hand platform (trex_dataset on Dexmate Vega-1).
4 of the 11 FiftyOne exports carry the complete source episode set (robocasa-MG_100: 2,400/2,400; robotwin_unified: 27,500/27,500; tavis-head-gr1t2 and tavis-head-reachy2: 800/800 each). The other 7 are disk-budget-constrained subsets of much larger sources — as small as 10 of AgiBotWorld2026's 102,078 episodes and 23 of trex_dataset's 5,464.
6 contain real-world teleoperation or human-collected data; 5 are simulated (InternData-A1, robocasa-MG_100, robotwin_unified, tavis-head-gr1t2, tavis-head-reachy2). InternData-A1's preview clips look photorealistic despite being 100% simulator output.
droid_3d's live FiftyOne export is RGB-only — the full source dataset's depth video and point-cloud streams use dtype values (depth_video, pointcloud) that FiftyOne's LeRobot importer doesn't recognize, so they're absent from the FiftyOne dataset even though they exist upstream.
Licenses vary, sometimes within one export: MIT (droid_3d, trex_dataset), Apache-2.0 (robocasa-MG_100, robotwin_unified), CC-BY-4.0 (fmb_multi, HIW-500-LeRobot, tavis-head-*), and CC-BY-NC-SA-4.0 (AgiBotWorld2026, InternData-A1). The 28-episode rh20t_cfg1 export mixes CC-BY-SA-4.0 and CC-BY-NC-4.0 by episode.

Why data organization matters more than architecture

Source: arXiv 2505.04769

A vision-language-action (VLA) model fuses three things:

a vision encoder that turns camera frames into tokens
a language encoder that turns instructions into tokens
and an action decoder that turns the fused representation into joint commands or end-effector deltas.

Most VLAs bolt this onto a pretrained vision-language backbone, because CLIP- or SigLIP-scale visual and linguistic priors are cheap to get from the internet.

Action grounding is expensive. A video of a person picking up a mug has no joint angles, end-effector pose, or gripper command attached. A robot policy needs that information to reproduce the motion. Vision-language models (VLMs) can draw on web-scale data; VLAs don't have equivalent action data.

Source: Generated using NanoBanana 2

That gap creates five differences from typical vision-language training data:

Temporal structure. Episodes are ordered sequences, not shuffleable independent samples — a policy has to model how actions at time T affect the state at T+1.
Proprioception. Joint positions, velocities, and end-effector poses are low-dimensional (10-50 values) but essential — vision shows where the object is, not where the robot's own arm is.
Action grounding. Understanding "pick up the cup" doesn't provide the motor command sequence needed to grasp that cup in that configuration.
Embodiment heterogeneity. A 7-DoF single arm, a 14-DoF bimanual rig, and a humanoid's 29-DoF whole-body state don't share an action space, so the datasets require remapping before concatenation.
Data scarcity. Teleoperation is slow and expensive to collect, so robot datasets run to thousands or millions of trajectories — not the billions of tokens or images that make LLM/VLM pretraining work.

Without a common structure, these differences block dataset combination.

RLDS provides a lossless episode/step hierarchy, Open X-Embodiment coarsely aligns actions into 7-DoF vectors across 22 embodiments, and LeRobot packages synchronized episodes in a compressed format. The 11 datasets below share that organization, which exposes the remaining differences in embodiments, sensors, state, and action.

LeRobot v3.0 vs. MCAP: two different jobs

If you've read our companion piece, 10 Multimodal MCAP Datasets in 2026, you've seen the other half of this problem. MCAP and LeRobot both solve robot-data organization, but for opposite inputs:

.	MCAP	LeRobot v3.0
What it stores	Continuous, multi-rate sensor log (camera, LiDAR, IMU, GPS, all on their own clocks)	Discrete, synchronized episodes (state/action/vision aligned to one frame grid)
Compression	Lossless (LZ4/Zstandard at the chunk level)	Lossy video compression (AV1/H.264), trades fidelity for size
Typical use	SLAM, sensor fusion, perception benchmarking, raw log archival	Imitation learning, VLA pretraining and fine-tuning
One sample is	One continuous recording (minutes long), decoded live	One episode (seconds to a couple minutes), pre-chunked into fixed frame steps
Streaming	Chunk-indexed seeking within a file	StreamingLeRobotDataset streams directly from the Hub, no local download
In FiftyOne	dataset.media_type == "multimodal", one sample = one continuous recording	dataset.media_type == "multimodal", one sample = one episode, per-frame data resolved through media_reference

How MCAP and LeRobot v3.0 differ across storage, compression, and typical use.

If you have 20 minutes of unsynchronized LiDAR, camera, and IMU data and need to seek to timestamp 4:32, use MCAP. If you have labeled robot demonstrations and need synchronized state/action/vision triples for training, use LeRobot. The formats cover different parts of the robot-data pipeline.

Source: Generated using NanoBanana 2

LeRobot v3.0 changes how it packages episodes. Version 2.1 used one Parquet file and one MP4 per episode, which strains filesystem limits at millions of episodes. Version 3.0 packs many episodes into shared Parquet and MP4 files and resolves them through relational metadata. That structure makes a 102,078-episode, 13.6TB release such as AgiBotWorld2026 practical to package.

The FiftyOne LeRobot schema: what's a sample, what's a stream

Every one of the 11 datasets below is loaded into FiftyOne the same way, so the field schema is consistent across all of them. Each is a multimodal FiftyOne dataset where one sample equals one episode. The sample-level fields you can query, filter, and sort on in the App are:

Field	FiftyOne type	What it holds
id	ObjectIdField	FiftyOne sample id
media_reference	MediaReferenceField	Pointer into the LeRobot source's data/.parquet, videos//*.mp4, and meta/ files for this episode — resolved on demand, not duplicated per sample
tags	ListField(StringField)	FiftyOne sample tags (empty by default)
metadata	Metadata	Standard FiftyOne sample metadata (size_bytes, mime_type)
created_at / last_modified_at	DateTimeField	FiftyOne bookkeeping timestamps
episode_index	IntField	Episode index within the export
task	StringField	Primary task label for the episode (free text, not fo.Classification — see below)
tasks	ListField(StringField)	Full list of task label(s) for the episode
length	IntField	Number of frames in the episode
duration	FloatField	Episode duration in seconds (length / fps)
robot_type	StringField	Robot platform identifier (empty/null for several of these 11 — see per-dataset notes)
fps	FloatField	Recording frame rate

Sample-level fields in a FiftyOne LeRobot dataset, and what each one holds.

Why is task a string instead of a Classification? These datasets store task identity as free text: an instruction, a skill-primitive name, or a placeholder such as "task 1". A fo.Classification field would imply a closed label set with confidence scores. robotwin_unified alone has 23,559 distinct task strings across 27,500 episodes. Use task and tasks for text search and grouping.

Where does per-frame data live? Video streams (observation.images.*), proprioceptive state (observation.state, observation.state.*), actions, and per-frame sensor data stay in the referenced LeRobot data/*.parquet and videos/*/*.mp4 files. FiftyOne surfaces them in the State & Action, Streams, and Statistics tabs as you scrub through an episode. A 60fps humanoid episode can carry a 44-dimensional state vector and 4 video streams per frame, so flattening these values into top-level sample fields would not scale. media_reference avoids that duplication.

The 11 LeRobot datasets at a glance

"Episodes" below reflects what's actually in each live FiftyOne export, not the full upstream source. 4 of 11 are complete imports; the other 7 are disk-budget-constrained subsets (as small as 10-68 episodes) selected for shard contiguity, not statistical representativeness — each entry states the selection method and the full source size for comparison.

Dataset	Description	Sensors	License (this export)	Real or Sim
AgiBotWorld2026	Real-world dual-arm humanoid manipulation collected via RL-style human-intervention rollout on the AgiBot G2 platform	RGB + depth, 7 cams, RL signals	CC-BY-NC-SA-4.0	Real
droid_3d	RGB-only slice of a large real-world DROID manipulation dataset whose full release adds depth and point clouds for 3D-aware policies	RGB only, 3 cams	MIT	Real
fmb_multi	Contact-rich Franka Panda assembly demonstrations across three boards, phase-labeled by skill primitive	RGB + force/torque, 4 cams	CC-BY-4.0	Real
HIW-500-LeRobot	Whole-body Unitree G1 humanoid teleoperation performing household chores across real homes in Southeast Asia	RGB stereo, 3 cams	CC-BY-4.0	Real
InternData-A1	A single Franka articulation task ("close the laptop") from a fully synthetic, zero-demonstration compositional simulation pipeline	RGB, 2 cams	CC-BY-NC-SA-4.0	Sim
rh20t_cfg1	Real Flexiv-arm teleoperation with ten synchronized camera views, part of RH20T's one-shot skill-transfer dataset	RGB, 10 cams	Mixed CC-BY-SA-4.0 / CC-BY-NC-4.0	Real
robocasa-MG_100	Synthetic kitchen manipulation generated via MimicGen augmentation of real human demonstrations	RGB, 3 cams	Apache-2.0	Sim
robotwin_unified	Bimanual ALOHA-embodiment simulation generated by an LLM code-gen agent and VLM observer, with heavy domain randomization	RGB, 3 cams	Apache-2.0	Sim
tavis-head-gr1t2	Simulated active-vision humanoid benchmark (GR1T2) comparing head-mounted vs. fixed cameras on identical manipulation tasks	RGB, active-vision head + fixed + 2 wrist cams	CC-BY-4.0	Sim
tavis-head-reachy2	The same active-vision benchmark as tavis-head-gr1t2, run on a Reachy2 humanoid instead	RGB, active-vision head + fixed + 2 wrist cams	CC-BY-4.0	Sim
trex_dataset	Bimanual dexterous manipulation with real per-fingertip tactile sensing on a Dexmate Vega-1 robot	RGB + tactile, 3 RGB + 20 tactile streams	MIT	Real

The 11 LeRobot v3.0 datasets, with sensors, license, and whether the data is real or simulated.

The 11 LeRobot datasets in detail

AgiBotWorld2026: real-world dual-arm manipulation with RL collection signals

FiftyOne app playing an AgiBotWorld2026 episode, with seven camera panels including head fisheyes, a reward signal plot, and the 216-dimension observation state vector.

FiftyOne Format Dataset on Hugging Face: Voxel51/AgiBotWorld2026-sample-10ep
Try it in a Hugging Face Space: harpreetsahota/AgiBotWorld2026-sample-10ep
Original Dataset: agibot-world/AgiBotWorld2026 (102,078 episodes, 13.6TB)
Additional Resources: Project homepage: agibot-world.com
License: CC-BY-NC-SA-4.0

AgiBot World is one of the largest real-world humanoid manipulation datasets released to date.

A team collected long-horizon household and commercial tasks using the AgiBot G2 dual-arm platform. They deployed it in real homes, retail spaces, and offices—not just a single controlled lab.

The 2026 release has three collection modes:

Pure imitation learning
RL-style rollout with human intervention (HG-DAgger)
Richer multi-signal interaction logging

And three annotation layers on top of the raw episodes:

Subtask segmentation
2D bounding boxes
Step-level language instructions

This FiftyOne export pulls from the HG-DAgger reinforcement-learning slice specifically, so you're looking at a robot mid-training, complete with reward signals and human-correction flags, not a polished expert demonstration. All 10 episodes come from a single ReinforcementLearning/Home/task_12192/HG-DAgger archive, one task ("Insert the loose change into the coin slot of the piggy bank dagger").

Because this is the RL collection category, every frame carries collection-process signals most imitation-learning datasets don't: reward, done, succeed, intervened (whether a human corrected the policy at that frame), truncated, and action_prob.

Fields you get: 7 camera streams (top_head, hand_left, hand_right, two head fisheyes, a rear fisheye, and a head_depth stream — declared dtype: video with gray16be pixel format, so it imports like RGB despite being depth data), a 216-dim observation.state vector, and a 44-dim action vector.

Known quirks. The 13.6TB source repo isn't a standard flat LeRobot layout — it ships as per-task .tar.gz archives under ImitationLearning/, ReinforcementLearning/, and RichInteraction/, and these three categories declare incompatible observation.state shapes (193 / 216 / 169 respectively), so a single LeRobot v3 export can only cover one category at a time. This export stays inside ReinforcementLearning. The source ships LeRobot v2.1 and was converted with lerobot.scripts.convert_dataset_v21_to_v30; AgiBot's three custom annotation layers (key_frame, instruction_segments, subtask/bounding-box labels) are dropped by that converter and aren't present here.

HIW-500-LeRobot — Humanoid teleoperation in real homes, sampled across all 11 tasks

FiftyOne app playing an HIW-500-LeRobot kitchen organization episode from a Unitree G1 humanoid, with head and wrist camera streams above a whole-body-control action plot.

FiftyOne Format Dataset on Hugging Face: Voxel51/HIW-500-LeRobot-48ep
Try it in a Hugging Face Space: harpreetsahota/HIW-500-LeRobot-48ep
Original Dataset: BitRobot/HIW-500-LeRobot (23,743 episodes, ~2TB); raw ROS bag / MCAP recordings of this same data are separately published at BitRobot/HIW-500
Additional Resources: Project page
License: CC-BY-4.0

Humanoids In-the-Wild is exactly what it sounds like.

The dataset captures hundreds of hours showing a Unitree G1 humanoid handling domestic tasks throughout actual residences in Southeast Asia. Environments, illumination, object arrangements, and individual operator control patterns naturally fluctuate between demonstrations—by design, offering raw, uncurated field recordings rather than sanitized laboratory trials. This curated selection spans all 11 task classes from the source collection (ranging from table preparation to hanging keys on pegs), delivering a representative window into the complete 23,743-trajectory dataset without requiring a multi-terabyte retrieval.

Rather than extracting the minimal contiguous block (which would limit coverage to a single activity), we sampled 48 demonstrations across 6 distinct file groups to ensure representation across every task class—from "setting the table" (15 demonstrations) to "hang hanger" (a single demonstration).

Fields you get: 3 camera streams (head at 480×1280, left_wrist/right_wrist at 480×640, all AV1), a 29-DoF observation.state (hips, knees, ankles, waist, shoulders, elbows, wrists), a 23-dim observation.state.wbc whole-body-control vector (pivot velocity/pose + both end-effector poses + trigger/squeeze), and a matching 23-dim action.

Known quirks. The source's language_persistent and language_events fields carried the dataset's 161 fine-grained subtask labels and 148K+ annotations; this export includes only the 11 coarse task categories, not the fine-grained subtask text. Camera intrinsics/extrinsics mentioned in the source docs also aren't exposed as FiftyOne fields.

tavis-head-gr1t2 and tavis-head-reachy2 — Active-vision humanoids, full datasets, in simulation

FiftyOne app playing a simulated GR1T2 humanoid ClutterPickCubeTask episode from tavis-head-gr1t2, with head, fixed, and wrist camera views and the 44-dimension state vector.

FiftyOne Format Dataset on Hugging Face: Voxel51/tavis-head-gr1t2-800ep · Voxel51/tavis-head-reachy2-800ep
Try it in a Hugging Face Space: harpreetsahota/tavis-head-gr1t2-800ep · harpreetsahota/tavis-head-reachy2-800ep
Original Dataset: tavis-benchmark org (both are complete, 800/800 episodes each)
Additional Resources: Paper (arXiv:2605.07943) · Code · Paired π₀ checkpoints for head-camera vs. fixed-camera comparison, published under the same tavis-benchmark org
License: CC-BY-4.0

TAVIS asks a specific, narrow question that most robot-learning datasets don't bother to isolate: does letting a humanoid move its own head/camera during manipulation actually help, and can you measure that benefit directly?

The benchmark pairs identical tasks and identical camera rigs across two different humanoid torsos — Fourier's GR1T2 and Pollen's Reachy2 — and records both a head-mounted "active vision" stream and a fixed-camera view for every episode, so you can train and compare policies with the head-camera on or off using the same demonstrations. It's simulation-only (built in NVIDIA IsaacLab), VR-teleoperated, and small (800 episodes per robot) by design, since the goal is controlled comparison, not scale.

Both robots share 5 task classes: ClutterPickCubeTask, ClutterPickLiftTask, ConditionalPickTask, MultiShelfScanTask, WaitThenActTask (250/250/100/100/100 episode split).

Fields you get: 4 camera streams (OBS_HEAD, OBS_FIXED, OBS_WRIST_LEFT, OBS_WRIST_RIGHT, all 480×640 AV1 at 60fps), a 19-dim action (bimanual arm targets + neck + grippers), observation.state (44-dim for GR1T2, 36-dim for Reachy2 — the two robots have different DoF counts, so don't assume the vectors are interchangeable), separate left/right end-effector position and orientation fields, and a per-frame language_instruction string (33 distinct values for Reachy2).

FiftyOne app playing the same active-vision ClutterPickCubeTask on a simulated Reachy2 humanoid, with four camera panels and per-frame state and action plots.

Known quirks. robot_type is null in both sources despite the org page naming the robots explicitly. A separate tavis-hands-* pair (300 episodes each, testing local-occlusion perception via wrist cameras rather than head search) exists in the same org but isn't part of this roundup.

trex_dataset — Bimanual dexterous manipulation with fingertip tactile sensing

FiftyOne app playing a T-Rex bimanual episode on a Dexmate Vega-1, showing an RGB camera view beside a grayscale fingertip tactile stream and per-fingertip tactile force readings.

FiftyOne Format Dataset on Hugging Face: Voxel51/trex-dataset-23ep
Try it in a Hugging Face Space: harpreetsahota/trex-dataset-23ep
Original Dataset: zekaiwang/trex_dataset (5,464 episodes, 1.5TB)
Additional Resources: Paper (arXiv:2606.17055) · Project page
License: MIT

T-Rex is built around a sensing modality almost no other manipulation dataset bothers with at this density: real per-fingertip tactile sensing.

This dataset captures five sensors per hand, on a bimanual Dexmate Vega-1 robot fitted with Sharpa Wave dexterous hands. Each episode includes not just RGB video but 20 additional tactile video streams (a raw sensor image and a deformation map per fingertip), plus 6-axis force/torque readings at each fingertip, collected via Manus glove and VIVE tracker teleoperation across 207 objects and 22 distinct motor primitives.

If you care about contact-rich, touch-conditioned manipulation rather than vision-only policies, this is one of the only public datasets built specifically to support it.

The 23-episode export here is the first contiguous shard of the full 5,464-episode dataset.

Fields you get: 3 RGB camera streams (head_left, left_wrist, right_wrist, 640×360) plus 20 tactile video streams — a raw sensor image and an estimated deformation map per fingertip, 2 hands × 5 fingers × 2 stream types — a 58-dim observation.state and matching action ([L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22] joint layout), and a 60-dim observation.tactile_force (per-fingertip 6-axis wrench).

Known quirks. The 20 tactile streams are encoded losslessly (libx264 -qp 0, since pixel values are physically meaningful sensor readings), which forces the H.264 High 4:4:4 Predictive profile — most browsers can't decode this in a plain <video> element. As a result, these streams show no thumbnail in a generic viewer. FiftyOne, ffmpeg, PyAV, or torchcodec decode them fine; the 3 RGB streams use standard yuv420p and preview everywhere.

fmb_multi — Contact-rich Franka assembly, phase-labeled

FiftyOne app playing an fmb_multi Franka Panda assembly episode, with four 256 by 256 camera views of a colored assembly board and end-effector force and torque plots.

FiftyOne Format Dataset on Hugging Face: Voxel51/fmb-multi-302ep
Try it in a Hugging Face Space: harpreetsahota/fmb-multi-302ep
Original Dataset: robot-lev/fmb_multi (1,804 episodes)
Additional Resources: Paper (arXiv:2401.08553) · Project page
License: CC-BY-4.0

The Functional Manipulation Benchmark was built to stress-test generalization in contact-rich assembly.

Moving beyond basic pick-and-place actions, this benchmark evaluates complex insertion and regrasping sequences where force feedback plays a critical role. The fmb_multi subset focuses on multi-object manipulation across three distinct assembly boards. Each trajectory features a Franka Panda robotic arm executing a structured skill workflow—approach, grasp, insert, place, and regrasp—while capturing 6-axis end-effector force/torque dynamics, along with visual and proprioceptive signals. Rather than aiming for sheer volume, the benchmark remains intentionally compact (under 2,000 total episodes) to prioritize testing policy generalization across diverse objects and board configurations.

The exported 302-episode subset is distributed almost uniformly across the three boards, containing 100, 100, and 102 episodes, respectively.

Fields you get: 4 camera streams (side_1, side_2, wrist_1, wrist_2, 256×256 AV1, converted from source BGR), observation.state (28-dim: joint position + velocity + ee_pose + ee velocity + gripper), separate observation.state.joint_position/.ee_pose/.gripper fields, observation.force/observation.torque (end-effector frame), a flattened 6×7 observation.jacobian, and a 7-dim cartesian action.

Known quirks. task here is the skill primitive active at frame 0 only — not an episode summary. Use tasks for the full ordered sequence an episode passes through. Depth was dropped by the upstream port (not this export); the source FMB release includes 4 depth maps per frame that never made it into fmb_multi. fps: 10 is nominal — the original .npy files carry no real timestamps, so frames map 1:1 onto a synthetic grid rather than a measured rate. Per-episode board/object_id metadata lives in the source's meta/fmb_episodes.json, not as a FiftyOne field — join back to it by episode_index if you need it.

rh20t_cfg1 — Flexiv arm, ten camera views, license split by episode

FiftyOne app playing an rh20t_cfg1 episode from a Flexiv arm, with ten synchronized camera views and a force and torque plot beneath the state and action panel.

FiftyOne Format Dataset on Hugging Face: Voxel51/rh20t-cfg1-28ep
Try it in a Hugging Face Space: harpreetsahota/rh20t-cfg1-28ep
Original Dataset: robot-lev/rh20t_cfg1 (4,258 episodes; RH20T's full dataset spans 110,000+ sequences across many robot configs, of which cfg1/Flexiv is one)
Additional Resources: Paper (arXiv:2307.00595) · Project page
License: mixed, 16 episodes CC-BY-SA-4.0 / 12 episodes CC-BY-NC-4.0 in this export

RH20T tests one-shot skill transfer: can a robot learn a new task from just a single demonstration?

To answer this, it collects an enormous spread of over 110,000 manipulation sequences across diverse robot platforms and camera setups. The rh20t_cfg1 subset focuses on a single Flexiv arm, paired with ten synchronized camera views per episode. RH20T licenses its data per scene, and this port preserves that original split rather than applying a blanket license. Roughly half the scenes permit commercial use, while the rest are non-commercial only.

Check meta/rh20t_episodes.json's folder field to identify the scene and licensing terms for each episode.

Fields you get: 10 camera streams (cam_<serial>, 360×640 AV1, one per fixed RealSense-style camera), observation.state (15-dim: ee_pose 7 + joint 7 + gripper 1 — note joint is all-zero for episodes where the source's has_joint flag was false), observation.force/torque plus a separate robot-mounted observation.robot_ft (6-axis), an 8-dim action (next ee_pose + gripper target), and a per-episode meta.rating quality score from the original RH20T annotation pass.

Known quirks. task values here are literally "task 1" through "task 6" — the LeRobot port never carried over RH20T's original natural-language task descriptions, only a numeric cluster id. Treat task as an opaque grouping key, not a description. Audio sidecars from the source (one .wav per episode) are deliberately excluded — they sit outside the declared LeRobot schema entirely, and the source data includes volunteer-recorded human-robot interaction with faces/voices that warrants care regardless.

robocasa-MG_100 — Full synthetic kitchen dataset via MimicGen

FiftyOne app grid view of the robocasa-MG_100 dataset, showing dozens of synthetic kitchen episode thumbnails beside a dashboard of task counts and episode duration.

FiftyOne Format Dataset on Hugging Face: Voxel51/robocasa-MG_100
Try it in a Hugging Face Space: harpreetsahota/robocasa-MG_100
Original Dataset: DAVIAN-Robotics/robocasa-MG_100 — all 2,400 episodes, full dataset, no subsetting
Additional Resources: Paper (arXiv:2406.02523) · MimicGen paper (arXiv:2310.17596) · Homepage
License: Apache-2.0

RoboCasa constructs 120 fully procedurally generated kitchen environments across ten distinct layouts and a dozen aesthetic styles to test how far synthetic augmentation can stretch a limited collection of human demonstrations.

The MG_100 dataset shows the MimicGen pipeline in action: rather than logging thousands of additional manual rollouts, researchers broke down 1,250 human teleoperation demonstrations into modular, object-centric primitives. They then synthesized 100 new rollouts for every task by stitching those components back together inside unseen spatial setups, keeping only successful executions. While synthetic, this approach directly addresses a key robotics bottleneck: evaluating how effectively automated generation can replace labour-intensive physical data collection. Specifically, MG_100 mirrors the paper's Generated-100 setup, taking a 1/30 random slice from the broader MimicGen output to yield 100 demonstrations across 24 atomic tasks (spanning 688,750 total frames, with trajectory lengths between 89 and 838 frames).

Fields you get: 3 camera streams (2 static workspace views + 1 wrist-mounted eye-in-hand, all 128×128 AV1), a 16-dim observation.state (base-to-eef position/quaternion + gripper qpos + base position/quaternion), a 12-dim delta-pose+gripper+base action, and a next.done termination flag.

Known quirks. Task episode counts are imbalanced by design — fixed-object tasks (e.g. "turn on the sink faucet") have exactly 100 episodes, while pick-and-place tasks split across many object-category variants, as few as 1 episode for some combinations, because MimicGen samples object identity per generation attempt. NavigateKitchen, the paper's 25th atomic task, is absent from every MimicGen-generated release (including this one) because MimicGen can't synthesize mobile-navigation trajectories — that's a property of the source data, not something excluded on import.

robotwin_unified — 27,500-episode bimanual simulation with domain randomization

FiftyOne app playing a robotwin_unified bimanual ALOHA episode with the instruction to lift a plastic bottle, showing the 14-degree-of-freedom state and action vectors.

FiftyOne Format Dataset on Hugging Face: Voxel51/robotwin-unified-27500ep
Try it in a Hugging Face Space: harpreetsahota/robotwin-unified-27500ep
Original Dataset: lerobot/robotwin_unified — this export is all 27,500 episodes, full aloha-embodiment portion, no subsetting
Additional Resources: Paper (arXiv:2506.18088) · Simulator/generator code (MIT)
License: Apache-2.0

RoboTwin 2.0 takes data generation a step beyond MimicGen-style augmentation—it removes the human demonstration requirement entirely.

An LLM writes executable task programs from a natural-language instruction, a SAPIEN-based simulator runs them, and a vision-language model watches the execution frame-by-frame to catch failures and drive the code-generation agent to fix its own mistakes, iterating until the task succeeds reliably. robotwin_unified is the bimanual ALOHA slice of that pipeline's output: 50 dual-arm tasks, heavy domain randomization across textures, lighting, and clutter, and a task-instruction vocabulary so varied that nearly every one of its 27,500 episodes has a unique description (23,559 distinct task strings, near one-to-one).

Domain randomization spans scene clutter (731-object asset library), background textures (11,000 AI-generated textures), lighting, tabletop height, and instruction phrasing.

Fields you get: 3 camera streams (cam_high overhead + cam_left_wrist/cam_right_wrist, 480×640 AV1), a 14-DoF observation.state (left/right waist, shoulder, elbow, forearm_roll, wrist_angle, wrist_rotate, gripper), and a matching 14-dim action.

Known quirks. Entirely simulated (SAPIEN-based) — the source paper itself cautions against treating performance here as a proxy for real-world deployment readiness without separate sim-to-real validation. Because near every episode has a unique task string, don't try to build a Classification schema over task — text search/filtering is the right access pattern here, not label grouping.

droid_3d — RGB-only in FiftyOne; depth and point clouds need a different loader

FiftyOne app grid view of the droid_3d RGB export, with episode thumbnails each captioned by its natural-language task description.

FiftyOne Format Dataset on Hugging Face: Voxel51/droid-3d-rgb-68ep
Try it in a Hugging Face Space: harpreetsahota/droid-3d-rgb-68ep
Original Dataset: ZibinDong/droid_3d (58,201 episodes, ~1.3TB)
Additional Resources: Paper (arXiv:2505.10105) — the paper this depth/point-cloud extension supports · Reader package for the full 3D data
License: MIT

DROID is one of the most widely used real-robot manipulation datasets in the field.

This dataset is assembled by pooling data collection across many labs on a standardized rig, and droid_3d builds on that foundation by adding a genuinely useful extension: synchronized depth video and quantized point clouds alongside the original RGB, purpose-built to support 3D-aware manipulation and representation-learning research (EmbodiedMAE). The catch for anyone working purely in FiftyOne or stock LeRobot is that those depth and point-cloud streams use dtypes neither tool currently decodes, so you get the RGB-and-language backbone of a dataset whose real distinguishing feature lives just outside this export.

The importer recognizes video and image dtypes, while the source declares depth and point clouds as depth_video and pointcloud, which have no built-in FiftyOne import path.

Fields you get: 3 RGB camera streams (wrist, external_0, external_1, 224×398 AV1), an 8-dim action, and up to 3 independent natural-language task descriptions per episode (language_1/_2/_3 — different phrasings/emphasis of the same action, not paraphrases; only language_1 is surfaced as the sample-level task).

Known quirks. If you need the depth or point-cloud streams, this FiftyOne export won't have them at any episode count — install lerobotdataset3d (pip install git+https://github.com/ZibinDong/lerobotdataset3d.git) and load the full ZibinDong/droid_3d repo directly with LeRobotDatasetDepthPointcloud instead. robot_type is empty here because DROID spans multiple robot platforms, with no single embodiment tag recorded at the dataset level.

InternData-A1 — One task, one embodiment, out of 1,488 possible exports

FiftyOne app grid view of the InternData-A1 close-the-laptop export, showing synthetic Franka episodes rendered across many different floor and surface textures.

FiftyOne Format Dataset on Hugging Face: Voxel51/interndata-a1-close-the-laptop-369ep
Try it in a Hugging Face Space: harpreetsahota/interndata-a1-close-the-laptop-369ep
Original Dataset: InternRobotics/InternData-A1 (637,498 trajectories, 401.4M frames, 4 embodiments, 70 tasks)
Additional Resources: Paper (arXiv:2511.16651) · Homepage
License: CC-BY-NC-SA-4.0

InternData-A1 pushes the synthetic-data bet furthest of any dataset in this series: no human demonstrations anywhere in its pipeline, not even as a seed.

Instead, a fully decoupled simulation stack composes scenes, objects, and scripted skill policies from an asset library. It renders the results at a scale (637,498 trajectories, 401.4M frames) that would be genuinely difficult to reach through teleoperation. The paper's headline result is what makes it worth paying attention to beyond the scale: a policy pretrained exclusively on this synthetic data reportedly transfers zero-shot to several real-world tasks, which is real (if early) evidence that scale and diversity in simulation can substitute for at least some real-world collection. The source repo contains 1,488 independent LeRobot v3.0 datasets, one per <embodiment>/<task_category>/<task> directory, each with its own schema.

This export covers franka-1/articulation_tasks/close_the_laptop: 369 of that task's 578 episodes, the largest complete shard-0 set obtainable without downloading every shard. Every episode shares the instruction "Close the laptop."

Fields you get: 2 camera streams (images.rgb.head 360×640, images.rgb.hand 480×640), camera intrinsics/extrinsics vectors, and a long list of per-component state/action fields (states.joint.position, actions.gripper.pose, master_actions.* for the teleop leader-arm commands, etc.) — plus two derived convenience fields, a 49-dim observation.state and 43-dim action, packed post-import specifically so FiftyOne's State & Action viewer tile (which only renders features literally named observation.state/action) can display this dataset's proprioception at all.

Known quirks. 100% synthetic — generated through a compositional simulation pipeline (asset specification, skill policies, task composition, and rendering as decoupled stages), not collected on real hardware, despite photorealistic preview clips. franka-1 and franka-2 are two different feature spaces in the source (differing image shape and gripper value range) — this export uses only franka-1; don't merge episodes across the two without checking meta/info.json on each. Because the source's 1,488 task directories span embodiments with genuinely incompatible schemas (single-arm Franka vs. dual-arm Genie-1/Lift-2/Split-Aloha), this export covers exactly one task/embodiment pair rather than a cross-embodiment sample.

Go deeper on these datasets. 5 humanoid and bimanual LeRobot datasets compares state and action spaces across five of these embodiments, and real vs. synthetic LeRobot datasets compares synthetic generation strategies and the published sim-to-real evidence. To bring your own LeRobot data into FiftyOne, see how to import LeRobot datasets.

Frequently Asked Questions

What's the difference between MCAP and LeRobot format?

MCAP stores continuous, multi-rate sensor logs (camera, LiDAR, IMU, all on independent clocks) losslessly, built for SLAM and sensor-fusion work where you need to seek to an arbitrary timestamp. LeRobot stores discrete, pre-synchronized episodes — state, action, and vision aligned to one frame grid — compressed for training imitation-learning and VLA policies. See our MCAP dataset roundup for the raw-sensor-log side of this.

Why are some of these FiftyOne datasets subsets instead of the full source?

Disk budget. 7 of the 11 sources range from several hundred GB to multiple TB — AgiBotWorld2026 alone is 13.6TB. For those, we imported a shard-contiguous subset (typically the smallest set of Parquet/MP4 files that produces a complete, non-truncated set of episodes). The 4 exceptions (robocasa-MG_100, robotwin_unified, tavis-head-gr1t2, tavis-head-reachy2) are complete because their full sources are small enough (under ~15GB each) to import entirely.

What does media_reference mean in a FiftyOne LeRobot dataset?

It's the field that points a FiftyOne sample to its episode's data without copying that data into the sample document itself—chunk/file indexes, frame ranges, and per-video timestamp ranges in the LeRobot data/, videos/, and meta/ files. FiftyOne's App resolves it live, rendering video and state/action streams in the Streams and State & Action tabs as you scrub through an episode, rather than loading everything into memory up front.

Why is task a string field instead of a classification label?

These 11 datasets don't store task identity as a fixed, closed set of labels. Values range from free-form natural language (droid_3d has 23,858 distinct task strings across its full source) to skill-primitive phase names (fmb_multi) and numeric placeholders (rh20t_cfg1's "task 1"-"task 6"). fo.Classification implies a taxonomy with confidence scores; a plain StringField matches the source data without inventing structure.

Can I use droid_3d's FiftyOne export for depth or point-cloud work?

No — this export is RGB-only at any episode count, because FiftyOne's LeRobot importer doesn't support the depth_video/pointcloud dtypes the full source uses for those modalities. For depth and point clouds, install the community lerobotdataset3d package and load ZibinDong/droid_3d directly with its LeRobotDatasetDepthPointcloud reader.

Which of these are humanoid vs. single-arm?

Humanoid: HIW-500-LeRobot (Unitree G1), tavis-head-gr1t2 (GR1T2), tavis-head-reachy2 (Reachy2). Single- or dual-arm: fmb_multi (Franka), rh20t_cfg1 (Flexiv), droid_3d, AgiBotWorld2026 (dual-arm), InternData-A1 (Franka, in this export), robocasa-MG_100 (Franka on a mobile base), robotwin_unified (ALOHA bimanual), trex_dataset (bimanual dexterous hands, not a full humanoid).

Which licenses permit commercial use?

MIT (droid_3d, trex_dataset) and Apache-2.0 (robocasa-MG_100, robotwin_unified) permit commercial use subject to their license terms. CC-BY-4.0 (fmb_multi, HIW-500-LeRobot, tavis-head-gr1t2, tavis-head-reachy2) also permits commercial use with attribution. CC-BY-NC-SA-4.0 (AgiBotWorld2026, InternData-A1) prohibits commercial use. rh20t_cfg1's 28-episode export mixes CC-BY-SA-4.0 and CC-BY-NC-4.0, so check the per-episode license table before using it commercially.

The Smoothest Episode in the Batch Was Also a Failure: Motion Smoothness vs. Task Success in RoboLab-EgoX Policy Rollouts

Harpreet Sahota — Wed, 28 Oct 2026 23:34:22 +0000

A policy rollout scored smoother than 119 of its 127 peers on every motion metric a quality scorer knows how to compute, and it still failed the task. Here’s how that episode was found, and what it says about scoring robot data in general.

If you work with robot data, you’ve probably done some version of quality scoring without calling it that.

You watch a recording, and your eye picks out the obvious problems: a shaky reach, a hand that overshoots and corrects itself, a pause that goes on a beat too long. Over the last couple of years, that intuition has started getting turned into actual software. A handful of metrics, with names like spectral arc length and log dimensionless jerk, read a robot’s motion data and reduce it to a number per recording, so you can sort a folder of a hundred episodes by how rough or smooth each one looks instead of sitting through all of them yourself.

A real-world example clearly illustrates the limitations of these metrics. After scoring 127 attempts at a single manipulation task, an episode ranking among the top ten smoothest in the entire dataset turned out to be a total task failure. Despite failing its objective—placing a cube into a bowl—the recording triggered zero warning flags and showed no statistical anomalies. Because smoothness and task completion represent two distinct aspects of a recording, looking solely at arm trajectory smoothness reveals nothing about whether the goal was achieved.

This post walks through how that episode turned up: what dataset it’s in, how an open-source FiftyOne plugin scored it, and what the result does and doesn’t prove about scoring robot data more broadly.

Key takeaways

Scoring all 127 episodes of RubiksCubeTask from Voxel51/RoboLab-EgoX with the demo_quality_scorer FiftyOne plugin shows that motion smoothness does not predict task success. The task split 77 successes to 50 labeled failures, the best-balanced task in a 4,000-episode, 28-task corpus.
One episode, RubiksCubeTask_cape_hill_sf0, instruction “Put the cube in the bowl,” labeled success: False, ranks 8th-smoothest of all 127 episodes on the plugin’s overall_score. It trips zero flag thresholds and doesn’t register as a statistical outlier either. Every check the scorer runs comes back clean.
Smoothness correlates with success across the batch as a whole: the median overall_score is 0.17 for successful episodes and 0.38 for failed ones, with lower values indicating smoother. This one episode is the exception, not proof the whole approach is broken. It’s rare enough that scoring stays useful and common enough that you can’t skip watching the episode yourself.
These episodes run for only 5.4 seconds at 15 frames per second, too short for the scorer’s automatic windowing, which is designed to target 100 samples per window. A manual override, down to a 2-second window, still produces a usable ranking, and the walkthrough below covers exactly where that workaround stops being trustworthy.
The single smoothest episode in the entire batch, ranked first of 127, is also a labeled failure. But unlike cape_hill_sf0, it does register on the scorer’s outlier tab, which makes it a weaker “this would have slipped through” story than an episode that clears every check, including outlier detection.

What motion smoothness metrics measure, and what they can't

A few years ago, robots that recorded their own sensor data mostly produced a pile of loosely related files: a video here, a CSV of joint angles there, maybe a separate GPS log if the robot moved outdoors.

Increasingly, that data gets recorded instead as a single file per attempt at a task, called an episode, which bundles every sensor stream together: cameras, LiDAR, joint encoders, the robot’s own estimate of its end-effector position, all ticking along at their own independent rates for as long as the task takes. The standard container for one of these files is MCAP, developed by Foxglove and now the default recording format for ROS 2. Because an MCAP file describes its own contents (what channels it has, what each one’s messages look like), any tool that understands the format can open a recording it’s never seen before without needing the original robot’s software.

FiftyOne is an open source tool for organizing, visualizing, and curating machine learning datasets, and as of version 1.19 it opens MCAP files natively.

FiftyOne multimodal viewer playing an MCAP file, with a LiDAR point cloud in a 3D panel and three synced camera feeds below.

Point a sample at a .mcap file, and FiftyOne decodes whatever cameras, point clouds, or robot-state channels it finds and lays them out in a synced, scrubbable viewer, just as it would show you images or video. That matters here because it’s what makes it practical to treat a folder of a hundred robot recordings as a dataset you can query, rather than a hundred files you’d otherwise have to open one at a time.

After converting episodes into a searchable dataset, the next challenge is deciding which ones merit a human's time to inspect. That’s the job of a quality scorer: software that reads each episode’s sensor data and produces a handful of numbers meant to flag which recordings look rough, broken, or otherwise unfit to train on. This post uses one such tool, an open-source FiftyOne plugin called demo_quality_scorer, and our analysis zeroes in on the specific metrics that practitioners typically rely on above all others: the measures of motion smoothness.

Every smoothness metric relies on a speed profile as its foundational baseline, tracking the instantaneous velocity of a specific component—whether an individual joint or the end-effector. From that baseline, several distinct mathematical formulas evaluate specific facets of motion cleanliness:

Spectral Arc Length (SPARC): Analyzes the frequency content of a speed profile and favors motion with low, concentrated energy, indicating a single deliberate movement.
Log Dimensionless Jerk (LDLJ): Integrates the square of jerk over a recording and normalizes it for fair comparison across different window lengths.
Jerk RMS: Provides a unit-based measurement of total jerk, making it effective for identifying localized rough spots.
Low/High-Frequency Power Ratio: Evaluates whether motion energy stems from deliberate, slow movement versus rapid vibration or continuous correction.

Shared Limitation: All four metrics evaluate the movement signal itself rather than the task's actual success or outcome.

That core limitation encapsulates the central message of this article. When a robot fluidly and purposefully transports a cube to an incorrect location or releases it a millimeter shy of its target, its resulting velocity profile appears indistinguishable from a flawless execution. While the trajectory remains pristine, the end result falls short.

This exact issue motivated the design of demo_quality_scorer: it serves as a triage tool rather than an automated filter. Scores dictate which recordings warrant human inspection first, rather than dictating automatic deletion. Comprehensive details on all plugin metrics—including outlier detection and sensor health checks—are available in two companion articles: Everyone Is Scoring Robot Data, Nobody Is Looking at It and Smooth Isn’t the Same as Correct.

Below, we demonstrate a concrete, verifiable example of this vulnerability in action, identified by evaluating the scorer against an independent dataset.

Step 1: Pick one task without downloading 21.8 GB

Browsing the Voxel51 organization on Hugging Face to the RoboLab-EgoX dataset card and its FiftyOne loading instructions.

The dataset used here, Voxel51/RoboLab-EgoX, contains 4,000 .mcap episodes across 28 manipulation tasks, totalling 21.8 GB. Finding one interesting episode doesn’t require all of it. Alongside the actual recordings, the Hugging Face repo includes a samples.json manifest with every episode’s metadata (which task it belongs to, whether it succeeded, and how long it ran), a couple of megabytes with no video or telemetry attached. Reading that file first tells you exactly which 127 files, out of 4,000, are worth downloading:

from huggingface_hub import hf_hub_download
import json

manifest_path = hf_hub_download(
    "Voxel51/RoboLab-EgoX", "samples.json", repo_type="dataset"
)
samples = json.load(open(manifest_path))["samples"]

by_task = {}
for s in samples:
    d = by_task.setdefault(s["task"], {"success": 0, "fail": 0})
    d["success" if s["success"] else "fail"] += 1

# Sort for the best success/fail balance, not just the most episodes
best = sorted(by_task.items(), key=lambda kv: -min(kv[1]["success"], kv[1]["fail"]))
print(best[0])  # ('RubiksCubeTask', {'success': 77, 'fail': 50})

Good to know. Why optimize for balance instead of raw episode count? HammersInLeftBinTask has more episodes than any other task, 227 of them, but only 11 are successes. A task that’s almost entirely failures doesn’t give you a meaningful “typical success” to compare a “typical failure” against. RubiksCubeTask’s 77/50 split does, which is why it’s the one used for the rest of this post.

With the task picked, downloading just its 127 files comes out to about 570 MB, a small fraction of the full dataset:

records = [s for s in samples if s["task"] == "RubiksCubeTask"]
for r in records:
    r["local_path"] = hf_hub_download(
        "Voxel51/RoboLab-EgoX", r["filepath"], repo_type="dataset", local_dir="./data"
    )

Step 2: Build the dataset and pick your motion signals

FiftyOne sample grid of RubiksCubeTask episodes, opening one episode's synced camera views and plotting its joint and action channels over time.

With the files on disk, building a FiftyOne dataset out of them is a matter of pointing each sample at its file and copying over whatever metadata is useful to query later:

import os
os.environ["VFF_MULTIMODAL"] = "1"  # set before fiftyone is imported, in every process

import fiftyone as fo

dataset = fo.Dataset("RoboLab-EgoX-RubiksCube", persistent=True)
dataset.add_samples([
    fo.Sample(
        filepath=r["local_path"],
        task=r["task"],
        take_name=r["take_name"],
        instruction=r["instruction"],
        success=r["success"],
        duration=r["duration"],
    )
    for r in records
])

print(dataset.media_type)  # multimodal

FiftyOne infers media_type from the file extension, the same way it would recognize a .jpg as an image, so pointing a sample at an .mcap file is the entire import step. No importer class, no schema mapping.

Before scoring anything, the scorer needs to know which of an episode’s channels actually represent the robot’s motion. This dataset ships a /joint-positions channel (13 joint angles) and an /end-effector-pose channel, both readings of where the robot’s own body actually was. It also ships an /actions channel, which is the policy’s commanded output rather than a sensor reading: what the policy told the robot to do, not what the robot’s state ended up being. Scoring /actions would measure the policy’s intent instead of its execution, and those two can diverge exactly in the cases worth catching, so /joint-positions and /end-effector-pose are the two signals used here.

Step 3: Score it, and work around a 5.4-second episode

The demo_quality_scorer compute episode quality form in FiftyOne, with motion smoothness metrics, position signal pickers, and manual window settings.

demo_quality_scorer computes SPARC, LDLJ, jerk RMS, and the PSD ratio for each time window, then summarizes each episode in two ways: the median across all its windows and the worst window. Automatic windowing is built to target 100 samples per window, with a 2-second floor, which works well for episodes running tens of seconds to several minutes. It doesn’t work here. At 15 frames per second for 5.4 seconds, a RubiksCubeTask episode has 81 samples total, fewer than automatic windowing would even ask for in a single window.

Install the plugin the same way you’d install any FiftyOne plugin:

import fiftyone.operators as foo

foo.execute_operator(
    "demo-quality-scorer/compute_episode_quality",
    dataset=dataset,
    params={
        "motion_enabled": True,
        "motion_cfg": {
            "position_signals": [
                '["/joint-positions","position"]',
                '["/end-effector-pose","pose_position"]',
            ],
            "auto_window": False,
            "win_s": 2.0,       # the plugin's own floor, and the shortest usable window
            "overlap": 0.5,
        },
        "health_enabled": True,
        "health_cfg": {
            "health_channels": ["/joint-positions", "/end-effector-pose", "/actions"]
        },
        "outliers_enabled": True,
    },
)

Keep it honest. A 2-second window on a 5.4-second episode, with 50% overlap between consecutive windows, works out to roughly four windows per episode. That’s enough for the plugin’s median-versus-worst-window summary to mean something, but nowhere close to the roughly 48 windows a longer, 87-second episode from a different corpus (ABC-130k, covered in the companion posts above) would produce. Trust the overall ranking here; don’t lean too hard on any single window’s flag the way you could with a longer recording. As a sanity check, this episode was first scored within a 30-episode slice of the same task, then again across the full 127 episodes. Its rank held steady at 8th-smoothest both times, but the underlying z-scores shifted between runs, because they’re fit fresh against whatever batch you score. That’s a property of the method, not a bug: a score always means “relative to this batch,” never “relative to some universal scale.”

Step 4: Read the ranking against the success label

FiftyOne Episode Quality panel showing motion smoothness histograms and a worst-first ranking table for 127 RubiksCubeTask episodes, then filtered by success label.

Once scoring finishes, each sample includes a quality field containing all computed metrics. Sorting by overall_score and printing the success label next to it is what actually surfaces the interesting cases:

rows = sorted(
    ({
        "take": s.take_name,
        "success": s.success,
        "overall": s.quality.overall_score,
        "n_flags": s.quality.n_flags,
        "outlier": s.quality.is_outlier,
    } for s in dataset),
    key=lambda r: r["overall"],
)

for r in rows[:10]:
    print(r)

Run that, and the ten smoothest episodes in the dataset print out worst-to-best by score, with success: False sitting a few rows down from the very top instead of clustered at the bottom where you’d expect it.

The RoboLab-EgoX episode that broke the story

FiftyOne multimodal viewer of a RubiksCubeTask episode with ego and wrist cameras showing a robot arm above a cube and bowl, alongside the episode's channel list." Confirm this is actually cape_hill_sf0 before naming it so, since the post says neither episode has been watched.

RubiksCubeTask_cape_hill_sf0, instruction “Put the cube in the bowl,” success: False, has an overall_score of -0.30, which makes it the 8th-smoothest episode out of 127. Its SPARC (-2.69) and LDLJ (-8.76) both sit near the clean end of the batch’s own distribution, and n_flags reads 0, meaning nothing crossed a warn threshold on the episode’s median. It doesn’t show up on the outlier tab either; is_outlier: False. Every mechanism the plugin runs, scored the same way described above, comes back and says this episode is fine.

For comparison, RubiksCubeTask_photo_studio_01_sf0 is the single smoothest episode in the entire batch, ranked first of 127 with an overall_score of -0.85, and it’s also labeled success: False. That’s the more dramatic number of the two, but it does register as a statistical outlier (is_outlier: True): an isolation-forest and nearest-neighbor check, fit across the whole batch, flags it as unusual, even though unusual doesn’t automatically mean bad. cape_hill_sf0 also clears that check, which is why it's the cleaner example of the two. Nothing about it, not its smoothness scores and not its statistical profile, would have made anyone look twice.

Episode	Success label	Rank (of 127)	overall_score	is_outlier
RubiksCubeTask_cape_hill_sf0	False	8th smoothest	-0.30	False
RubiksCubeTask_photo_studio_01_sf0	False	1st smoothest	-0.85	True

Two labeled failures in the RubiksCubeTask batch rank among the smoothest episodes by every motion metric, but only cape_hill_sf0 also clears outlier detection.

Keep it honest. RoboLab-EgoX episodes are simulated policy rollouts recorded on NVIDIA’s RoboLab benchmark, not human teleoperation, so it’s worth saying plainly that this isn’t real-world hardware. The success field is the benchmark’s own ground-truth task-completion label, decided by the simulator, not a model’s prediction layered on after the fact.

What this means for triage, not gates

Run this same scorer over a different corpus, ABC-130k, 40 teleoperated manipulation episodes spanning 40 different tasks, and its single lowest-quality episode by verdict turns out to be a sensor dropout: a channel that simply stopped reporting partway through the recording. That’s a real, common failure mode, and the scorer catches it well, because dropout is fundamentally a timestamp problem and sensor-health metrics read exactly that. But it’s a different failure mode from a wrong-but-smooth action, and finding a real example of the second kind took scoring a second corpus, one with genuine success and failure labels to check the ranking against, rather than assuming ABC-130k would eventually produce one.

That’s the actual argument for scoring episodes at all. Not that it replaces a person watching the recording, but that it tells that person which four episodes out of forty, or which eight out of a hundred and twenty-seven, are worth the ninety seconds it takes to look. cape_hill_sf0 would never have made that list on smoothness alone; smoothness is exactly what says it’s fine. It made the list only because someone thought to sort by score and then cross-reference against a label the scorer itself never sees.

Try it yourself

Voxel51/RoboLab-EgoX on Hugging Face
demo_quality_scorer on GitHub
Everyone Is Scoring Robot Data, Nobody Is Looking at It, the full quality-metric stack, including sensor health and outliers
Smooth Isn’t the Same as Correct, SPARC, LDLJ, jerk RMS, and the PSD ratio worked through in detail
Open MCAP Robotics Logs in FiftyOne, No ROS Needed, what the multimodal viewer actually does with a file like this

Frequently Asked Questions

What is an “episode” in robot data, exactly?

One continuous recording of a robot attempting a task, start to finish, with every sensor it carries (cameras, joint encoders, whatever else is onboard) logged together on a shared timeline rather than split into separate files. It’s the natural unit for this kind of data because the sensors are all ticking at their own independent rates for the same stretch of time, and splitting that into individual frames or per-sensor files throws away the relationships between streams that actually matter.

Why not just widen the window instead of shrinking it to 2 seconds?

Because 2 seconds is already the plugin’s floor, not an arbitrary choice made for this post. AUTO_MIN_WINDOW_S in demo_quality_scorer’s own windowing code is set to 2.0 regardless of what automatic windowing would otherwise compute for a given signal. There’s no smaller manual value that stays meaningful for a duration-sensitive metric like LDLJ, whose value scales roughly with window length to the fourth power, which means comparing windows of different lengths to each other isn’t valid in the first place.

Is RoboLab-EgoX real robot data?

No, it’s simulated. The episodes are policy rollouts recorded on NVIDIA’s RoboLab manipulation benchmark and converted to MCAP by Voxel51, not real-world teleoperated demonstrations collected on physical hardware. The finding here, that smoothness and task success are different properties a scorer can’t conflate, generalizes well beyond this one dataset. The specific numbers and the specific episode do not represent physical hardware, and that distinction is worth repeating whenever this example comes up.

Does a low overall_score mean an episode is good?

Only relative to whatever batch it was scored against. overall_score is a weighted mean of z-scores, fit fresh every time the scorer runs, oriented so that a higher number always means worse. That makes it useful for ranking episodes within a single run, but it’s not comparable across different scoring runs, different signal choices, or different datasets. A score of -0.30 in this batch says nothing about what -0.30 would mean in a different one.

What does n_flags: 0 actually rule out?

Only that no individual metric’s episode-level median crossed the warn threshold (a z-score of 2 or higher) against this batch’s own distribution. It doesn’t rule out one rough window inside an otherwise clean episode, which demo_quality_scorer tracks separately as a flagged interval, and it says nothing at all about whether the task was actually completed, which is not something any motion metric here is designed to measure.

Why did the smoothest RoboLab-EgoX episode fail the task?

Because smoothness metrics measure how the arm moved, not whether the cube landed in the bowl, so a fluid trajectory to the wrong place scores as clean.