Most “video dataset” posts show you a folder of .mp4 files. That's the wrong abstraction. If you're training a VLM or building a retrieval system, the video file is one field in a record — and the quality of everything around it decides whether your pipeline works. Here's how we structure records in the video datasets we just launched at Thordata, and the engineering choices behind it.
The record, not the file. Each video maps to a structured entry with four layers:
{
"video": {
"title": "...",
"description": "...",
"publish_time": "...",
"category_labels": ["..."],
"engagement": { "views": 0, "likes": 0 }
},
"creator": {
"channel_id": "UC...",
"handle": "@...",
"channel_url": "..."
},
"multimodal": {
"captions": "...",
"transcript": "...",
"audio_scene_info": "..."
},
"provenance": {
"platform": "youtube",
"video_url": "https://...",
"platform_video_id": "..."
}
}
Provenance is a first-class layer, not an afterthought — retroactive provenance means re-collecting the dataset. Field availability varies by platform and dataset scope, so treat the schema as a contract you inspect per dataset, not a universal constant.
Formats chosen per data type. Metadata and transcripts ship as JSON; tabular analysis works better in CSV or Parquet, and Parquet in particular pays off once you're filtering millions of rows by category or language. Video files deliver as .mkv or .mp4, audio as .m4a or .mp3. Don't force one format onto everything — pick per data type, and export all three metadata formats from the same source of truth.
Delivery is engineering, not logistics. Datasets land where your stack already reads from: Amazon S3, Azure Blob, GCS, SFTP, or Webhook, on-demand or on a custom schedule. On our side that means 8–10 PB of daily delivery capacity; on your side it means zero new ingestion code. If your workflow is API-shaped instead of batch-shaped, on-demand retrieval with field-level filtering hits the same records.
Sizing the corpus. 17B+ indexed video URLs across 700M+ channels in 100+ languages is the ceiling, not the unit of purchase. The practical move: pull a sample, inspect the schema and field coverage for your platforms, define filters (content type, upload date, view metrics, quality), validate with a test batch, then scale to full delivery. Custom collection covers the gaps the predefined datasets don't.
Quality notes worth knowing. Videos deliver up to 2K resolution, audio at the best available source quality, and every dataset we ship is consent-approved content cleared for AI training — check that box before you check any other.
The mental model that ties it together: treat a video dataset like a database with media attachments, not a media folder with a spreadsheet. Your training loop, your dedup jobs, and your future self will all benefit.
We just launched these datasets at Thordata — sample access and docs:https://www.thordata.com/products/multi-platform-video-datasets?op=rhea&from=x
Top comments (0)