DEV Community

rhea hollis
rhea hollis

Posted on

Provenance Labels for Video Datasets: Write Them at Collection Time

#tutorial #python #machinelearning #datascience

Every video dataset I've inherited has the same hole: the clips are fine, the metadata about where they came from doesn't exist. This post is the schema and the write path I now put in on day one, so the question "can we still use this clip?" has an answer two years later.

The four fields that matter 

@dataclass
class Provenance:
    source_url: str      # exact page, not the domain
    license_at_capture: str
    captured_at: datetime  # UTC, always
    session_geo: dict      # {"country": ..., "city": ..., "asn": ...}
Enter fullscreen mode Exit fullscreen mode

Anything beyond these four is optional. Anything missing from these four makes the row unanswerable later.

Write it in the same transaction as the clip

The mistake is a two-phase design: download first, enrich later. Enrichment never happens. The provenance block has to be written in the same code path that saves the clip:

def save_clip(video, session):
    prov = Provenance(
        source_url=video.page_url,
        license_at_capture=video.license_text,
        captured_at=datetime.now(timezone.utc),
        session_geo=session.geo,   # from the proxy session itself
    )
    store.write(video.id, video.blob, provenance=prov)
Enter fullscreen mode Exit fullscreen mode

session.geo is the interesting part. If you collect through a residential proxy network, the session's exit country, city, and ASN are known at request time — providers like Thordata expose session targeting directly, so the geo column needs no separate IP-lookup step (their free trial is here if you want to try the pattern: https://www.thordata.com/?ls=dev&lk=dev-1). Logging it costs one dictionary write and makes the row auditable.

Reject rows without provenance

Schema enforcement is what separates a label from a hope:

store.add_constraint("provenance", required=True, on_missing="reject")
Enter fullscreen mode Exit fullscreen mode

A rejected write at collection time costs one clip. A missing field discovered at audit time costs the re-collection of the entire dataset, because backfilling a source URL you never saved means re-downloading the clip and re-checking the license — you pay for the same row twice.

The dedup dividend

One side effect: source_url plus a content hash gives you cheap dedup across collection runs. Most "duplicate" rows I've cleaned up were the same clip collected twice through different exits — the label catches it before you store the second copy.

Provenance is the cheapest field you'll ever add and the most expensive one to backfill. Write it at collection time.

Top comments (0)