DEV Community: rhea hollis

Localized Video Data Collection: A Practical Pipeline from Search to Structured Records

rhea hollis — Mon, 21 Sep 2026 08:05:37 +0000

Localized Video Data Collection: A Practical Pipeline from Search to Structured Records

Collecting video data across regions is not just a download problem. Search coverage, language, location, captions, duplicates, and rights all affect whether the final records are useful. A staged pipeline makes those decisions visible.

Stage 1: discover a representative seed set

Start with a narrow set of public search results, channels, or category pages. Save the original query, translated query when used, locale, timestamp, and source URL. This lets you distinguish a discovery gap from a retrieval failure.

Do not optimize for the largest possible URL list on day one. A smaller sample that covers the languages and content types you care about is more useful for testing the schema and quality checks.

Stage 2: filter with explicit rules

Turn “relevant video” into rules that can be rerun:

duration range;
language or language mix;
publication window;
creator or channel exclusions;
duplicate threshold;
review status.

Store the rule version on each batch. If “recent” changes from 90 days to 30 days, you should be able to reproduce both views.

Stage 3: retrieve with bounded retries

Network failures are expected. Classify them, retry only transient errors, and keep an idempotent queue keyed by platform and source ID. This prevents a retry from silently creating a second record.

If the target page is localized, test a small set of required countries or cities separately. Residential routing can help a collector observe public, region-specific pages, but it does not replace permission, rate limiting, or a valid client implementation. Keep proxy credentials in environment variables and outside the dataset.

Stage 4: normalize without losing the evidence

Store raw response metadata beside the normalized record. Convert timestamps to UTC while retaining the original timezone when available. Keep the original title and description before removing markup or normalizing punctuation.

For multilingual video, preserve the original language label, your detected language, and the confidence or review status as separate fields. This is more useful than pretending every item has one certain language.

Stage 5: validate the batch

For every batch, report:

duplicate and near-duplicate rate;
missing required fields;
caption or transcript alignment;
language distribution;
download and decode failures;
items needing manual review.

A dashboard that shows only record count hides the expensive problems.

When a ready-made dataset is sensible

Build your own collector when the source mix or filtering logic is central to the research. Consider a ready-made or custom dataset when the team needs structured metadata, captions, transcripts, engagement fields, and a predictable delivery process more than it needs to own every retrieval step.

Thordata describes multi-platform video data with source and platform fields, content signals, ready-to-use datasets, custom collection, API access, and JSON/CSV/Parquet delivery. Ask for a representative sample and confirm the fields, media availability, refresh schedule, and rights documentation before integrating it: https://www.thordata.com/products/multi-platform-video-datasets?op=rhea&from=x

A practical handoff checklist

Before giving records to a model or analyst, deliver the schema, manifest, checksums, collection window, rule version, exclusion list, and a short quality report. The handoff should make clear what is covered, what is missing, and what must not be inferred.

How to Build a Video Dataset for Multimodal AI: Schema and Provenance First

rhea hollis — Mon, 21 Sep 2026 08:05:13 +0000

How to Build a Video Dataset for Multimodal AI: Schema and Provenance First

Many video dataset projects start with a large download queue. A better starting point is a small schema and a clear answer to one question: can another person explain where each record came from and what happened to it?

Design the record before collecting files

Separate source facts from derived labels. A practical record can include:

{
  "source_url": "...",
  "platform": "...",
  "platform_id": "...",
  "creator_id": "...",
  "published_at": "...",
  "language": "...",
  "duration_seconds": 0,
  "caption": "...",
  "transcript": "...",
  "rights_status": "review_required",
  "collected_at": "...",
  "schema_version": "1.0"
}

Keep the raw source fields. If a title is later cleaned, translated, or classified, store the derived value in another field instead of overwriting the original.

Provenance is part of model quality

For each item, record the source URL or platform ID, collection time, transformation version, and exclusion reason when an item is removed. This helps a team investigate a surprising training example, refresh a deleted source, or explain why two exports differ.

A URL is not the same thing as redistribution permission. Rights review, platform terms, creator consent where required, and regional privacy rules belong in the dataset plan. Keep a review state such as pending, approved, restricted, or removed, with a reason code.

Five checks to automate early

Duplicate detection: use platform IDs first, then perceptual hashes or short audio fingerprints for near-duplicates.

Temporal consistency: check that duration, caption timestamps, transcript length, and scene boundaries agree.

Language quality: store detected language separately from an uploader-provided label. Mixed-language content should not be forced into a single class.

Creator leakage: split by creator or channel when the same style or footage could otherwise appear in both training and evaluation.

Decode and field completeness: report missing media, unreadable files, absent captions, and missing metadata as separate failure categories.

Choose delivery around the next user of the data

JSON is convenient for nested records, CSV for flat analysis, and Parquet for columnar processing. Whichever format you use, ship a manifest with checksums, schema version, collection window, exclusions, and known gaps.

Thordata’s multi-platform video dataset page describes structured metadata, captions, transcripts, engagement signals, ready-to-use datasets, custom collection, API access, and JSON/CSV/Parquet delivery. Those options are useful when a team wants to spend more time on modeling and less time maintaining every collector. Request a sample schema first and compare it with your required fields, languages, refresh cadence, and rights process: https://www.thordata.com/products/multi-platform-video-datasets?op=rhea&from=x

A small baseline beats a huge guess

Before scaling, train or evaluate a small baseline on a fixed sample. Track required-field coverage, duplicate rate, transcript alignment, language distribution, and manual-review volume. If adding more records does not improve the baseline, improve sampling or labels before buying more storage.

Rotating vs Sticky Residential Proxies: How to Choose for Web Scraping

rhea hollis — Mon, 21 Sep 2026 08:04:13 +0000

Rotating vs Sticky Residential Proxies: How to Choose for Web Scraping

Choosing a residential proxy is often framed as a question of IP count. In practice, the more useful question is: does this workflow need a new IP for each request, or continuity across several requests? That distinction affects parsing, login-free sessions, retries, and debugging.

When rotating residential proxies make sense

Rotating sessions are a good fit for independent requests: product pages, public listings, search-result samples, and broad market checks. The collector can move from one URL to the next without depending on a cookie or a previous response.

Rotation is not a cure for aggressive crawling. Keep concurrency bounded, use a clear user agent, respect site rules, and retry only transient failures. Measure valid records rather than counting HTTP 200 responses.

When a sticky session is the better tool

Some workflows need a little continuity. Examples include:

walking through pagination where a short-lived cookie is set;
checking several pages in one browser-like flow;
reproducing a localized page while debugging a parser;
running a short sequence of requests that should share the same network identity.

For these cases, a sticky session with an explicit time limit is easier to reason about than changing the IP after every request. It should still expire. A session that lasts longer than the job needs can make failures harder to isolate and may waste traffic.

A small benchmark you can repeat

Use the same URL list, parser version, and request schedule for both modes. Record:

required-field completion;
empty or blocked responses;
median and p95 latency;
retry count;
cost per valid record.

If a page is location-sensitive, repeat the test for the exact country or city required by the project. Do not assume that a generic endpoint represents every local result.

Example request configuration

Keep credentials outside source code and dataset records. A simple Python test can look like this:

import os
import requests

proxy_url = os.environ["PROXY_URL"]
response = requests.get(
    "https://example.com/public-page",
    proxies={"http": proxy_url, "https": proxy_url},
    timeout=20,
    headers={"User-Agent": "ResearchBot/1.0 (+contact@example.com)"},
)
print(response.status_code, len(response.content))

The code is deliberately small. The important part is the surrounding log: target URL, region, session mode, response status, parser result, and timestamp.

Where Thordata fits

Thordata’s residential-proxy documentation describes country, city, and other location targeting, along with rotating and sticky session options over HTTP/HTTPS. That makes it a candidate for a controlled comparison, not a reason to skip your own benchmark. A practical starting point is the residential proxy page: https://www.thordata.com/?ls=dde&lk=dde

Responsible collection

Collect only public data you have a legitimate reason to use. Follow the target site’s terms, applicable rate limits, and copyright requirements. A slower, observable collector is easier to maintain than a fast one that cannot explain where its requests came from.

From Public Video URLs to a Maintainable Data Pipeline

rhea hollis — Sun, 20 Sep 2026 14:21:00 +0000

Collecting video data is not one request followed by one download. It is a sequence of discovery, filtering, retrieval, normalization, quality control, and refresh. Treating it as a pipeline makes failures visible and keeps a useful dataset from becoming a one-off archive.

1. Discovery: build a narrow seed set

Start with search results, known channels, or public category pages. Save the query, locale, timestamp, and source URL for every seed. Do not begin with maximum volume; begin with a set that represents the task you want to measure.

For multilingual work, keep the original query and a translated query as separate fields. This makes it possible to tell whether a gap came from search coverage or from later filtering.

2. Filtering: make policy executable

Turn vague requirements into rules: duration range, language, date window, creator exclusions, duplicate threshold, and review status. Store the rule version on every run. When a stakeholder changes “recent” from 90 days to 30 days, you should be able to reproduce both outputs.

3. Retrieval: design for partial failure

Network requests fail. A good collector records the failure class, retries only transient errors, and moves on. Keep a queue with an idempotent key such as (platform, platform_id, variant). That prevents a retry from creating a second copy.

When location matters, test a small set of regions separately. A residential proxy can help a collector observe public, localized pages, but it does not replace permission, rate limiting, or a valid client implementation. Keep proxy credentials in environment variables and outside dataset records.

4. Normalization: preserve the raw evidence

Store the raw response metadata and a normalized record side by side. Normalize timestamps to UTC while retaining the original timezone when it is available. Keep the original title and description before cleaning punctuation or HTML. These small choices make later audits possible.

5. Quality control: sample every batch

For each batch, inspect a fixed sample and calculate:

duplicate and near-duplicate rates;
missing required fields;
caption or transcript alignment;
language distribution;
download and decode failures.

Publish the report with the batch identifier. A dashboard that shows only item count hides the most expensive problems.

6. Delivery: match the consumer

JSON is convenient for nested metadata, CSV works well for flat analysis, and Parquet is usually a better fit for columnar processing. Deliver a manifest with checksums, schema version, collection window, exclusions, and known gaps. The manifest is as important as the media files.

Thordata presents video data from public platforms with structured metadata, captions, transcripts, and engagement signals, and describes delivery through files, APIs, and cloud or integration endpoints. That can be useful when a team wants to spend its time on modeling rather than maintaining every collector. Still, evaluate a sample against your own schema, languages, and rights process.

A refresh policy keeps the dataset honest

Define what changes trigger a refresh: new uploads, edited captions, deleted sources, or a changed moderation decision. Keep immutable batch snapshots and a current index. This gives model builders a stable experiment set while allowing the production view to evolve.

Video Datasets That Survive Contact With a Real Model

rhea hollis — Sun, 20 Sep 2026 11:21:00 +0000

A video dataset can look impressive in a spreadsheet and still fail in training. The usual causes are ordinary: duplicate clips, missing timestamps, unclear source history, captions that do not match the audio, and a split that leaks the same creator into both training and evaluation.

Define the record before collecting media

Start with a schema that separates source facts from derived labels. A useful minimum record might contain:

{
  "source_url": "...",
  "platform": "...",
  "platform_id": "...",
  "creator_id": "...",
  "published_at": "...",
  "language": "...",
  "duration_seconds": 0,
  "caption": "...",
  "transcript": "...",
  "rights_status": "review_required",
  "collected_at": "...",
  "schema_version": "1.0"
}

Keep collected_at and schema_version. They make a refresh auditable and prevent a silent change in meaning when a field is renamed later.

Five quality checks worth automating

Identity. Deduplicate on stable platform IDs where available, then use perceptual hashes or short audio fingerprints for near-duplicates.

Time. Check that duration, timestamps, captions, and scene boundaries are internally consistent. A transcript that extends beyond the video is a useful failure signal.

Language. Store the detected language and confidence separately from the uploader’s language label. Mixed-language videos should not be forced into one bucket.

Content safety. Keep a review status and a reason code. A single yes/no moderation flag is hard to audit and easy to misuse.

Splits. Split by creator, channel, or source group when the task could otherwise leak style or repeated footage across train and test sets.

Provenance is a model feature

For every record, preserve where it came from, when it was observed, which transformation produced the caption or transcript, and what was removed. This lets a team investigate a surprising model output without rebuilding the entire collection job.

It also clarifies what a dataset does not grant. A source URL is not automatically a licence to redistribute a video. Rights review, platform terms, consent requirements, and regional rules belong in the project plan.

Choosing delivery: files, API, or a custom job

Ready-to-use files are convenient when the schema and coverage already match your task. API access is useful when you need on-demand filtering or regular refreshes. A custom collection is justified when the required fields, languages, or source mix are specific enough that a generic package would create too much cleanup.

Thordata’s video-data pages describe structured metadata, captions, transcripts, engagement fields, and exports such as JSON, CSV, and Parquet, with ready-to-use, API, and custom collection options. Ask for a sample schema and a small evaluation set first. Check field availability, refresh timing, and rights documentation before committing to a larger delivery.

Measure usefulness, not only volume

Report the percentage of records that pass every required-field check, the duplicate rate, transcript alignment, language mix, and the number of items needing manual review. Then train a small baseline model. If quality does not improve when the dataset grows, more records are not the answer; better sampling or labels are.

A Practical Residential Proxy Playbook for Reliable Web Data Collection

rhea hollis — Sun, 20 Sep 2026 08:22:21 +0000

Residential proxies are useful when a website serves different content by location or treats data-centre traffic differently. They are not a magic bypass, though. A stable collection job still depends on request pacing, clear scope, retries, and good data validation.

Start with the target, not the proxy

Before choosing a provider, write down four things:

Which public pages are in scope?
Which countries, cities, or languages matter?
How often will the same page be requested?
What should happen when a request is blocked or returns an empty page?

This short brief prevents a common mistake: buying a large pool when the real problem is an overly aggressive crawler or an incorrect locale setting.

Rotating or sticky sessions?

Use a rotating session when requests are independent, such as collecting product pages across many public URLs. A fresh IP can reduce concentration on one address, but rotation alone will not fix a broken parser or an invalid request pattern.

Use a sticky session when a workflow needs continuity. Login-free flows such as pagination, a multi-step form, or a page that sets a short-lived cookie are easier to debug when the same session is kept for a limited period. Set a clear expiry instead of keeping a session forever.

A small, observable request loop

Keep the first test small. Log the URL, timestamp, chosen region, status code, response time, and parser result. Never log proxy passwords or full authorization URLs.

import os
import time
import requests

proxy = os.environ["PROXY_URL"]
proxies = {"http": proxy, "https": proxy}

for url in seed_urls:
    response = requests.get(
        url,
        proxies=proxies,
        timeout=20,
        headers={"User-Agent": "ResearchBot/1.0 (+contact@example.com)"},
    )
    record_result(url, response.status_code, len(response.content))
    time.sleep(1.5)

The important part is not the code itself. It is the feedback loop: measure success by the fields you actually need, not only by HTTP 200 responses.

What to check in a provider

Geo controls: Can you target the country or city required by the project?
Session controls: Are rotation and sticky behaviour explicit and testable?
Protocol and tooling: Does the service document HTTP/HTTPS integration for your client?
Usage visibility: Can you see traffic, errors, and spend while testing?
Acceptable-use and sourcing: Is there a clear policy for lawful public-data collection?

Thordata’s residential-proxy documentation describes HTTP/HTTPS access, rotating and sticky sessions, and location targeting. Its product pages also present residential IP coverage across many regions. Treat those as capabilities to verify in your own trial, not as a substitute for a benchmark on your target pages.

A simple benchmark that travels well

Run the same 50–100 public URLs through your existing route and a candidate proxy route. Compare:

required-field completion;
median and p95 response time;
block, timeout, and empty-page rates;
location accuracy;
cost per valid record.

Keep the URL list and parser version fixed. If the result is inconclusive, change one variable at a time: region, session mode, concurrency, or retry policy.

Responsible collection is part of reliability

Respect the site’s terms, robots guidance where applicable, rate limits, and copyright rules. Collect only public data that you have a legitimate reason to use, and provide a contact route in your user agent when appropriate. A slower, explainable pipeline is usually easier to keep running than a fast one that creates complaints.

Building Reliable Web Scraping Pipelines in 2026: Why Proxy Infrastructure Matters

rhea hollis — Fri, 11 Sep 2026 09:13:33 +0000

Modern web scraping is no longer just about sending HTTP requests and parsing HTML.

As websites become more dynamic and anti-bot systems become more advanced, developers need to think beyond code. Browser fingerprints, IP reputation, request patterns, and infrastructure quality all affect whether a scraping workflow succeeds.

A crawler that works perfectly in development may fail completely when running at scale.

The difference is often not the scraper itself — it is the infrastructure behind it.

The Real Challenges Behind Large-Scale Web Scraping

Many scraping failures come from three common problems:

1. Unstable IP Sources

Using shared or low-quality IPs often leads to:

High block rates
Frequent CAPTCHA challenges
Unstable sessions
Poor data collection efficiency

For production scraping, IP quality directly impacts success rate.

2. Scaling Requests Without Losing Stability

Small scripts can work with a single connection.

However, large-scale projects require:

Multiple locations
Different browsing patterns
Session management
Reliable connection performance

A good proxy system should support different workloads instead of forcing every project into the same setup.

3. Collecting Data From Different Regions

Many real-world projects require location-specific data:

E-commerce price monitoring
Search engine research
Ad verification
Market intelligence

A crawler needs access from different countries, cities, or networks to collect accurate regional data.

Choosing the Right Proxy Infrastructure

For serious scraping projects, residential proxies are commonly used because they represent real consumer network connections.

A reliable residential proxy solution should provide:

Large IP availability
Geographic targeting
Rotating sessions
Stable long-term connections

This allows developers to build more resilient data pipelines.

Where Thordata Fits

Thordata provides residential and mobile proxy infrastructure designed for:

Web scraping
AI data collection
Browser automation
Market research
Business intelligence

Key features:

100M+ real residential IPs
Coverage across 195+ countries
Rotating and sticky sessions
Flexible geo-targeting

Developers can test workflows before scaling with a 3-day free trial.

Final Thoughts

Successful scraping is not only about writing better crawlers.

The modern data stack requires:

Better extraction logic
+
Reliable browser automation
+
High-quality proxy infrastructure

Building the right foundation early saves significant maintenance time later.

Try Thordata free for 3 days and evaluate your workflow.

Successful scraping is not only about writing better crawlers.

The modern data stack requires:

Better extraction logic
+
Reliable browser automation
+
High-quality proxy infrastructure

Building the right foundation early saves significant maintenance time later.

Try Thordata free for 3 days and evaluate your workflow.

Action Recognition Field Notes: Boundary Relabeling, Near-Miss Pairs, and Splits That Don't Lie

rhea hollis — Fri, 04 Sep 2026 09:13:46 +0000

An action recognition run that stalls at "decent" usually isn't starved of architecture. It's starved of honest labels. This post is the code for the three fixes that moved accuracy more than any tuning: boundary relabeling, near-miss sampling, and source-aware splits.

1. Relabel to temporal boundaries

Fixed 16-frame windows make "opening a door" and "closing a door" nearly the same clip. Trim to the detected action, keep context padding on both sides:

def trim_to_action(clip, onset, offset, fps, pad=0.5):
    lo = max(0, int(onset * fps - pad * fps))
    hi = min(len(clip), int(offset * fps + pad * fps))
    return clip[lo:hi]   # boundaries carry the signal, not the window

The relabel of ~200 near-miss clips outperformed a week of architecture experiments.

2. Oversample the near-miss pairs

Aggregate class balance hides the pairs that matter (sit down vs. bend-to-pick-up). Mine the confusion matrix and oversample against nearest neighbors:

def pair_weights(cm, labels, boost=3.0):
    pairs = {}
    for i, j in zip(*cm.topk_offdiagonal(k=5)):   # top confusable pairs
        pairs[(labels[i], labels[j])] = boost
    return pairs   # feed a WeightedRandomSampler, not a blanket class weight

Blanket class weights lift everything evenly; pair boosts lift exactly what's broken.

3. Split by scene and source, never randomly

Random splits leak the same actor and room into train and test. The split needs provenance:

def honest_split(rows, by=("scene_id", "source_id"), ratios=(0.7, 0.15, 0.15)):
    groups = defaultdict(list)
    for r in rows:
        groups[tuple(r[k] for k in by)].append(r)  # provenance fields drive the split
    return random_group_split(groups, ratios)

This only works when every row knows where it came from — which is why lineage labels belong in the schema. Collections that ship source metadata per clip (Thordata's video datasets, for one — free trial here: https://www.thordata.com/?ls=dev&lk=dev-1) make the honest split a one-liner instead of a reconstruction project.

The pattern across all three fixes: the data work is the model work. Boundaries in the labels, pairs in the sampler, provenance in the split.

An Honest Price Monitor: Session Config, ASN Pinning, and Decoy Detection

rhea hollis — Thu, 03 Sep 2026 07:59:05 +0000

A price monitor that fails loudly is fine — you fix it. The dangerous one returns 200s and plausible numbers while the dataset rots. This post is the code for the three fixes that matter: session config, ASN pinning, and decoy detection.

1. Size the session to the crawl

The classic mistake is a default timeout that expires mid-catalog. The exit rotates, half the store re-downloads, and the two halves of the catalog are seen from different locations — the prices stop being comparable.

SESSION_MINUTES = max(category_page_count * avg_seconds_per_page / 60 * 1.5, 30)

session = provider.create_session(
    sticky_minutes=SESSION_MINUTES,   # > longest category, plus margin
    country=store.country,
    asn=store.target_asn,             # pin the exit to one network
)

Sticky sessions of 30–90 minutes with country and ASN targeting are standard on residential providers — Thordata exposes all three at session creation (free trial here: https://www.thordata.com/?ls=dev&lk=dev-1) — so this is configuration, not engineering.

2. Log the exit country as a first-class column

Mixed-location data reads as spread but is mostly geo-pricing noise. The fix costs one column:

@dataclass
class PriceRow:
    sku: str
    price: float
    currency: str
    exit_country: str   # from the session, at capture time
    exit_asn: str
    captured_at: datetime

Comparisons then happen GROUP BY exit_country — anything else compares apples to a different country's apples.

3. Variance-check before storage

Decoy pages arrive with a 200 status and plausible digits. Structural parsing won't catch them; statistics will:

def is_plausible(row: PriceRow, history: pd.Series, threshold: float = 0.30) -> bool:
    median = history.tail(7 * 24).median()  # 7-day rolling median
    if median <= 0:
        return True  # not enough history yet
    return abs(row.price - median) / median <= threshold

# plausible -> store; implausible -> flag for review, never store silently

A 30% threshold against a 7-day median catches currency swaps and decoy prices while letting genuine flash sales through to review.

The full loop

for store in stores:
    with provider.session(country=store.country, asn=store.target_asn,
                          sticky_minutes=SESSION_MINUTES) as s:
        for page in store.catalog:
            row = parse_price(s.get(page.url))
            if is_plausible(row, history[row.sku]):
                store_row(row)
            else:
                flag_for_review(row)

Segment the schedule too: hourly snapshots for flash-deal SKUs, daily for the stable catalog — hourly-everything burns ~20x the bandwidth for marginal signal.

The monitor that assumes its data is lying until proven otherwise is the one that survives.

Provenance Labels for Video Datasets: Write Them at Collection Time

rhea hollis — Wed, 02 Sep 2026 02:19:18 +0000

Every video dataset I've inherited has the same hole: the clips are fine, the metadata about where they came from doesn't exist. This post is the schema and the write path I now put in on day one, so the question "can we still use this clip?" has an answer two years later.

The four fields that matter

@dataclass
class Provenance:
    source_url: str      # exact page, not the domain
    license_at_capture: str
    captured_at: datetime  # UTC, always
    session_geo: dict      # {"country": ..., "city": ..., "asn": ...}

Anything beyond these four is optional. Anything missing from these four makes the row unanswerable later.

Write it in the same transaction as the clip

The mistake is a two-phase design: download first, enrich later. Enrichment never happens. The provenance block has to be written in the same code path that saves the clip:

def save_clip(video, session):
    prov = Provenance(
        source_url=video.page_url,
        license_at_capture=video.license_text,
        captured_at=datetime.now(timezone.utc),
        session_geo=session.geo,   # from the proxy session itself
    )
    store.write(video.id, video.blob, provenance=prov)

session.geo is the interesting part. If you collect through a residential proxy network, the session's exit country, city, and ASN are known at request time — providers like Thordata expose session targeting directly, so the geo column needs no separate IP-lookup step (their free trial is here if you want to try the pattern: https://www.thordata.com/?ls=dev&lk=dev-1). Logging it costs one dictionary write and makes the row auditable.

Reject rows without provenance

Schema enforcement is what separates a label from a hope:

store.add_constraint("provenance", required=True, on_missing="reject")

A rejected write at collection time costs one clip. A missing field discovered at audit time costs the re-collection of the entire dataset, because backfilling a source URL you never saved means re-downloading the clip and re-checking the license — you pay for the same row twice.

The dedup dividend

One side effect: source_url plus a content hash gives you cheap dedup across collection runs. Most "duplicate" rows I've cleaned up were the same clip collected twice through different exits — the label catches it before you store the second copy.

Provenance is the cheapest field you'll ever add and the most expensive one to backfill. Write it at collection time.

Video Scraping Is a Bandwidth Problem: Four Rules for Proxying Large-File Collection

rhea hollis — Tue, 01 Sep 2026 08:38:24 +0000

Most proxy advice is written for text: rotate per request, retry on failure, move on. The first time you point that stack at video, it falls apart in new and expensive ways. Four rules that fixed it for me.

Rule 1: One sticky session per file

An IP switch mid-download means a restart or a corrupted file — you pay for the same bytes twice. Bind the session ID to the file, not the job:

# pseudo-Python
for video in queue:
    sid = new_session_id()          # sticky, up to 90 min
    for attempt in range(3):
        if download(video.url, session=sid):
            break                    # completed on ONE ip
    release_session(sid)             # fresh ip for the NEXT file

Ninety minutes of stickiness covers most clips end-to-end; anything longer belongs in a resumable-download design, not a longer session.

Rule 2: Rotate between files, never during one

Address diversity still matters — you're just moving the rotation boundary. Fresh IP per file gives you the same spread as per-request rotation would, without ever breaking a transfer in progress.

Rule 3: Parallelize by file, and check your concurrency cap

Video throughput comes from per-file parallelism: one lane per file, each on its own sticky session. The bottleneck is almost never your code — it's providers that cap concurrent sessions. Pools without such caps exist — Thordata's residential network, for one (unlimited concurrency, 100M+ IPs; trial traffic here: https://www.thordata.com/?ls=dev&lk=dev-1) — but the pattern is provider-agnostic: verify the cap, then size your worker pool to it.

Rule 4: Budget in cost-per-file, not cost-per-month

At $0.65/GB, a 50MB clip costs ~3.3 cents and a 20,000-clip run costs ~$650 in bandwidth. Multiply average file size by count by price before every run — and remember that every restarted file is bandwidth you pay for twice. Cheap-but-flaky pools are the expensive ones.

The mental shift that ties it together: with text, the proxy is a per-request anonymizer. With video, it's a transfer layer — and transfer layers are engineered around sessions, lanes, and bandwidth, not around hiding.

If you're evaluating providers for large-file collection, these four rules are the checklist — and a trial with a session-control provider like Thordata (trial traffic: https://www.thordata.com/?ls=dev&lk=dev-1) is a cheap way to validate them before you commit bandwidth.

From Video URLs to Training Records: How a Model-Ready Video Dataset Is Structured

rhea hollis — Mon, 31 Aug 2026 07:21:46 +0000

Most “video dataset” posts show you a folder of .mp4 files. That's the wrong abstraction. If you're training a VLM or building a retrieval system, the video file is one field in a record — and the quality of everything around it decides whether your pipeline works. Here's how we structure records in the video datasets we just launched at Thordata, and the engineering choices behind it.

The record, not the file. Each video maps to a structured entry with four layers:

{
"video": {
"title": "...",
"description": "...",
"publish_time": "...",
"category_labels": ["..."],
"engagement": { "views": 0, "likes": 0 }
},
"creator": {
"channel_id": "UC...",
"handle": "@...",
"channel_url": "..."
},
"multimodal": {
"captions": "...",
"transcript": "...",
"audio_scene_info": "..."
},
"provenance": {
"platform": "youtube",
"video_url": "https://...",
"platform_video_id": "..."
}
}
Provenance is a first-class layer, not an afterthought — retroactive provenance means re-collecting the dataset. Field availability varies by platform and dataset scope, so treat the schema as a contract you inspect per dataset, not a universal constant.

Formats chosen per data type. Metadata and transcripts ship as JSON; tabular analysis works better in CSV or Parquet, and Parquet in particular pays off once you're filtering millions of rows by category or language. Video files deliver as .mkv or .mp4, audio as .m4a or .mp3. Don't force one format onto everything — pick per data type, and export all three metadata formats from the same source of truth.

Delivery is engineering, not logistics. Datasets land where your stack already reads from: Amazon S3, Azure Blob, GCS, SFTP, or Webhook, on-demand or on a custom schedule. On our side that means 8–10 PB of daily delivery capacity; on your side it means zero new ingestion code. If your workflow is API-shaped instead of batch-shaped, on-demand retrieval with field-level filtering hits the same records.

Sizing the corpus. 17B+ indexed video URLs across 700M+ channels in 100+ languages is the ceiling, not the unit of purchase. The practical move: pull a sample, inspect the schema and field coverage for your platforms, define filters (content type, upload date, view metrics, quality), validate with a test batch, then scale to full delivery. Custom collection covers the gaps the predefined datasets don't.

Quality notes worth knowing. Videos deliver up to 2K resolution, audio at the best available source quality, and every dataset we ship is consent-approved content cleared for AI training — check that box before you check any other.

The mental model that ties it together: treat a video dataset like a database with media attachments, not a media folder with a spreadsheet. Your training loop, your dedup jobs, and your future self will all benefit.

We just launched these datasets at Thordata — sample access and docs:https://www.thordata.com/products/multi-platform-video-datasets?op=rhea&from=x