DEV Community

rhea hollis
rhea hollis

Posted on

Video Scraping Is a Bandwidth Problem: Four Rules for Proxying Large-File Collection

#webdev #python #tutorial #scraping

Most proxy advice is written for text: rotate per request, retry on failure, move on. The first time you point that stack at video, it falls apart in new and expensive ways. Four rules that fixed it for me.

Rule 1: One sticky session per file

An IP switch mid-download means a restart or a corrupted file — you pay for the same bytes twice. Bind the session ID to the file, not the job:

# pseudo-Python
for video in queue:
    sid = new_session_id()          # sticky, up to 90 min
    for attempt in range(3):
        if download(video.url, session=sid):
            break                    # completed on ONE ip
    release_session(sid)             # fresh ip for the NEXT file
Enter fullscreen mode Exit fullscreen mode

Ninety minutes of stickiness covers most clips end-to-end; anything longer belongs in a resumable-download design, not a longer session.

Rule 2: Rotate between files, never during one

Address diversity still matters — you're just moving the rotation boundary. Fresh IP per file gives you the same spread as per-request rotation would, without ever breaking a transfer in progress.

Rule 3: Parallelize by file, and check your concurrency cap

Video throughput comes from per-file parallelism: one lane per file, each on its own sticky session. The bottleneck is almost never your code — it's providers that cap concurrent sessions. Pools without such caps exist — Thordata's residential network, for one (unlimited concurrency, 100M+ IPs; trial traffic here: https://www.thordata.com/?ls=dev&lk=dev-1) — but the pattern is provider-agnostic: verify the cap, then size your worker pool to it.

Rule 4: Budget in cost-per-file, not cost-per-month

At $0.65/GB, a 50MB clip costs ~3.3 cents and a 20,000-clip run costs ~$650 in bandwidth. Multiply average file size by count by price before every run — and remember that every restarted file is bandwidth you pay for twice. Cheap-but-flaky pools are the expensive ones.

The mental shift that ties it together: with text, the proxy is a per-request anonymizer. With video, it's a transfer layer — and transfer layers are engineered around sessions, lanes, and bandwidth, not around hiding.

If you're evaluating providers for large-file collection, these four rules are the checklist — and a trial with a session-control provider like Thordata (trial traffic: https://www.thordata.com/?ls=dev&lk=dev-1) is a cheap way to validate them before you commit bandwidth.

Top comments (0)