One Video Request, Ten Thousand Text Rows: Budgeting Proxy Bandwidth in Multimodal Collection

The pricing model for residential proxies — cents per GB, billed at the exit — is nearly invisible when you collect text. A listing page is 40 KB; at $8 per residential GB that is a fraction of a cent, and nobody opens a spreadsheet. Then a team decides to build a multimodal corpus: image-text pairs, egocentric video clips, audio snippets. Same plumbing, same rotation config, same retry policy — and the first invoice looks like a typo, because the unit of work changed by three orders of magnitude and the proxy policy never followed.

The thesis: multimodal collection is not "more scraping." It is a bandwidth-engineering problem wearing a scraping costume. The proxy decisions that are free for text — retry liberally, rotate per request, fetch through premium residential tiers — become actively ruinous for assets unless you redesign around three moves: split metadata from payloads, pay for failed bytes only once, and route the two classes through different economics.

Do the arithmetic the dashboard is hiding

Let's make the cost structure explicit, because intuition fails here. A typical HTML listing fetched through a rotating residential exit: ~40 KB of transferred bytes, including handshake overhead. A 4K still image from a CDN: 2–5 MB. A 10-second 1080p clip: 10–50 MB. A single training sample in a video corpus can be 10,000 times the bytes of a text row. At $8/GB, a 20 MB clip is ~$0.16 — sounds fine until you scale to a 50,000-clip crawl with a 6% asset-level failure-and-retry rate:

GB = 1024 ** 3


def crawl_cost(n_items, item_bytes, price_per_gb, fail_rate, retries=1):
    billed_items = n_items * (1 + fail_rate * retries)
    return billed_items * item_bytes / GB * price_per_gb


text_rows = crawl_cost(500_000, 40 * 1024, 8, fail_rate=0.04)
clips     = crawl_cost(50_000, 20 * 1024 ** 2, 8, fail_rate=0.06)
print(f"text corpus: ${text_rows:,.0f}")    # ~ $125
print(f"video corpus: ${clips:,.0f}")       # ~ $6,600+

Ten times fewer items, fifty times the bill — before considering that a clip dropped at 90% transfer re-buys 90% of itself on naive retry. The failure rate may be identical to your text pipeline; the failure price is what changed. Any budget conversation that starts with "it's the same pipeline" has already lost.

Split the two fetches — they are not one job

A multimodal pipeline actually contains two entirely different crawl patterns stitched together:

1. Metadata collection: discover URLs, captions, durations, license fields, thumbnails' existence. Small responses, high request counts, aggressive anti-bot defenses (you are hitting search, detail, and listing pages). This is classic scraping. Residential rotation is right here — it is exactly what the previous fifteen articles in this series argued.
2. Payload fetching: pull the actual bytes from CDNs once URLs are known. Huge responses, lower request counts, generally much friendlier defenses, because CDNs are built for exactly this traffic and rarely fingerprint per-client the way app pages do.

Stitching them into one loop that uses one proxy policy means either over-paying (residential rotation for megabytes) or under-defending (datacenter exits on the defended pages). Separate them and each class gets the exit tier its counterparty actually requires:

from dataclasses import dataclass


@dataclass
class PipelinePolicy:
    metadata_proxies: list[str]   # rotating residential; geo coverage matters
    asset_proxies: list[str]      # cheaper static/datacenter/ISP exits
    max_asset_mib: int = 80       # refuse absurd payloads before they bill
    connect_timeout_s: float = 10
    read_stall_timeout_s: float = 30


async def fetch_asset_policy_ok(session, url, policy: PipelinePolicy):
    # Cheap pre-flight: HEAD the asset, check size, THEN decide to pay for it.
    async with session.head(url, proxy=policy.asset_proxies[0],
                            allow_redirects=True) as r:
        size = int(r.headers.get("content-length") or 0)
        if size > policy.max_asset_mib * 1024 * 1024:
            return "skipped-oversize", 0
        if r.status >= 400:
            return "meta-fail", 0
    return await stream_to_disk(session, url, size, policy)

That HEAD pre-flight looks trivial. Multiply by a corpus where 3% of discovered assets are 400 MB misfiled originals and you just cut the bill measurably — you refused to spend proxy GB on bytes you would have discarded at the dedup stage anyway.

Never pay for a failed payload twice

A text row that fails at 99% of its transfer costs you nothing extra worth counting. A video that fails at 90% of 20 MB just burned 18 MB of paid residential traffic and now wants to do it again. Two engineering responses, both standard CDN behavior that most scrapers forget exists:

Range-resume. Most asset CDNs support Range headers. Persist the partial file and the ETag, resume the interrupted remainder, and cap the re-buyable fraction to what was actually missing.

import os

import aiohttp


def _resume_state(dest: str, expected_size: int) -> int:
    """How many already-paid bytes sit on disk, and is it worth resuming."""
    if not os.path.exists(dest):
        return 0
    size = os.path.getsize(dest)
    return 0 if (expected_size and size >= expected_size) else size


async def stream_to_disk(session, url, expected_size, policy, dest="clip.mp4"):
    downloaded = _resume_state(dest, expected_size)
    headers = {"Range": f"bytes={downloaded}-"} if downloaded else {}

    timeout = aiohttp.ClientTimeout(
        connect=policy.connect_timeout_s,
        sock_read=policy.read_stall_timeout_s,   # stall detector, NOT total time
    )
    async with session.get(url, proxy=policy.asset_proxies[0],
                           headers=headers, timeout=timeout) as resp:
        if downloaded and resp.status != 206:
            downloaded = 0            # CDN ignored Range: restart from scratch
        with open(dest, "ab" if downloaded else "wb") as fh:
            async for chunk in resp.content.iter_chunked(256 * 1024):
                fh.write(chunk)
                downloaded += len(chunk)

    complete = (expected_size == 0) or (downloaded >= expected_size)
    return ("ok" if complete else "partial"), downloaded

Stall timeouts instead of total timeouts. Note the sock_read timeout: abort a transfer only when it stops moving for thirty seconds. A total=30 timeout on a 50 MB file across a slow-but-working exit is a self-inflicted wound: you cut healthy transfers at arbitrary byte offsets, re-bill yourself for the severed tail, and your "failure rate" is an artifact of the constant you typed. Total timeouts fit text (small objects, all-or-nothing). Stall detection fits payloads (large objects, resumable). This single distinction is worth real money in a video crawl.

Deduplicate on identity, not on paid bytes

The cheapest proxy GB is the one never spent. Asset corpora have heavy natural duplication — the same stock clip embedded on hundreds of pages, the same product image under two URLs with different query strings. A hash-only pipeline discovers this after paying for every copy through an exit. Move identity upstream: canonicalize URLs (strip tracking params), keep a seen-set of content hashes from HEAD-level ETags and size pairs when available, and skip the GET when a cheaper identity signal already says "we have this." One regex over a discovered-URL list routinely removes 5–15% of asset GETs in image corpora, at zero collection cost, because you are refusing bytes you had already bought once.

Connection reuse, the unglamorous multiplier

Per-request rotation is the correct default for the metadata phase; applied to payloads it is actively wasteful, because every new exit means a new TCP+TLS handshake with the origin CDN — 10–50 KB of protocol chatter per asset, paid again per GB, plus the round trips that make megabyte transfers slower and stall-prone. For the asset phase, keep one healthy exit per target CDN host for the duration of a shard's work, and let HTTP keep-alive amortize the handshake across fifty clips instead of paying it fifty times. One line of policy difference — rotate metadata, coalesce assets — and the aggregate savings is double-digit percent of the payload bill, with the defense posture of the metadata phase untouched.

The honest version

Multimodal corpora do not break your proxy budget because videos are expensive; they break it because teams point a text-shaped pipeline at video-shaped physics. Do the arithmetic in dollars-per-sample before launch, not after the invoice. Split the pipeline into its two fetch classes and give each the exit tier its counterparty actually defends against. Refuse oversize payloads with a HEAD before you buy them, resume partial downloads with Range instead of re-buying them, and abort on stalls instead of totals. Canonicalize identity upstream so dedup skips GETs instead of skipping INSERTs. None of this is exotic; it is exactly the plumbing conversation you would have inside a CDN-heavy download service — you just happen to be paying per egress GB, which means every ignored handshake, every re-bought severed tail, every duplicate clip is a line item. Text forgives sloppy bandwidth engineering. Video bills you for it.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele