DEV Community: Flora

The Hard Part of Video Datasets Isn't the Scraper: Reconciling YouTube, TikTok, and Shorts Into One Schema

Flora — Thu, 17 Sep 2026 09:08:09 +0000

Every team that says "we want video data for trend analysis" starts the same way. Someone points a scraper at three platforms - YouTube, TikTok, and Shorts - pulls a few thousand rows, and everything looks fine in the notebook. Three weeks later the pipeline starts lying. Not loudly. It merges rows, drops rows, or produces per-platform engagement numbers that look real but aren't comparable.

The scraper wasn't the problem. The schema was.

Why three "video" objects don't fit

Each platform has an internally consistent model of what a video is, and the differences don't line up cleanly.

Identity. YouTube keys on videoId (11-char base64-ish string) with channelId for the creator. TikTok keys on a numeric-string video.id with author.secUid and author.uniqueId. Shorts are technically YouTube videos, but a video only becomes a Short if it's vertical and under three minutes - the same videoId can appear in a Shorts feed and the main feed, and each surface has its own URL pattern and a different engagement velocity. Any "cross-platform" table you build ends up needing a canonical concept-level id plus an alias table, not one id.

Time. YouTube's snippet.publishedAt is RFC 3339 UTC with second precision. TikTok's create_time is a Unix-seconds integer with no timezone attached, and the same value looks different for a video first posted via a scheduler. Shorts inherit the underlying YouTube publish time even if the video was re-cut and re-uploaded as a Short a month later. Duration is worse: YouTube returns ISO-8601 via contentDetails.duration (PT1M32S, which will bite you at PT0S for live streams and at P1DT2H for marathons), while TikTok returns integer seconds.

Engagement. The set of counters you can pull differs. YouTube exposes views, likes, comment count. TikTok exposes plays, likes, comments, shares, and saves - and, since it removed public dislikes, no dislikes at all. Shorts expose "plays" that behave differently from a YouTube "view" even for the same underlying asset. Once you ask "engagement per hour since publish" across the three, you can't naively divide one number into another.

Text. YouTube splits title, description, and tags (which are separate from the hashtags a creator drops in the description). TikTok puts everything in one caption field with hashtags embedded, plus duetInfo / stitchInfo flags. Shorts inherit the YouTube text model, but hashtags matter disproportionately in that feed.

Audio. TikTok exposes music.id, music.original, music.duration - a sound-driven medium. YouTube has no equivalent field. A viral TikTok clip is often one sound replicated across thousands of videos; without a sound_id in your normalized row, you can't cluster them.

Aspect ratio and length bucket. 9:16 vs 16:9 is not a cosmetic column. If the dataset feeds a vision model, you'll bucket by aspect ratio before you resize. Mixing 4K landscape with 720p vertical without a bucket column silently degrades training.

The shape of a normalized row

After enough rounds of patches, the row I keep has roughly these fields. It isn't universal - the point is that it's stable enough to write queries against.

canonical_video_id        string    stable pipeline-assigned id
platform                  enum      youtube | tiktok | shorts
platform_native_id        string    videoId / video.id
creator_id                string    channelId / secUid
creator_handle            string    display handle
publish_ts                int64     unix seconds, UTC
duration_s                int32
aspect_ratio              float     e.g. 0.5625 for 9:16
title                     string    caption on TikTok
hashtags                  array<string>
description               string    empty on TikTok
play_count                int64     views on YT, plays on TT/Shorts
like_count                int64
comment_count             int64
share_count               int64     null on YouTube (not exposed)
save_count                int64     null except TikTok
sound_id                  string    TikTok only
is_sound_original         bool      TikTok only
first_seen_ts             int64     when your pipeline saw this row
last_seen_ts              int64

Anything the platform doesn't give you is null, never zero. Zero on share_count means "we observed zero shares"; null means "the platform doesn't expose this". If you skip that rule, per-platform comparisons come out backwards.

A normalize function that survives review

import re, hashlib
from datetime import datetime
from isodate import parse_duration   # pip install isodate

def canonical_id(platform: str, native_id: str) -> str:
    """Stable id for a row that may be observed on multiple surfaces (Shorts + YouTube)."""
    return hashlib.sha1(f"{platform}:{native_id}".encode()).hexdigest()[:16]

def norm_youtube(item: dict) -> dict:
    sn = item["snippet"]
    cd = item.get("contentDetails", {})
    st = item.get("statistics", {})
    pub = datetime.fromisoformat(sn["publishedAt"].replace("Z", "+00:00"))
    dur_iso = cd.get("duration")
    duration_s = int(parse_duration(dur_iso).total_seconds()) if dur_iso else 0
    desc = sn.get("description", "") or ""
    hashtags = re.findall(r"#(\w+)", desc)
    return {
        "platform": "youtube",
        "platform_native_id": sn["resourceId"]["videoId"],
        "creator_id": sn["channelId"],
        "creator_handle": sn.get("channelTitle", ""),
        "publish_ts": int(pub.timestamp()),
        "duration_s": duration_s,
        "aspect_ratio": None,          # not reliably derivable from v3 API
        "title": sn["title"],
        "hashtags": hashtags,
        "description": desc,
        "play_count": int(st.get("viewCount", 0)),
        "like_count": int(st["likeCount"]) if "likeCount" in st else None,
        "comment_count": int(st["commentCount"]) if "commentCount" in st else None,
        "share_count": None,
        "save_count": None,
        "sound_id": None,
        "is_sound_original": None,
    }

def norm_tiktok(item: dict) -> dict:
    stats = item.get("stats", {})
    ts = int(item.get("create_time", 0))
    caption = item.get("desc", "") or ""
    hashtags = re.findall(r"#([\w\u4e00-\u9fff]+)", caption)
    video = item.get("video", {})
    w = video.get("width") or 0
    h = video.get("height") or 0
    music = item.get("music") or {}
    return {
        "platform": "tiktok",
        "platform_native_id": str(item["id"]),
        "creator_id": item["author"]["secUid"],
        "creator_handle": item["author"]["uniqueId"],
        "publish_ts": ts,
        "duration_s": int(video.get("duration", 0) // 1000),   # ms -> s
        "aspect_ratio": (h / w) if (h and w) else None,
        "title": caption.split("#", 1)[0].strip(),
        "hashtags": hashtags,
        "description": "",
        "play_count": int(stats.get("playCount", 0)),
        "like_count": int(stats.get("diggCount", 0)),
        "comment_count": int(stats.get("commentCount", 0)),
        "share_count": int(stats.get("shareCount", 0)),
        "save_count": int(stats["collectCount"]) if "collectCount" in stats else None,
        "sound_id": str(music.get("id")) if music else None,
        "is_sound_original": bool(music.get("original", False)) if music else None,
    }

def to_row(n: dict, first_seen: int) -> dict:
    row = dict(n)
    row["canonical_video_id"] = canonical_id(n["platform"], n["platform_native_id"])
    row["first_seen_ts"] = first_seen
    row["last_seen_ts"] = first_seen
    return row

Small, and not exhaustive. The point is that each norm_* maps a platform-specific response into the same field names and uses None for what the platform doesn't expose. Anything downstream (dedup, engagement curves, model input) reads row["like_count"] and doesn't care where it came from.

The aliasing that turns platform_native_id into canonical_video_id is where most pipelines eventually break. Rule of thumb: keep two ids - one for the row you observed on a given surface, one for the concept ("this piece of content") that may show up on two or three surfaces. A YouTube Short's videoId is the same as its main-feed videoId, but you still want those two observations to be joinable at concept level, not deduped into oblivion.

Things that break in week three

Timezones. TikTok's create_time is nominally UTC-seconds, but tooling sometimes interprets it in the creator's local zone. Fix it at ingest, not at query time.

Deleted / private videos. A row you scraped last week is 404 today. Keep first_seen_ts and last_seen_ts on every row, and never overwrite the earliest first_seen. Your "engagement over time" charts need both columns.

Play-count ceilings. TikTok's public playCount is known to saturate around the 600k mark in some response paths. Treat values at that magnitude as a floor, not a true count.

Shorts inheriting YouTube comments. If a creator re-edits a landscape video into a vertical Short with the same underlying asset, the comment stream is shared. Dedup comments by comment id, not by row.

duration at PT0S and P1DT2H. Both are technically valid ISO-8601 durations, and both will crash a naive parse_duration(d).seconds. Use .total_seconds(), and bucket live vs VOD before aggregating.

Tags are not hashtags. YouTube tags are metadata the creator picks (often keyword-stuffed); hashtags in the description are what viewers actually see. Do not merge them into one column or your topic model will drift.

Aspect ratio isn't in the YouTube v3 API. You can infer verticality from duration under 3 minutes plus a channel that ships Shorts, but inference is not measurement. If the pipeline depends on a real aspect ratio, either scrape the player metadata, key it from a video-download pass, or leave it null.

Buy or scrape?

If you need today's videos, or one specific creator's corpus, scrape it - the field mapping above is essentially the whole job. If you need a historical window across thousands of creators ("all videos that mention brand X in the last 90 days, with their engagement curves"), scraping row-by-row is slower, blockier, and more IP-hungry than you'd want. That's the case where a pre-built multi-platform dataset is a better shape: fewer requests, more coverage per dollar, and you start from a normalized table instead of an ETL job.

I work with Thordata on their video products (partner disclosure - this is not a neutral review), and their multi-platform video datasets and video data scraper sit on either side of that trade. What I'd actually recommend, though, is running the normalize function above on a small sample from whatever source you end up with - buy, scrape, or a mix - before you commit to a schema. The mapping step is the fragile part. Fetch is a boring choice you can make later on cost per row.

Beyond the Exit IP: How WebRTC and DNS Undo Your Residential Proxy (and How to Verify It)

Flora — Thu, 17 Sep 2026 05:41:14 +0000

A residential proxy only controls the path your HTTP requests take. It does not automatically control every way a browser can reveal where it is. That gap is where a surprising number of "clean IP but still blocked" cases come from, and WebRTC is the loudest version of it.

Here is the failure I see most often in scraping setups that already do IP hygiene right. You route traffic through a residential exit in Berlin. The HTTP layer looks fine: your fetch goes out a Berlin ASN, the site logs a Berlin address, geolocation APIs agree. Then the same page opens an RTCPeerConnection, fires a couple of STUN binding requests, and the reflexive candidate it gathers reports your real outbound address 鈥?the datacenter box the headless browser actually runs on, or a home broadband IP that is nowhere near Berlin. One request says Berlin, the other says "actually, Frankfurt, AS-NOT-A-RESIDENTIAL." Modern anti-bot does not need to block you on that. It just needs to stop trusting you and quietly downgrade you to a challenge or a thin-data page.

What the proxy does not govern

When you hand Chromium an HTTP or SOCKS5 proxy, it proxies the connections you make through the normal network stack. WebRTC's ICE gathering is a different animal: the browser tries to discover the addresses the peer can be reached at, and part of that discovery is direct UDP STUN. On many default configurations that UDP path does not respect the proxy. So the IP you want the page to see and the IP WebRTC thinks it has diverge.

There is a wrinkle that makes this less obvious than it used to be. Recent Chromium builds redact local and private ICE candidates behind mDNS 鈥?instead of a plain 192.168.x.x in the candidate list you now get something like a1b2c3.local. That killed the old-school "print the local IP from WebRTC" trick and made a lot of people think WebRTC was safe now. It is not, for our purpose: the redaction covers local addresses, but the server-reflexive candidate 鈥?the public address a STUN server saw you come from 鈥?still gets collected unless you stop it. And that public reflexive address is exactly the geo/ASN contradiction that matters.

The blunt fix: force WebRTC through the proxy

The supported way to stop non-proxied UDP is the webrtc-ip-handling-policy switch. disable_non_proxied_udp tells the browser not to send UDP that is not going through the proxy, which collapses the leak because the direct STUN path is gone.

from playwright.sync_api import sync_playwright

PROXY = {
    "server": "http://gateway.your-residential-provider.net:8000",
    "username": "cust-session-42",   # session/user encodes the target geo
    "password": "***",
}

def launch(p):
    return p.chromium.launch(
        headless=True,
        args=[
            "--webrtc-ip-handling-policy=disable_non_proxied_udp",
            "--disable-features=AsyncDns",
        ],
        proxy=PROXY,
    )

with sync_playwright() as p:
    browser = launch(p)
    ctx = browser.new_context(locale="de-DE", timezone_id="Europe/Berlin")
    page = ctx.new_page()
    page.goto("https://example-target.com/listing", wait_until="domcontentloaded")
    # ... your extraction ...
    browser.close()

Two things to note in that snippet, because they are where people get bitten. First, disable_non_proxied_udp may break WebRTC entirely for pages that genuinely need it 鈥?that is fine when you are scraping a listing page, but it is a real behavior change, not free. Second, the flag is not version-proof. Chromium has moved around how command-line WebRTC policies are honored across releases, and enterprise-policy variants (WebRtcIPHandling) can take precedence. The flag is necessary-but-verify, not set-and-forget. Which is the whole point of the next section.

Do not trust the fix 鈥?measure it

The honest version of proxy work is that you do not know you closed the leak until you have looked at what the browser actually gathers and compared it to what an HTTP request shows. That test is small and worth putting in CI.

Run one gather of ICE candidates inside the browser, and one plain HTTP echo of your public IP through the same proxy, and assert they tell a consistent story.

JS_GATHER_CANDIDATES = """
async () => {
  const out = [];
  const pc = new RTCPeerConnection({ iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] });
  pc.createDataChannel('');
  pc.onicecandidate = e => { if (e.candidate) out.push(e.candidate.candidate); };
  const done = new Promise(res => { pc.onicegatheringstatechange =
      () => pc.iceGatheringState === 'complete' && res(); });
  await pc.createOffer().then(o => pc.setLocalDescription(o));
  // ICE gathering can hang forever behind some proxies; bound it.
  await Promise.race([done, new Promise(r => setTimeout(r, 4000))]);
  pc.close();
  return out;
}
"""

import json, urllib.request
def http_public_ip(proxy_url):
    req = urllib.request.Request("https://api.ipify.org?format=json")
    with urllib.request.urlopen(req, timeout=8) as r:
        return json.load(r)["ip"]

# Inside a page context after launch():
candidates = page.evaluate(JS_GATHER_CANDIDATES)
http_ip = http_public_ip("http://gateway.your-residential-provider.net:8000")
print("HTTP sees:", http_ip)
for c in candidates:
    print("candidate:", c)

What you are looking for: any candidate line containing host or srflx with a public IPv4 that is not the HTTP sees: address. .local mDNS entries are the browser redacting a private address and are acceptable; a raw public srflx that disagrees with the proxy is the leak. If your provider exposes a geo lookup, resolve both IPs and confirm the country (and ideally ASN type) matches. If they diverge, the policy flag did not take on that build, and you are shipping a contradiction to every anti-bot that checks it.

The same "the proxy is not a magic cloak" logic extends past WebRTC. --disable-features=AsyncDns matters because Chromium's built-in asynchronous resolver can resolve names directly and leak the DNS query path to your host's resolver, which geolocates to the wrong place even when HTTP is proxied. And a proxy does nothing at all about Intl.DateTimeFormat().resolvedOptions().timeZone, the WebGL renderer string, or navigator.hardwareConcurrency 鈥?so pairing a Berlin exit IP with a America/New_York timezone and a Google cloud GPU is its own tell. IP is one signal; the job is to keep the signals consistent, not to swap one number and call it done.

Operationally, the cheapest thing you can do is treat the echo-vs-candidate check as a log line instead of a one-off experiment. On every Nth navigation, record the HTTP echo IP, its resolved country and ASN type, and any public srflx candidate you gathered, then alert when they disagree. A mismatch that shows up once is a flaky STUN path; one that shows up on every request from a given exit is the proxy configuration or the browser build, and you want to catch it before the target's trust score does. The failure is rarely loud 鈥?you get valid-looking HTML with prices hidden, results truncated, or a CAPTCHA only on your best-selling queries 鈥?so the signal has to come from your own instrumentation, not from a non-200.

Why the override trick is the tempting wrong answer

The internet full of "stealth" snippets shows you overriding window.RTCPeerConnection with an init script that rewrites or empties the candidate list. It is quick, and it is worse than the flag for a specific reason: a hand-rolled RTCPeerConnection is trivially detectable. A target can call Function.prototype.toString.call(window.RTCPeerConnection) and see that the "native" constructor now prints JavaScript instead of [native code], or compare the methods it expects against what you left behind. An empty candidate array is also a signature: real browsers on real networks gather something. You replaced a geo contradiction with a bot-ness contradiction, which is often the more expensive one.

So the practical ranking, from most to least defensible: honor the policy at the engine level and verify it; if you cannot get the engine policy to hold on your build, run WebRTC through a proxy that actually carries the UDP path; and treat a raw JS monkey-patch as a last resort that itself needs to be made consistent with the rest of the fingerprint. When the whole point is a believable real browser and the target leans hard on these signals, that is also the honest case for using a managed scraping browser that owns the browser-fingerprint-and-IP consistency problem for you, instead of rebuilding it flag by flag and version by version.

The part I would tell you in person

I write this while partnered with Thordata, so treat the product mention as advertising and the reasoning as the part worth keeping. The reasoning generalizes: your provider's residential line (residential proxies) buys you a plausible exit IP; it does not buy you a plausible browser. Thordata's scraping browser exists for the case where you have decided the fingerprint-consistency tax is bigger than the maintenance tax of doing it yourself 鈥?which, on the targets that actually matter, is usually the right call.

The takeaway that survives any provider: after you fix the IP, run the candidate-vs-echo test above on your own build. A closed leak you have measured is worth more than a leak you assumed the flag handled. If the two numbers agree, you stopped contradicting yourself across protocols. If they do not, you did not, and the cleanest residential IP in your pool will not save you from it.

That "Residential" IP Might Be a Datacenter in Disguise - A Scriptable Proxy Audit

Flora — Thu, 17 Sep 2026 02:10:38 +0000

The scenario nobody plans for

You buy a residential proxy plan. Weeks later your scraper's block rate quietly doubles, and support tells you the pool is "100% real residential IPs, peer-to-peer sourced." You have no way to check that claim, so you either trust it or you churn.

This is uncomfortable to write about as someone affiliated with a proxy vendor — which is exactly why you should learn to verify it yourself. Proxy type is not a marketing adjective. It is a property of an IP address that anyone can look up, because every public IP is registered to an ASN, and ASNs are classified as hosting providers, ISPs, or mobile carriers at the registry level. If your "residential" exit IPs resolve to Hetzner or OVH address space, no amount of branding changes what the target website sees.

In this post I'll walk through what actually distinguishes the three main proxy types, then give you a small Python script that samples egress IPs through your gateway and classifies them. It runs in about twenty minutes and uses only requests plus two free public APIs.

Why the distinction matters mechanically

When a site's anti-bot system scores a visiting IP, the single most informative feature is the reputation of the /24 (or /48) block the IP belongs to. And the dominant driver of block-level reputation is the ASN's registered type:

Datacenter (hosting) space — ASN owned by a cloud or hosting provider: AWS, Google Cloud, Hetzner, OVH, Vultr, DigitalOcean, Contabo, and thousands of smaller ones. Every serious bot-management vendor maintains a hosting-space list. Requests from these blocks get elevated scrutiny almost regardless of behavior. These IPs are fast and cheap, which is why datacenter proxies have a legitimate role (more on that later), but they are not residential.
Consumer ISP (fixed-line) space — ASN owned by Comcast, AT&T, Deutsche Telekom, Telefónica, Jio, and so on. This is what a true residential pool is: end-user devices behind consumer connections. Traffic looks like normal households, so block rates are much lower — at the cost of latency, because your request now traverses real users' links.
Mobile carrier space — IPs from T-Mobile, Vodafone, Airtel and similar, typically behind CGNAT. Hundreds or thousands of real users share the same egress IP, so the blocklists are reluctant to poison a single IP. This is the highest-trust tier, and the most expensive per gigabyte.

There is a fourth category that shows up in audits: static ISP proxies, which are IPs allocated from consumer ISP space but parked in a rack rather than attached to home devices. They register as "residential type" in most geo databases, behave like residential to many blocklists, and offer datacenter-ish speed. That gray zone is precisely where mislabeling is easiest to sell — and easiest to detect, because the ASN owner is public record.

The three independent signals

Any single check can be fooled; three checks agreeing is persuasive.

ASN owner and type. Registry data (RIPE/ARIN/APNIC via whois, or conveniently via ip-api.com) gives the organization behind the IP. "HETZNER-AS" in your residential plan is a red flag; "COMCAST-7922" in your datacenter plan is a surprise in the other direction.
Reverse DNS (PTR record). Consumer ISPs usually assign customer PTR names (cpe-76-xxx.res.spectrum.com, pppoe-..., host-by-...). Datacenter blocks use provider PTR patterns (static.94.23.44.5.ovh.net). Many residential IPs have no PTR at all — absence is normal and not a failure signal, but a PTR naming a hosting provider is a strong one.
Geolocation consistency. Compare the city/country your gateway claims to use against what three-party geo databases report for the egress IP. Frequent city-level disagreement means either sloppy pool metadata or traffic exiting somewhere other than what you paid for. Be careful here: consumer-grade geolocation databases genuinely disagree at city level even for honest IPs, so treat this as evidence, not a verdict.

The audit script

Requirements: Python 3.9+, pip install requests. The free ip-api.com endpoint is HTTP-only and rate-limited to about 450 requests/minute, so we sleep between samples.

A quick manual check first — before you need the whole script:

# what IP do I actually egress as?
curl -s -x http://USER:PASS@gateway.thordata.com:9999 https://api.ipify.org?format=json

# what does the registry think that IP is?
curl -s "http://ip-api.com/json/203.0.113.42?fields=status,country,city,isp,org,as"

If the org field says anything resembling a hosting provider while you're paying for residential, you already have your answer. The script below automates a statistically meaningful version of that check: sample N egress IPs, look up each one, classify, and report the distribution.

"""proxy_audit.py — sample egress IPs through a gateway and classify pool composition.

Usage:
  python proxy_audit.py --proxy http://USER:PASS@GATEWAY:PORT --sessions 30
Most residential/ISP gateways rotate the exit IP per connection (or per
minute); if your gateway supports sticky sessions, do NOT enable them here,
or you will measure one IP thirty times.
"""

import argparse
import json
import socket
import time
from collections import Counter

import requests

DATACENTER_KEYS = [
    "hetzner", "ovh", "vultr", "digitalocean", "contabo", "scaleway",
    "amazon", "microsoft", "google cloud", "fastly", "cloudflare",
    "akamai", "equinix", " Choopa", "linode", "upcloud", "xtom",
    "serverius", "pskz", "hostkey", "bluevps",
]
MOBILE_KEYS = [
    "t-mobile", "telekom mobile", "vodafone", "orange", "airtel",
    "jio", "china mobile", "china unicom", "verizon wireless", "at&ts",
]

def classify(meta: dict, ptr: str | None) -> str:
    blob = f"{meta.get('org', '')} {meta.get('isp', '')} {ptr or ''}".lower()
    if any(k in blob for k in MOBILE_KEYS):
        return "mobile-carrier"
    if any(k in blob for k in DATACENTER_KEYS):
        return "datacenter"
    # An ASN owned by a known consumer ISP with no hosting keyword.
    if any(k in blob for k in ["comcast", "spectrum", "att", "verizon",
                              "telefonica", "movistar", "deutsche telekom",
                              "bt ", "telecom", "broadband", "cable"]):
        return "consumer-isp (residential-like)"
    return "unclassified"   # be honest: unknown is not a verdict

def reverse_dns(ip: str) -> str | None:
    try:
        return socket.gethostbyaddr(ip)[0]
    except (socket.herror, socket.gaierror, OSError):
        return None        # no PTR — common for real residential IPs

def sample_once(proxy_url: str) -> dict | None:
    proxies = {"http": proxy_url, "https": proxy_url}
    try:
        r = requests.get("https://api.ipify.org?format=json",
                         proxies=proxies, timeout=25)
        r.raise_for_status()
        ip = r.json()["ip"]
    except Exception as e:
        return {"error": f"egress failed: {type(e).__name__}"}
    try:
        m = requests.get(
            f"http://ip-api.com/json/{ip}"
            "?fields=status,country,regionName,city,isp,org,as",
            timeout=15).json()
    except Exception as e:
        return {"ip": ip, "error": f"lookup failed: {type(e).__name__}"}
    ptr = reverse_dns(ip)
    return {"ip": ip, "city": m.get("city"), "country": m.get("country"),
            "as": m.get("as"), "org": m.get("org"), "ptr": ptr,
            "class": classify(m, ptr)}

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--proxy", required=True, help="http://user:pass@host:port")
    ap.add_argument("--sessions", type=int, default=30)
    ap.add_argument("--sleep", type=float, default=2.0)
    ap.add_argument("--out", default="audit_results.jsonl")
    args = ap.parse_args()

    rows, tally = [], Counter()
    with open(args.out, "w") as f:
        for i in range(args.sessions):
            row = sample_once(args.proxy)
            row["sample"] = i
            rows.append(row)
            f.write(json.dumps(row) + "\n")
            tally[row.get("class", row.get("error", "error"))] += 1
            print(f"[{i+1:>3}] {row.get('ip','—'):<16} "
                  f"{row.get('city',''):<14} {row.get('class', row.get('error'))}")
            time.sleep(args.sleep)   # respect ip-api free-tier limits

    print("\n=== pool composition ===")
    for k, v in tally.most_common():
        print(f"{v:>4}  {100*v/args.sessions:5.1f}%  {k}")
    print(f"\nRaw per-IP evidence saved to {args.out} — keep it; it is the "
          "receipt you show your provider when something looks off.")

if __name__ == "__main__":
    main()

A couple of implementation notes that will save you a debugging cycle:

Pass the proxy under both "http" and "https" keys in the dict — requests needs the explicit https entry to tunnel TLS over the proxy, and api.ipify.org is HTTPS-only.
socket.gethostbyaddr throwing is expected behavior for many genuine residential IPs (no reverse record configured). Only treat a PTR that names a hosting provider as adverse evidence.
The keyword lists above are a starting point, not a taxonomy. When the script prints unclassified, look up that ASN's whois once by hand; registry "status" fields (ASSIGNED PI vs LEGACY vs allocatable-to-subscribers) tell you more than any keyword list.

Reading the results

What I'd expect a clean run to look like: a residential pool dominated by consumer-isp with a tail of unclassified (that's normal — there are ~60,000 consumer ISPs worldwide, far more than any keyword list covers). Now the failure signatures:

Majority datacenter on a residential plan. Either you were resold hosting space, or a mixed pool is configured to prefer the cheap half. Take the JSONL evidence to support with specific IPs and timestamps.
A stable handful of IPs repeating across samples. Two causes: sticky-session mode left on (fix your gateway config), or the pool genuinely has a few /24s it keeps reusing — which is also worth knowing, because reuse concentrates blocklists.
City mismatch rate above roughly 20%. Escalate gently; geo DB lag is real, but systemic country-level mismatch never is.
Mobile plan full of consumer-isp. Mobile proxies are priced at a premium for the carrier-CGNAT property; fixed-ISP IPs in that pool means you are overpaying.

And to be fair about the gray zone: static ISP proxies legitimately occupy the middle. They come from consumer-ISP ASN space (so this audit will happily classify them "residential-like") even though they sit in a rack. If you want datacenter reliability at residential-ASN reputation — dashboard monitoring, ad verification where sessions must persist for weeks — that's the right product, and the pricing gap is large: Thordata, for example, lists its ISP line starting at $2 per IP (fetched from their pricing page on 2026-09-17), a fraction of typical residential per-GB economics. Just buy it knowingly, not by accident.

Pools lie in both directions

The uncomfortable lesson from running audits is that labeling errors go both ways: some "enterprise residential" pools are 70% cloud, and some "cheap datacenter" pools turn out to hold real ISP space someone sold as shared hosting by mistake. The only durable response is to make verification part of your pipeline — sample N exits after each provider change, keep the JSONL, and diff the composition monthly. It costs you thirty lines of cron and thirty minutes of attention, and it converts every vague suspicion into a specific, fixable ticket.

Disclosure: I work with Thordata (sponsored account). The audit script above uses no Thordata-specific tooling — run it against any vendor's gateway. If you want to see what our residential pool composition looks like under this exact script, the product page is here: https://www.thordata.com/products/residential-proxies?ls=dev&lk=DEV — and new signups get a trial allowance, no code needed.

Residential proxies: what to measure before scaling

Flora — Thu, 17 Sep 2026 02:00:13 +0000

Residential proxies: a production evaluation guide

Proxy selection is often reduced to a question about IP volume or advertised speed. That is not enough for a production data workflow. The right proxy type depends on the target geography, session model, request pattern, failure tolerance and the value of each successful record.

This guide explains how to evaluate the infrastructure before scaling a compliant public-web data project.

1. Match the proxy type to the workflow

Residential proxies are useful when location diversity and natural traffic patterns matter. Test geography, session stability and successful results before scaling.

Residential proxies are generally useful when geographic diversity and residential network context matter. ISP proxies can be useful for repeatable sessions and predictable connectivity. Mobile proxies are relevant when the workflow needs mobile-network context. Datacenter proxies are often attractive for speed and controlled large-volume workloads.

The choice should be made from measured results, not from the product name alone.

2. Build a controlled test

Use a small, permitted set of public URLs that represents the real workload. Keep the test constant while changing only one variable at a time:

ext target URLs ? proxy type ? country ? session mode ? request rate ? result validation

For each request, record the timestamp, target country, proxy type, session identifier, response status, response time, retry count and whether the returned data is complete. Do not store credentials or personal information in the test logs.

3. Measure the metrics that affect cost

At minimum, calculate:

Success rate = successful usable results / total attempts
Median and P95 response time
Retry rate
Session break rate
Geographic accuracy rate
Cost per usable result

The last metric is usually more useful than cost per gigabyte. A low price is not efficient if failed requests create retries, manual review or incomplete downstream records.

4. Handle failure explicitly

Use bounded retries with exponential backoff. Stop retrying when the error indicates an invalid request or a policy restriction. Separate transport failures, timeouts, empty results and parsing failures so that one generic retry loop does not hide the real problem.

A basic workflow should also include a circuit breaker: if the success rate for a location falls below an agreed threshold, pause that route and alert the operator instead of increasing traffic.

5. Scale gradually

Move from a small test to a limited production run, then compare the same metrics again. Introduce concurrency gradually and watch response time, error rate, bandwidth and downstream parser quality. Keep a rollback path so a target, location or route can be paused without stopping the entire pipeline.

Today's related industry signal is reliable proxy infrastructure and practical AI data workflows. It is a reminder that data collection infrastructure needs operational measurement as well as a working API.

Thordata reference: https://www.thordata.com/blog/ai-trends

6. Responsible use

Use proxy infrastructure only for lawful, compliant workflows involving permitted public data. Respect applicable laws, website terms, robots directives, rate limits and opt-out requests. Do not use it for credential testing, fake accounts, click manipulation, spam, private-data collection or attempts to evade platform enforcement.

Try Thordata: https://www.thordata.com/?ls=dev&lk=DEV

Rotating Residential Proxies Without Losing Your Mind: 407 Auth, Sticky Sessions, and Backoff in Practice

Flora — Wed, 16 Sep 2026 09:16:18 +0000

The problem nobody tells you about

Most scraping tutorials stop at "set proxies={'http': ...} and go." That works for a weekend project pulling a few hundred pages. It falls apart the moment you scale: you start seeing 407 Proxy Authentication Required, mysterious Connection reset by peer, 429s that arrive in bursts, and — worst of all — silently degraded data where your parser returns an empty list because the site served you a captcha page with a 200 OK.

Rotating residential proxies fix the IP-reputation half of this, but they introduce their own failure modes. This post walks through the four that actually bite in production, with runnable Python you can lift into your own pipeline.

I'll use a generic residential gateway model (host + port + username + password) because that is how nearly every provider of this product category, including the residential proxies I run most of my scraping through at Thordata, hands you credentials. The code is provider-agnostic — swap in your own gateway host and auth string.

How residential rotation actually works

A residential proxy network puts you behind real ISP-assigned devices. You don't get one IP; you get a gateway that maps your request onto a pool of residential IPs. The two knobs that matter are:

Rotation trigger. Either every new connection (fresh exit IP per request) or sticky — the same exit IP held for a window (often 1–30 minutes) so a multi-step flow (login → add to cart → checkout) doesn't look like it's teleporting across a country between calls.
Targeting. Country / city / ISP selection. Tighter targeting = smaller pool = higher chance two of your workers collide on the same residential IP.

The golden rule: rotate for breadth, stick for depth. Collecting 50,000 independent product pages? Rotate every request. Filling out a paginated listing that sets a session cookie? Stick to one IP for that logical "session."

Failure mode 1: `407 Proxy Authentication Required`

The single most common first-hour error. It almost always means the credentials weren't attached to the request. Two gotchas:

Some gateways authenticate by IP allowlist, not user/pass. If you configured user/pass but your egress IP isn't allowlisted, you still get 407.
The requests library only sends proxy auth when the URL scheme matches. http:// proxy URLs are used for http:// targets; if you forget the https proxy, TLS requests bypass auth and fail differently.

Here's a minimal, correct example with requests:

import requests

GATEWAY = "gateway.provider.example"
PORT = 9000

# Rotation is often encoded in the username itself.
# A "zone" token or session id appended to the user string selects
# sticky vs rotating behavior on many residential gateways.
def proxy_for(session_id: str | None = None) -> dict:
    user = f"{BASE_USER}"
    if session_id:
        user = f"{BASE_USER}-session-{session_id}"  # sticky: same IP for window
    url = f"http://{user}:{BASE_PASS}@{GATEWAY}:{PORT}"
    return {"http": url, "https": url}

BASE_USER = "your-user"
BASE_PASS = "your-pass"

resp = requests.get(
    "https://api.ipify.org?format=json",
    proxies=proxy_for(),           # rotating
    timeout=(5, 20),
)
print(resp.status_code, resp.json())  # confirms the exit IP changed per call

A few things worth internalizing from that snippet:

timeout is a tuple (connect, read). Residential exits can hang on connect; a bare timeout=20 won't cut off a connection that never completes the handshake as cleanly.
The https key in the proxies dict is what makes TLS requests carry auth. Omit it and half your requests silently skip the proxy.

Failure mode 2: sticky sessions done right

For a multi-step flow, generate one session token per logical session and reuse it for every request in that session. The token is what the gateway hashes to pick (and hold) a residential IP.

import secrets, requests

def run_multi_step_flow():
    sid = secrets.token_hex(6)          # stable across the flow
    proxies = proxy_for(session_id=sid) # same exit IP until window expires
    s = requests.Session()
    s.proxies = proxies
    s.headers["User-Agent"] = "Mozilla/5.0 ..."

    home = s.get("https://shop.example/", timeout=(5, 20))
    home.raise_for_status()
    # cookie jar keeps the site-side session; sid keeps the IP-side session
    cart = s.post("https://shop.example/cart", data={"id": "abc"}, timeout=(5, 20))
    cart.raise_for_status()
    return home.status_code, cart.status_code

The trap: the sticky window is server-side. If your flow takes longer than the window (say 30 minutes of retries), the IP rotates mid-flow and the target site flags the discontinuity. Keep flows short, and if a step fails hard, start a new session id rather than retrying on the stale one.

Failure mode 3: concurrency and the collision problem

Residential pools are finite per geo. Fire 100 concurrent workers all targeting "United States" and two workers get the same residential IP; the target site's rate limiter sees a single very busy user and throttles you. This is where async shines — but you must bound concurrency.

import asyncio, aiohttp, random, time

GATEWAY, PORT = "gateway.provider.example", 9000
USER, PASS = "your-user", "your-pass"

async def fetch(session, url):
    # rotating: no session token -> fresh residential IP per connection
    proxy = f"http://{USER}:{PASS}@{GATEWAY}:{PORT}"
    async with session.get(url, proxy=proxy, timeout=aiohttp.ClientTimeout(total=25)) as r:
        if r.status in (429, 503):
            raise aiohttp.ClientResponseError(r.request_info, r.history, status=r.status, message="throttled")
        return await r.text()

async def worker(sem, session, urls, results):
    for u in urls:
        async with sem:
            for attempt in range(4):
                try:
                    results[u] = await fetch(session, u)
                    break
                except (aiohttp.ClientError, asyncio.TimeoutError) as e:
                    # exponential backoff + jitter: never hammer a 429
                    delay = (2 ** attempt) + random.uniform(0, 0.5)
                    await asyncio.sleep(delay)
            else:
                results[u] = None  # exhausted, mark and move on

async def main(urls, n_workers=20):
    sem = asyncio.Semaphore(n_workers)
    results = {}
    chunks = [urls[i::n_workers] for i in range(n_workers)]
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*(worker(sem, session, c, results) for c in chunks))
    return results

if __name__ == "__main__":
    sample = [f"https://target.example/item/{i}" for i in range(500)]
    t0 = time.time()
    out = asyncio.run(main(sample))
    ok = sum(1 for v in out.values() if v)
    print(f"{ok}/{len(sample)} ok in {time.time()-t0:.1f}s")

Key points:

n_workers is your dial. Start at 10–20 for residential and watch for 429s — that's your ceiling. Datacenter proxies tolerate far more; residential never does, because real residential IPs get rate-limited like real humans.
Backoff with jitter, not fixed sleeps. Synchronized retries across workers recreate the very burst that got you throttled.
total=25 bounds each request; without it, a single hung residential exit stalls a worker forever.

Failure mode 4: silent success (the one that ruins data)

The scariest failure is a 200 OK that isn't real content — a captcha, an "access denied" interstitial, or a stub page. status_code == 200 is not "I got the data."

Guard every response with a shape assertion, and on mismatch, rotate (don't stick — a captcha means this residential IP is now known):

def is_real_product_page(html: str) -> bool:
    # cheap structural probe; adapt to your target
    return ("data-price" in html) and ("captcha" not in html.lower())

def scrape_with_shape_retry(url, max_rotations=5):
    for i in range(max_rotations):
        r = requests.get(url, proxies=proxy_for(), timeout=(5, 20))  # rotating each try
        if r.status_code == 200 and is_real_product_page(r.text):
            return r.text
    return None  # this URL is poisoning your dataset — quarantine it

Quarantine failures to a side list and re-run them later at lower concurrency. A batch that keeps failing shape checks on rotating residential often needs a different geo — the pool in that region is simply "hot" (over-used) that hour.

Results and the坑 (pitfalls) I keep hitting

DNS leaks. Some proxies resolve DNS outside the tunnel, so the target sees your real resolver location even though the HTTP request came from a residential IP. Route DNS through the proxy or verify with a WebRTC/DNS leak test before trusting a run.
TLS fingerprinting. Even with a clean residential IP, a python-requests/2.x TLS handshake screams automation. If a site is aggressive, rotate through a client that mimics browser TLS (e.g. curl_cffi with impersonate="chrome") — the IP alone won't save you.
NTP drift on backoff is a non-issue, but retry budgets are: always cap total attempts per URL, or one flaky endpoint can starve your whole worker pool.
Pricing is usage-based per GB on residential, and captcha pages you accidentally download still burn bandwidth. The shape-check above is also a cost control.

For per-GB tiers and geo availability, check the current numbers on the residential pricing page rather than trusting any figure I'd quote — these change and I'd rather point you at the live table than write down a number that goes stale.

Putting it together

The mental model is small: rotate for breadth, stick for depth, bound concurrency, assert shape, back off with jitter, and never trust a 200. Get those five habits and a residential-proxy pipeline stops being a slot machine and starts being a data line.

If you want a residential pool that exposes exactly these two knobs (a rotating zone and a sticky-session token in the username) with per-GB billing and no forced minimums to experiment, that's the residential proxies I reach for, and the same gateway shape means the code above drops in with just your host and auth. New accounts usually get a small free allowance to validate a geo before committing — grab it from the site and run your first 500 URLs through the main() above to calibrate n_workers.

How to Reduce Proxy Costs Without Sacrificing Data Quality

Flora — Wed, 16 Sep 2026 09:00:10 +0000

How to Reduce Proxy Costs Without Sacrificing Data Quality

The cheapest proxy is not always the lowest-cost option. A failed request can create retry traffic, parser work, manual review and delayed data delivery.

Start by separating three metrics:

Cost per request
Cost per successful response
Cost per validated usable record

The third metric is the one that matters most for a production workflow. Measure it by proxy type, country, session mode and target category.

Use a bounded test with representative public URLs. Keep the parser and required fields constant, then compare residential, ISP, mobile and datacenter routes where appropriate. Track response time, retry rate, session breaks, missing fields and duplicate records.

Scale the route with the best usable-record economics, not simply the lowest advertised unit price. If a route starts producing incomplete results, reduce concurrency and review the failure category before increasing traffic.

Thordata provides residential, ISP, mobile and datacenter proxies for compliant ecommerce research, SEO monitoring, market research and localization testing. Limited free trial access can be used to validate the workflow before scaling.

Use proxy infrastructure responsibly. Do not use it for private-data collection, fake accounts, spam, click manipulation, credential testing or attempts to evade platform enforcement.

Start testing: https://www.thordata.com/?ls=dev&lk=DEV

How to Choose a Proxy for a Production Web Data Workflow

Flora — Wed, 16 Sep 2026 06:00:12 +0000

How to Choose a Proxy for a Production Web Data Workflow

Choosing a proxy by IP count or headline price is rarely enough. A production workflow should match the proxy type to the target, geography, session model and quality requirements.

Start with the workflow

Residential proxies can support geographic diversity and residential network context. ISP proxies can support stable, repeatable sessions. Mobile proxies provide mobile-network context. Datacenter proxies can be useful for fast and controlled workloads when the source and intended use allow them.

Measure usable results

Run a bounded test with representative public URLs. Record location accuracy, response time, session stability, retry rate, data completeness and cost per usable result. A request that returns HTTP 200 but misses the required fields is not a successful business result.

Use a controlled rollout

Start with a small workload, bounded concurrency and limited retries. Increase volume only after the route produces repeatable results. Keep separate failure categories for timeouts, empty results, parser failures and policy restrictions.

Test before scaling

Thordata offers residential, ISP, mobile and datacenter proxies for compliant web data workflows. A free trial is available for testing; check the current plan and usage terms before starting a production workload.

Use the trial to compare the proxy type, country and session mode that fit your actual workflow rather than choosing only by the advertised IP count.

Responsible use

Use proxy infrastructure for lawful workflows such as public ecommerce research, SEO monitoring, market research, advertising verification and localization testing. Do not use it for credential testing, fake accounts, click manipulation, spam, private-data collection or attempts to evade platform enforcement.

Start a Thordata trial: https://www.thordata.com/?ls=dev&lk=DEV

Datacenter proxies: what to measure before scaling

Flora — Wed, 16 Sep 2026 02:00:13 +0000

Datacenter proxies: a production evaluation guide

This guide explains how to evaluate the infrastructure before scaling a compliant public-web data project.

1. Match the proxy type to the workflow

Datacenter proxies are often a fast and cost-efficient option for large workloads. Measure success rate and retry cost alongside raw response speed.

The choice should be made from measured results, not from the product name alone.

2. Build a controlled test

Use a small, permitted set of public URLs that represents the real workload. Keep the test constant while changing only one variable at a time:

ext target URLs ? proxy type ? country ? session mode ? request rate ? result validation

3. Measure the metrics that affect cost

At minimum, calculate:

Success rate = successful usable results / total attempts
Median and P95 response time
Retry rate
Session break rate
Geographic accuracy rate
Cost per usable result

The last metric is usually more useful than cost per gigabyte. A low price is not efficient if failed requests create retries, manual review or incomplete downstream records.

4. Handle failure explicitly

A basic workflow should also include a circuit breaker: if the success rate for a location falls below an agreed threshold, pause that route and alert the operator instead of increasing traffic.

5. Scale gradually

Thordata reference: https://www.thordata.com/blog/ai-trends

6. Responsible use

Try Thordata: https://www.thordata.com/?ls=dev&lk=DEV

How to Reduce Proxy Costs Without Sacrificing Data Quality

Flora — Tue, 15 Sep 2026 09:39:34 +0000

How to Reduce Proxy Costs Without Sacrificing Data Quality

The cheapest proxy is not always the lowest-cost option. A failed request can create retry traffic, parser work, manual review and delayed data delivery.

Start by separating three metrics:

Cost per request
Cost per successful response
Cost per validated usable record

The third metric is the one that matters most for a production workflow. Measure it by proxy type, country, session mode and target category.

Use proxy infrastructure responsibly. Do not use it for private-data collection, fake accounts, spam, click manipulation, credential testing or attempts to evade platform enforcement.

Start testing: https://www.thordata.com/?ls=dev&lk=DEV

How to Choose a Proxy for a Production Web Data Workflow

Flora — Tue, 15 Sep 2026 06:00:12 +0000

How to Choose a Proxy for a Production Web Data Workflow

Choosing a proxy by IP count or headline price is rarely enough. A production workflow should match the proxy type to the target, geography, session model and quality requirements.

Start with the workflow

Measure usable results

Use a controlled rollout

Test before scaling

Use the trial to compare the proxy type, country and session mode that fit your actual workflow rather than choosing only by the advertised IP count.

Responsible use

Start a Thordata trial: https://www.thordata.com/?ls=dev&lk=DEV

Mobile proxies: what to measure before scaling

Flora — Tue, 15 Sep 2026 02:00:12 +0000

Mobile proxies: a production evaluation guide

This guide explains how to evaluate the infrastructure before scaling a compliant public-web data project.

1. Match the proxy type to the workflow

Mobile proxies fit workflows that depend on real 4G/5G network context. Validate location, session behavior and data quality rather than judging by IP count alone.

The choice should be made from measured results, not from the product name alone.

2. Build a controlled test

Use a small, permitted set of public URLs that represents the real workload. Keep the test constant while changing only one variable at a time:

ext target URLs ? proxy type ? country ? session mode ? request rate ? result validation

3. Measure the metrics that affect cost

At minimum, calculate:

Success rate = successful usable results / total attempts
Median and P95 response time
Retry rate
Session break rate
Geographic accuracy rate
Cost per usable result

The last metric is usually more useful than cost per gigabyte. A low price is not efficient if failed requests create retries, manual review or incomplete downstream records.

4. Handle failure explicitly

A basic workflow should also include a circuit breaker: if the success rate for a location falls below an agreed threshold, pause that route and alert the operator instead of increasing traffic.

5. Scale gradually

Thordata reference: https://www.thordata.com/blog/ai-trends

6. Responsible use

Try Thordata: https://www.thordata.com/?ls=dev&lk=DEV

How to Reduce Proxy Costs Without Sacrificing Data Quality

Flora — Mon, 14 Sep 2026 09:00:14 +0000

How to Reduce Proxy Costs Without Sacrificing Data Quality

The cheapest proxy is not always the lowest-cost option. A failed request can create retry traffic, parser work, manual review and delayed data delivery.

Start by separating three metrics:

Cost per request
Cost per successful response
Cost per validated usable record

The third metric is the one that matters most for a production workflow. Measure it by proxy type, country, session mode and target category.

Use proxy infrastructure responsibly. Do not use it for private-data collection, fake accounts, spam, click manipulation, credential testing or attempts to evade platform enforcement.

Start testing: https://www.thordata.com/?ls=dev&lk=DEV

DEV Community: Flora

The Hard Part of Video Datasets Isn't the Scraper: Reconciling YouTube, TikTok, and Shorts Into One Schema

Why three "video" objects don't fit

The shape of a normalized row

A normalize function that survives review

Things that break in week three

Buy or scrape?

Beyond the Exit IP: How WebRTC and DNS Undo Your Residential Proxy (and How to Verify It)

What the proxy does not govern

The blunt fix: force WebRTC through the proxy

Do not trust the fix 鈥?measure it

Why the override trick is the tempting wrong answer

The part I would tell you in person

That "Residential" IP Might Be a Datacenter in Disguise - A Scriptable Proxy Audit

The scenario nobody plans for

Why the distinction matters mechanically

The three independent signals

The audit script

Reading the results

Pools lie in both directions

Residential proxies: what to measure before scaling

Residential proxies: a production evaluation guide

1. Match the proxy type to the workflow

2. Build a controlled test

3. Measure the metrics that affect cost

4. Handle failure explicitly

5. Scale gradually

6. Responsible use

Rotating Residential Proxies Without Losing Your Mind: 407 Auth, Sticky Sessions, and Backoff in Practice

The problem nobody tells you about

How residential rotation actually works

Failure mode 1: 407 Proxy Authentication Required

Failure mode 2: sticky sessions done right

Failure mode 3: concurrency and the collision problem

Failure mode 4: silent success (the one that ruins data)

Results and the坑 (pitfalls) I keep hitting

Putting it together

How to Reduce Proxy Costs Without Sacrificing Data Quality

How to Reduce Proxy Costs Without Sacrificing Data Quality

How to Choose a Proxy for a Production Web Data Workflow

How to Choose a Proxy for a Production Web Data Workflow

Start with the workflow

Measure usable results

Use a controlled rollout

Test before scaling

Responsible use

Datacenter proxies: what to measure before scaling

Datacenter proxies: a production evaluation guide

1. Match the proxy type to the workflow

2. Build a controlled test

3. Measure the metrics that affect cost

4. Handle failure explicitly

5. Scale gradually

6. Responsible use

How to Reduce Proxy Costs Without Sacrificing Data Quality

How to Reduce Proxy Costs Without Sacrificing Data Quality

How to Choose a Proxy for a Production Web Data Workflow

How to Choose a Proxy for a Production Web Data Workflow

Start with the workflow

Measure usable results

Use a controlled rollout

Test before scaling

Responsible use

Mobile proxies: what to measure before scaling

Mobile proxies: a production evaluation guide

1. Match the proxy type to the workflow

2. Build a controlled test

3. Measure the metrics that affect cost

4. Handle failure explicitly

5. Scale gradually

6. Responsible use

How to Reduce Proxy Costs Without Sacrificing Data Quality

How to Reduce Proxy Costs Without Sacrificing Data Quality

Failure mode 1: `407 Proxy Authentication Required`