DEV Community: Xavier Fok

IP Geolocation Isn't Enough for Regional App Testing

Xavier Fok — Tue, 22 Sep 2026 08:38:08 +0000

You point your test at a proxy in the right country, the app store shows the right storefront, and you ship. Then support tickets come in from real users on actual carrier networks seeing a different payment flow than what you tested.

I run a mobile proxy network on real SIM cards in Singapore, and this came out of running it. The gap usually isn't the IP's country. It's the IP's ASN classification. Platforms that do regional testing at scale check whether an IP belongs to a "hosting," "residential," or "mobile" range, and a VPN or datacenter exit almost always lands in "hosting" no matter what country it geolocates to.

Here's a script that checks that classification directly instead of trusting the country field alone:

import os
import requests

PROXY_URL = os.environ["PROXY_URL"]  # e.g. socks5h://user:pass@host:port

session = requests.Session()
session.proxies.update({"http": PROXY_URL, "https": PROXY_URL})
session.headers.update({"Accept-Language": "en-SG,en;q=0.9"})

ip_data = session.get("https://ipapi.co/json/", timeout=15).json()
print("IP:", ip_data.get("ip"))
print("Org:", ip_data.get("org"))
print("Mobile:", ip_data.get("mobile", "not reported"))

the country check passes, the classification doesn't

When your app makes a request through a proxy, the server on the other end doesn't just read the IP's country. It reads the IP's ASN, and IP intelligence providers classify most ASN blocks as roughly "hosting," "residential," or "mobile." A VPN exit or a datacenter proxy in the right country still resolves to a "hosting" ASN, because that's genuinely what it is. A consumer SIM issued by an actual carrier resolves to "mobile," because that's genuinely what it is too.

This matters because platforms that maintain IP reputation scoring (which by now includes most app stores, ad networks, and payment processors) use that classification as a signal independent of the country field. Your test session can show the correct storefront and still get routed through a different code path than a real subscriber would see, because the country check passed but the carrier check didn't. You won't notice in testing. You'll notice when a support ticket describes a payment flow you never saw.

reading the ASN instead of the flag

The script above checks org and mobile from an IP intelligence API. That catches the obvious case: a datacenter IP shows a hosting company in org and nothing in mobile, a carrier IP shows the carrier's registered ASN name and mobile: true. A few fields worth reading closely in the response:

org: the ASN's registered organization; you want a carrier name, not a hosting company
mobile: a boolean some providers report separately from org, worth checking even when org looks right
country: the field everyone checks first and the one that matters least here

If you need to go further and confirm which carrier specifically, not just mobile-or-not, checking a proxy's ASN before you trust it covers the WHOIS lookup in more detail than fits here.

don't stop at the storefront

A clean ASN doesn't guarantee the rest of the flow behaves correctly. Extend the same session to check what you actually care about:

# continues using `session` from above

store_resp = session.get(
    "https://play.google.com/store/apps",
    timeout=15,
    allow_redirects=True,
)
print("store url:", store_resp.url)

cdn_resp = session.get(os.environ["CDN_URL"], timeout=15)
geo_headers = ["cf-ipcountry", "x-amz-cf-pop", "x-geo", "x-country"]
for key, value in cdn_resp.headers.items():
    if key.lower() in geo_headers:
        print(f"{key}: {value}")

The storefront URL tells you which region the app store resolved you into. The CDN headers tell you which edge and geo-routing bucket actually served the response, which matters if your app pulls region-specific assets or feature flags from a CDN rather than from the store itself.

For flows that involve login, cart, or checkout, keep the same session and the same proxy connection for every step. A proxy that rotates IP on every new connection hands you a different IP mid-flow, breaks session cookies, and sometimes trips fraud checks that a real single-session user would never hit.

the DNS leak that undoes all of it

Worth checking explicitly: if you're using SOCKS5, make sure DNS resolution happens through the proxy, not locally. In requests, that's the difference between socks5h:// and socks5://: the h routes the hostname lookup through the tunnel. Without it, your traffic exits from the right IP but the DNS query that preceded it left from your real location. Some platforms cross-reference the two and flag the mismatch even when the HTTP request itself looks clean.

what actually bit us running this on real hardware

Real carrier IPs sit behind carrier-grade NAT, and carriers reassign addresses on their own schedule as part of normal CGNAT operation. That's standard, documented behavior, not a fault in any particular proxy. The practical effect: a sticky session that's supposed to hold one IP for the length of a test can still get reassigned mid-flow if the test runs long enough, because the reassignment happens at the carrier level, underneath the proxy layer entirely. We've had multi-hour automated test runs lose their IP partway through for exactly this reason. If you're running anything longer than a few minutes against a mobile carrier IP, build in a check that confirms the IP hasn't changed before you trust a multi-step result.

the tradeoff you're actually making

Datacenter proxies are faster. Sub-100ms round trips are normal. A real SIM going through an actual carrier network typically lands somewhere between 80 and 350ms depending on signal conditions at the moment you test. For geo-testing, that difference almost never matters, because you're checking correctness of a code path, not throughput. For a high-volume automated pipeline making thousands of requests an hour, it might, and that's worth weighing against how accurately the test needs to reflect what a real subscriber sees.

If your platform makes any decision (payment method, feature flag, content variant) based on whether a connection looks like a real mobile subscriber, testing that decision from a hosting-classified IP will tell you the storefront works and nothing about whether the decision itself is correct.

The Future of Proxy Technology: Trends Shaping 2026 and Beyond

Xavier Fok — Sun, 08 Mar 2026 16:25:26 +0000

The proxy industry is evolving rapidly. Detection systems are getting smarter, new proxy technologies are emerging, and the cat-and-mouse game between operators and platforms continues to intensify. Here are the trends defining the next era of proxy technology.

Trend 1: AI-Powered Bot Detection

Platforms are deploying machine learning models that analyze user behavior in real-time:

Behavioral biometrics — ML models learn individual user patterns (typing rhythm, mouse movement curves, scroll velocity)
Anomaly detection — AI flags behavior that deviates from baseline user patterns
Cross-session profiling — ML correlates behavior across sessions to identify automated accounts
Real-time scoring — Every action updates a trust score that determines whether to challenge or block

Impact on operators:
Simple randomized delays are no longer sufficient. Automation must produce genuinely human-like behavioral patterns that pass ML scrutiny.

Trend 2: HTTP/2 and HTTP/3 Fingerprinting

Beyond TLS fingerprinting, platforms now analyze HTTP/2 and HTTP/3 connection characteristics:

SETTINGS frame values — Each browser sends different initial settings
Priority/weight parameters — How resources are prioritized differs by browser
Header order — HTTP/2 pseudo-headers appear in browser-specific orders
Flow control — Window sizes and flow control behavior vary by implementation

This creates another layer of fingerprinting that most automation tools cannot currently spoof.

Trend 3: 5G Mobile Proxies

5G networks are changing the mobile proxy landscape:

Faster speeds — 5G proxies match or exceed residential proxy speeds
Lower latency — Sub-10ms latency for nearby 5G towers
New IP ranges — Fresh IP pools as carriers deploy 5G infrastructure
Network slicing — Carriers can identify proxy usage through network slice analysis

Opportunity: Early adopters of 5G mobile proxies benefit from completely fresh IP pools.

Trend 4: Browser-as-a-Service

Cloud-based browser services are replacing local anti-detect browser installations:

No local fingerprint leaks — Browser runs in the cloud, not on your machine
Scalable — Spin up hundreds of browser instances on demand
API-driven — Full programmatic control over browser profiles
Team collaboration — Shared browser profiles accessible from anywhere

Services offering cloud browsers with built-in proxy integration are becoming the new standard for multi-account operations.

Trend 5: Residential Proxy Quality Crisis

The residential proxy market faces a quality problem:

Oversaturation — Too many users sharing the same IP pools
Detection improvements — Platforms can identify residential proxy networks through behavioral analysis
Ethical concerns — Some providers source IPs through questionable SDK bundling
Pool degradation — Popular pools get burned faster as more operators use them

Result: Premium, ethically-sourced residential proxies command higher prices while budget options become less effective.

Trend 6: Decentralized Proxy Networks

Blockchain-based proxy networks are emerging:

Peer-to-peer — Users share bandwidth for tokens/cryptocurrency
No central authority — Harder for platforms to block entire networks
Economic incentives — Users earn by sharing their residential IPs
Privacy by design — Decentralized architecture limits data collection

Still early stage, but worth watching as the technology matures.

Trend 7: Advanced Fingerprint Resistance

Browsers themselves are adding fingerprinting resistance:

Firefox — Enhanced Tracking Protection includes fingerprint blocking
Safari — Intelligent Tracking Prevention reduces fingerprintable surface
Chrome — Privacy Sandbox aims to limit cross-site tracking

As browsers reduce fingerprinting surface area, the playing field between automated and genuine browsers levels. This benefits operators whose automation uses real browser engines.

Trend 8: API-First Proxy Management

Proxy providers are shifting from dashboard-centric to API-first approaches:

Infrastructure as Code — Define proxy configurations in version-controlled files
Programmatic everything — Create, manage, and monitor proxies entirely through APIs
Integration ecosystems — Native integrations with automation tools, CI/CD pipelines, and monitoring systems
Self-service scaling — Automatically scale proxy resources based on demand

Trend 9: Compliance and Regulation

The regulatory landscape is tightening:

Data privacy laws expanding globally (GDPR influence spreading)
Anti-bot legislation being discussed in several jurisdictions
Platform liability for proxy-enabled activities increasing
KYC requirements for proxy providers becoming more common

Operators need to stay informed about legal developments in their operating jurisdictions.

Trend 10: Convergence of VPN and Proxy Markets

The line between VPN and proxy services is blurring:

VPN providers adding residential IP options
Proxy providers adding encryption and privacy features
Hybrid products offering both VPN tunneling and proxy rotation
Consumer VPN infrastructure being repurposed for commercial proxy use

What This Means for Operators

Invest in quality over quantity — Cheap, bulk proxies are becoming less effective
Stay current with detection methods — What worked last year may not work today
Use real browser engines — Library-based approaches face increasing detection
Diversify providers — No single provider will remain optimal indefinitely
Build adaptable systems — Infrastructure that can quickly adapt to new detection methods
Consider compliance early — Regulatory requirements are only increasing
Monitor industry developments — The proxy landscape changes rapidly

For the latest proxy technology analysis and industry insights, visit DataResearchTools.

Residential Proxy Providers: What to Look For When Choosing in 2026

Xavier Fok — Sun, 08 Mar 2026 16:24:54 +0000

The residential proxy market is crowded with providers making similar claims — millions of IPs, global coverage, 99.9% uptime. How do you separate the genuine from the marketing fluff? Here is a practical evaluation framework.

The Evaluation Framework

1. Pool Size and Quality

What providers claim: "50 million residential IPs worldwide"

What actually matters:

How many IPs are available right now, not total ever seen
How many IPs are in your target geography
What percentage are currently active and not blacklisted
How often are new IPs added to the pool

How to verify:

Request a trial and run 1000 requests, logging unique IPs
Check IP blacklist status on multiple databases
Test during different times of day (pool size varies)

2. Geographic Coverage

Questions to ask:

Which countries have the most IPs?
Is city-level targeting available?
How deep is the pool in each city?
Are ASN and ISP targeting available?

Why it matters:
A provider with 10 million IPs concentrated in 5 countries is useless if you need IPs in Southeast Asia.

3. Success Rate on Your Target

This is the most important metric and the hardest to evaluate from marketing materials.

def test_success_rate(provider_proxy, target_urls, num_requests=100):
    successes = 0
    failures = {}

    for i in range(num_requests):
        url = random.choice(target_urls)
        try:
            response = requests.get(
                url,
                proxies={"http": provider_proxy, "https": provider_proxy},
                timeout=15
            )
            if response.status_code == 200:
                successes += 1
            else:
                status = str(response.status_code)
                failures[status] = failures.get(status, 0) + 1
        except Exception as e:
            error_type = type(e).__name__
            failures[error_type] = failures.get(error_type, 0) + 1

    print(f"Success rate: {successes}%")
    print(f"Failures: {failures}")

Always test against YOUR target sites, not generic test endpoints.

4. Speed and Latency

What to measure:

Average response time
P95 response time (95th percentile)
Connection time vs data transfer time
Speed variance throughout the day

Acceptable ranges:

Use Case	Acceptable Latency
Web scraping	Under 3 seconds
Account management	Under 5 seconds
Real-time monitoring	Under 1 second

5. Session Support

Key questions:

What is the maximum sticky session duration?
How reliable are sticky sessions (does the IP actually stay the same)?
Can you control session duration?
What happens when a sticky session expires?

Testing sticky sessions:

def test_sticky_session(proxy, session_id, num_checks=10, delay=60):
    ips_seen = []

    for i in range(num_checks):
        proxy_url = f"http://user-session_{session_id}:pass@gateway:port"
        response = requests.get(
            "https://httpbin.org/ip",
            proxies={"http": proxy_url, "https": proxy_url},
            timeout=10
        )
        ip = response.json()["origin"]
        ips_seen.append(ip)
        print(f"Check {i+1}: {ip}")
        time.sleep(delay)

    unique_ips = len(set(ips_seen))
    print(f"Unique IPs over {num_checks} checks: {unique_ips}")
    print(f"Session stable: {unique_ips == 1}")

6. Pricing Model

Common models:

Model	Best For	Typical Cost
Pay-per-GB	Variable usage	$5-15/GB
Monthly subscription	Predictable usage	$50-500/month
Pay-per-request	API-heavy operations	$1-5 per 1000 requests
Pay-per-IP	Dedicated IPs	$2-10/IP/month

Watch for:

Bandwidth counting method (response only vs request + response)
Minimum commitment periods
Overage charges
Hidden fees for premium features (geo-targeting, sticky sessions)

7. API and Integration

Must-have features:

Well-documented API
Usage monitoring endpoints
Session management API
Multiple authentication methods
Code examples in major languages

Nice-to-have:

Webhook notifications
Real-time usage dashboard
IP quality scoring
Automatic IP rotation configuration

8. Support and Reliability

Response time for support tickets
Live chat availability
Technical documentation quality
Status page with incident history
SLA guarantees

Red Flags

No free trial — They do not want you to test before paying
No refund policy — Signals low confidence in their product
Unrealistic claims — "100% success rate" or "never blocked"
No usage dashboard — You cannot monitor what you cannot measure
Poor documentation — Indicates immature or unreliable infrastructure
Long-term contracts only — Locking you in before you can evaluate
No subnet diversity info — All IPs might come from the same ASN

Evaluation Checklist

[ ] Tested success rate on target sites (aim for 95%+)
[ ] Verified IP pool depth in target geographies
[ ] Measured latency during peak and off-peak hours
[ ] Tested sticky session reliability
[ ] Reviewed pricing model and calculated projected costs
[ ] Evaluated API documentation and integration ease
[ ] Checked support responsiveness
[ ] Read independent reviews from operators in your niche
[ ] Compared at least 3 providers
[ ] Verified no hidden fees or gotchas in terms of service

For residential proxy provider comparisons and testing guides, visit DataResearchTools.

Proxy Failover: Building Resilient Infrastructure That Never Goes Down

Xavier Fok — Sun, 08 Mar 2026 16:24:19 +0000

Your proxy provider goes down at 3 AM. Your scraping pipeline stops. Your accounts cannot log in. Your monitoring goes blind. Single points of failure in proxy infrastructure are unacceptable for serious operations. Here is how to build resilient systems.

Common Failure Modes

Provider-Level Failures

Complete outage — Provider gateway goes offline
Partial degradation — Slow responses, increased error rates
Pool exhaustion — All IPs in your target geo are used or blocked
Authentication issues — API key expired, billing problems

Network-Level Failures

DNS resolution failure — Cannot resolve proxy gateway hostname
Connection timeouts — Network path to proxy is congested
SSL/TLS errors — Certificate issues on proxy gateway

IP-Level Failures

IP burned — Target platform blocked the proxy IP
IP misclassified — Proxy IP detected as datacenter instead of residential
Geographic mismatch — IP geolocates to wrong location

Failover Architecture

Your Application
        |
        v
   Proxy Router
   /     |     \\
Provider Provider Provider
   A       B       C
(Primary) (Secondary) (Tertiary)

Multi-Provider Setup

class ResilientProxyManager:
    def __init__(self):
        self.providers = [
            {
                "name": "primary",
                "gateway": "gateway.provider-a.com:8080",
                "auth": "user_a:pass_a",
                "priority": 1,
                "healthy": True,
                "consecutive_failures": 0
            },
            {
                "name": "secondary",
                "gateway": "gateway.provider-b.com:8080",
                "auth": "user_b:pass_b",
                "priority": 2,
                "healthy": True,
                "consecutive_failures": 0
            },
            {
                "name": "tertiary",
                "gateway": "gateway.provider-c.com:8080",
                "auth": "user_c:pass_c",
                "priority": 3,
                "healthy": True,
                "consecutive_failures": 0
            }
        ]

    def get_proxy(self):
        # Sort by priority, filter healthy
        available = sorted(
            [p for p in self.providers if p["healthy"]],
            key=lambda p: p["priority"]
        )

        if not available:
            # All providers down - reset and try again
            self.reset_all()
            available = self.providers

        provider = available[0]
        return f"http://{provider["auth"]}@{provider["gateway"]}"

    def report_failure(self, provider_name):
        provider = self.get_provider(provider_name)
        provider["consecutive_failures"] += 1

        if provider["consecutive_failures"] >= 3:
            provider["healthy"] = False
            self.schedule_health_check(provider, delay=60)

    def report_success(self, provider_name):
        provider = self.get_provider(provider_name)
        provider["consecutive_failures"] = 0
        provider["healthy"] = True

Circuit Breaker Pattern

The circuit breaker prevents hammering a failed provider:

import time

class CircuitBreaker:
    CLOSED = "closed"      # Normal operation
    OPEN = "open"          # Provider failed, blocking requests
    HALF_OPEN = "half_open"  # Testing if provider recovered

    def __init__(self, failure_threshold=5, recovery_timeout=60):
        self.state = self.CLOSED
        self.failures = 0
        self.threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.last_failure_time = 0

    def can_execute(self):
        if self.state == self.CLOSED:
            return True

        if self.state == self.OPEN:
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = self.HALF_OPEN
                return True
            return False

        if self.state == self.HALF_OPEN:
            return True

        return False

    def record_success(self):
        self.failures = 0
        self.state = self.CLOSED

    def record_failure(self):
        self.failures += 1
        self.last_failure_time = time.time()

        if self.failures >= self.threshold:
            self.state = self.OPEN

Health Check System

import threading

class HealthChecker:
    def __init__(self, proxy_manager, check_interval=30):
        self.proxy_manager = proxy_manager
        self.interval = check_interval
        self.running = True

    def start(self):
        thread = threading.Thread(target=self.run, daemon=True)
        thread.start()

    def run(self):
        while self.running:
            for provider in self.proxy_manager.providers:
                self.check_provider(provider)
            time.sleep(self.interval)

    def check_provider(self, provider):
        proxy = f"http://{provider["auth"]}@{provider["gateway"]}"

        try:
            start = time.time()
            response = requests.get(
                "https://httpbin.org/ip",
                proxies={"http": proxy, "https": proxy},
                timeout=10
            )
            latency = time.time() - start

            if response.status_code == 200:
                provider["healthy"] = True
                provider["latency"] = latency
                provider["consecutive_failures"] = 0
            else:
                self.handle_check_failure(provider)

        except Exception:
            self.handle_check_failure(provider)

    def handle_check_failure(self, provider):
        provider["consecutive_failures"] += 1
        if provider["consecutive_failures"] >= 3:
            provider["healthy"] = False
            send_alert(f"Provider {provider["name"]} is DOWN")

Failover Best Practices

Use at least 2 providers — Single provider is single point of failure
Test failover regularly — Simulate provider failures to verify your system works
Set up monitoring — Know about failures before they impact operations
Maintain warm standby — Secondary providers should have active credentials and tested connectivity
Document provider SLAs — Know what uptime each provider guarantees
Budget for redundancy — Failover providers cost money even when idle
Automate recovery — Manual failover at 3 AM is not reliable

Cost of Downtime vs Cost of Redundancy

Metric	Single Provider	Multi-Provider
Monthly proxy cost	$200	$250-300
Expected uptime	99%	99.9%+
Monthly downtime	~7 hours	~43 minutes
Impact of outage	Total stop	Seamless failover

The extra 25-50% cost for redundancy prevents hours of downtime per month.

For proxy failover architecture and infrastructure resilience guides, visit DataResearchTools.

TLS Fingerprinting: The Bot Detection Method You Cannot Ignore

Xavier Fok — Sun, 08 Mar 2026 16:23:19 +0000

You have the right proxy, the right browser fingerprint, and human-like behavior. But you are still getting blocked. The culprit? TLS fingerprinting — a detection method that works at the network level, before your JavaScript even loads.

What Is TLS Fingerprinting?

When your browser connects to a website over HTTPS, it performs a TLS handshake. The very first message in this handshake — the Client Hello — contains a wealth of identifying information:

Cipher suites — Which encryption methods your client supports
TLS extensions — Additional features like SNI, ALPN, signature algorithms
Elliptic curves — Which curves your client supports
Extension order — The sequence in which extensions appear
Compression methods — Supported compression algorithms

JA3 and JA4 Fingerprints

JA3

The original TLS fingerprinting method. It hashes the Client Hello parameters into a 32-character MD5 hash:

JA3 = MD5(
    TLSVersion,
    Ciphers,
    Extensions,
    EllipticCurves,
    EllipticCurvePointFormats
)

Example JA3 hashes:

Chrome 121 on Windows:  cd08e31494f9531f560d64c695473da9
Firefox 121 on Windows: e4bb02e2d9e1e3e0e5c84899b8a949c9
Python requests:        b32309a26951912be7dba376398abc3b
curl:                   456523fc94726331a8d0515e28e1a8e7

Notice how Python requests and curl have completely different fingerprints from real browsers. This is a dead giveaway.

JA4

The next generation, providing more detailed fingerprinting:

JA4 = Protocol + Version + SNI + CipherCount + ExtensionCount + ALPN +
      Truncated SHA256(Ciphers) + Truncated SHA256(Extensions + SigAlgs)

JA4 is harder to spoof because it captures more handshake details.

Why This Matters for Proxy Users

Consider this scenario:

Your anti-detect browser claims to be Chrome 121 on Windows
Your proxy provides a clean residential IP
Your browser fingerprint (Canvas, WebGL) is perfect
BUT your automation tool uses Python requests, which has a completely different TLS fingerprint

The server sees a Client Hello from Python, not Chrome. Instant detection.

Expected:  Chrome TLS fingerprint + Chrome User-Agent
Actual:    Python TLS fingerprint + Chrome User-Agent
Result:    BLOCKED

Detection in the Wild

Major platforms and CDNs using TLS fingerprinting:

Cloudflare — Compares JA3 to known bot signatures
Akamai — Uses TLS fingerprinting in Bot Manager
PerimeterX — Cross-references TLS with browser fingerprint
DataDome — Real-time TLS analysis
Google — Uses TLS signals in bot detection

Solutions

Solution 1: Use Real Browsers

The most reliable approach. Puppeteer and Playwright control real browser engines, so TLS fingerprints match genuine browsers.

from playwright.sync_api import sync_playwright

# Playwright uses a real Chromium engine
# TLS fingerprint matches Chrome
with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://target.com")  # Real Chrome TLS fingerprint

Solution 2: TLS-Spoofing Libraries

For HTTP-level requests without full browser overhead:

curl_cffi (Python):

from curl_cffi import requests

# Impersonate Chrome TLS fingerprint
response = requests.get(
    "https://target.com",
    impersonate="chrome121",
    proxies={"https": "http://user:pass@proxy:port"}
)

tls-client (Python):

import tls_client

session = tls_client.Session(
    client_identifier="chrome_121",
    random_tls_extension_order=True
)

response = session.get(
    "https://target.com",
    proxy="http://user:pass@proxy:port"
)

Solution 3: Custom TLS Configuration

For advanced users, configure TLS parameters manually:

import ssl
import urllib3

# Create custom SSL context matching Chrome
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.set_ciphers(
    "TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:"
    "TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256"
)
ctx.minimum_version = ssl.TLSVersion.TLSv1_2

Testing Your TLS Fingerprint

Verify your fingerprint before deploying:

# Check your JA3 fingerprint
response = requests.get("https://ja3er.com/json")
print(response.json())
# Compare against known browser fingerprints

Best Practices

Match TLS to User-Agent — If claiming Chrome, use Chrome TLS fingerprint
Use browser engines for account operations — Never use raw HTTP libraries for logged-in sessions
Test fingerprints regularly — Browser updates change TLS fingerprints
Use TLS-spoofing libraries for scraping — curl_cffi is excellent for high-volume scraping
Monitor for new detection methods — JA4 and HTTP/2 fingerprinting are evolving

For TLS fingerprinting guides and bot detection evasion strategies, visit DataResearchTools.

Proxy Bandwidth Optimization: Cut Costs Without Sacrificing Performance

Xavier Fok — Sun, 08 Mar 2026 16:22:36 +0000

Residential and mobile proxy bandwidth is expensive — $5-50 per GB. Every wasted byte is wasted money. Here are practical techniques to dramatically reduce your proxy bandwidth consumption.

Where Bandwidth Gets Wasted

1. Downloading Full Pages When You Need One Data Point

A typical web page is 2-5 MB. If you only need a price or title, you are wasting 99% of the bandwidth on images, CSS, JavaScript, and ads.

2. Loading Resources You Do Not Need

Images, videos, fonts, tracking scripts, and ads consume the bulk of page weight but are rarely needed for data extraction.

3. Redundant Requests

Scraping the same page multiple times because of poor caching or retry logic.

4. Failed Requests

Requests that fail (403, CAPTCHA, timeout) still consume bandwidth.

Optimization Techniques

1. Block Unnecessary Resources

In Playwright or Puppeteer, intercept and block heavy resources:

from playwright.sync_api import sync_playwright

def create_optimized_page(browser):
    page = browser.new_page()

    # Block images, fonts, stylesheets, media
    page.route("**/*.{png,jpg,jpeg,gif,svg,webp}", lambda route: route.abort())
    page.route("**/*.{woff,woff2,ttf,eot}", lambda route: route.abort())
    page.route("**/*.css", lambda route: route.abort())
    page.route("**/analytics*", lambda route: route.abort())
    page.route("**/tracking*", lambda route: route.abort())
    page.route("**/ads*", lambda route: route.abort())

    return page

This alone can reduce bandwidth by 60-80%.

2. Use HTTP Instead of Headless Browsers

Headless browsers download and render everything. Direct HTTP requests download only the HTML.

import requests

# Headless browser: Downloads 3-5 MB per page
# Direct HTTP: Downloads 50-200 KB per page

response = requests.get(
    url,
    proxies=proxy,
    headers={"Accept-Encoding": "gzip, deflate, br"},  # Enable compression
    timeout=15
)

Use headless browsers only when JavaScript rendering is required.

3. Enable Compression

Always request compressed responses:

headers = {
    "Accept-Encoding": "gzip, deflate, br",
    # Server will send compressed response
    # requests library decompresses automatically
}

Compression typically reduces HTML payload by 70-80%.

4. Use APIs Instead of Scraping

Many sites have APIs (official or unofficial) that return structured JSON — much smaller than full HTML pages.

# Scraping HTML: ~200 KB per product
response = requests.get("https://site.com/product/123")

# Using API: ~2 KB per product (100x smaller)
response = requests.get("https://api.site.com/products/123")

5. Implement Smart Caching

import hashlib
import json
import time

class ProxyCache:
    def __init__(self, cache_ttl=3600):
        self.cache = {}
        self.ttl = cache_ttl

    def get(self, url):
        key = hashlib.md5(url.encode()).hexdigest()
        if key in self.cache:
            entry = self.cache[key]
            if time.time() - entry["timestamp"] < self.ttl:
                return entry["data"]  # Cache hit - zero bandwidth
        return None

    def set(self, url, data):
        key = hashlib.md5(url.encode()).hexdigest()
        self.cache[key] = {
            "data": data,
            "timestamp": time.time()
        }

6. Conditional Requests

Use HTTP conditional headers to avoid downloading unchanged content:

# First request
response = requests.get(url, proxies=proxy)
etag = response.headers.get("ETag")
last_modified = response.headers.get("Last-Modified")

# Subsequent requests
headers = {}
if etag:
    headers["If-None-Match"] = etag
if last_modified:
    headers["If-Modified-Since"] = last_modified

response = requests.get(url, proxies=proxy, headers=headers)
if response.status_code == 304:
    # Content unchanged - minimal bandwidth used
    pass

7. Minimize Retry Bandwidth

When retrying failed requests, do not retry immediately on the same proxy:

def smart_retry(url, proxy_manager, max_retries=3):
    for attempt in range(max_retries):
        proxy = proxy_manager.get_fresh_proxy()  # Different proxy each time
        try:
            response = requests.get(url, proxies=proxy, timeout=10)
            if response.status_code == 200:
                return response
            elif response.status_code in [403, 429]:
                proxy_manager.mark_failed(proxy)
                continue  # Try different proxy
        except requests.Timeout:
            proxy_manager.mark_slow(proxy)
            continue
    return None

Bandwidth Savings Summary

Technique	Bandwidth Reduction
Block images/media	60-80%
HTTP vs headless browser	90-95%
Enable compression	70-80%
Use APIs vs scraping	95-99%
Caching	100% on cache hits
Conditional requests	95%+ on unchanged content

Cost Impact Example

Scraping 10,000 pages per day:

Method	Per-Page Size	Daily Bandwidth	Monthly Cost ($10/GB)
Headless, no optimization	3 MB	30 GB	$300
Headless, blocked resources	500 KB	5 GB	$50
Direct HTTP, compressed	50 KB	500 MB	$5
API requests	2 KB	20 MB	$0.20

Optimization can reduce your proxy costs by 99%.

For proxy optimization guides and cost-saving strategies, visit DataResearchTools.

How Ad Verification Companies Use Proxies at Scale

Xavier Fok — Sun, 08 Mar 2026 16:22:07 +0000

Ad verification is a multi-billion dollar industry, and proxies are its backbone. Companies like DoubleVerify, IAS, and MOAT use massive proxy networks to verify ads are displayed correctly worldwide. Here is how the industry works and what you can learn from it.

What Is Ad Verification?

Ad verification ensures that digital advertisements:

Display correctly — Right creative, right format, right placement
Reach the right audience — Proper geographic and demographic targeting
Appear in brand-safe environments — Not next to inappropriate content
Are viewable — Actually visible to users, not hidden below the fold
Are not fraudulent — Real impressions from real users, not bots

Why Ad Verification Needs Proxies

Geographic Verification

An advertiser running campaigns in 50 countries needs to verify ads display correctly in each location. You cannot check a geo-targeted ad in Brazil from an office in New York.

Proxies provide:

Exit IPs in every target country
City-level targeting for local ad campaigns
Mobile IPs to verify mobile ad delivery

Publisher Compliance

Publishers sometimes:

Show different ads to verification bots vs real users
Place ads in non-viewable positions
Stack multiple ads in the same slot
Display ads next to prohibited content

Residential and mobile proxies make verification requests indistinguishable from real user traffic.

Fraud Detection

Ad fraud costs the industry billions annually. Proxies help detect:

Click farms — Coordinated fake clicks from suspicious IP ranges
Domain spoofing — Fake sites pretending to be premium publishers
Ad stacking — Multiple ads layered in a single ad slot
Pixel stuffing — Ads rendered in 1x1 pixel frames

Ad Verification Proxy Architecture

Ad Campaign Dashboard
        |
        v
Verification Engine
        |
        v
Geo-Distributed Proxy Network
   /    |    |    \\
  US    EU   Asia  LATAM
  |     |    |     |
  v     v    v     v
Publisher Websites
        |
        v
Ad Rendering Analysis
        |
        v
Report Generation

Scale of Operations

Major ad verification companies operate at staggering scale:

Millions of proxy IPs across 100+ countries
Billions of ad impressions verified monthly
Real-time verification within milliseconds of ad display
Continuous monitoring 24/7/365

Techniques Used

1. Headless Browser Verification

Render pages in headless browsers through proxies to see exactly what users see:

from playwright import sync_playwright

def verify_ad(url, proxy, expected_ad):
    with sync_playwright() as p:
        browser = p.chromium.launch(
            proxy={"server": proxy}
        )
        page = browser.new_page()
        page.goto(url)

        # Check if expected ad is present
        ad_element = page.query_selector(expected_ad["selector"])

        if ad_element:
            # Verify viewability
            is_visible = ad_element.is_visible()
            box = ad_element.bounding_box()

            return {
                "ad_found": True,
                "viewable": is_visible,
                "position": box,
                "screenshot": page.screenshot()
            }

        return {"ad_found": False}

2. Multi-Device Simulation

Verify ads across device types using different proxy and browser configurations:

Desktop + residential proxy
Mobile + mobile proxy
Tablet + residential proxy
Smart TV + datacenter proxy

3. Geographic Sweep

Systematically check ad delivery across all target locations:

def geographic_sweep(ad_url, countries):
    results = {}

    for country in countries:
        proxy = proxy_pool.get_proxy(country=country)
        result = verify_ad(ad_url, proxy, expected_ad)
        results[country] = {
            "ad_displayed": result["ad_found"],
            "correct_creative": result.get("creative_match"),
            "viewable": result.get("viewable"),
            "proxy_ip": proxy.ip,
            "timestamp": datetime.now()
        }

    return results

Lessons for Your Operations

1. Geographic Diversity Matters

Ad verification companies maintain IPs in every country. For your operations, ensure proxy coverage matches your target geographies.

2. Residential IPs Are Essential

Verification companies use residential and mobile IPs because publishers detect and serve different content to datacenter IPs.

3. Scale Requires Automation

At billions of verifications per month, everything is automated. Build automation early, even if you start small.

4. Monitoring Is Non-Negotiable

Every verification request is tracked, measured, and analyzed. Apply the same rigor to your proxy operations.

For ad verification proxy guides and proxy infrastructure tutorials, visit DataResearchTools.

DNS and WebRTC Leaks: The Hidden Proxy Killers

Xavier Fok — Sun, 08 Mar 2026 16:21:39 +0000

You have set up your proxy perfectly. Your IP shows the right location. But your real identity is leaking through DNS queries and WebRTC connections. These invisible leaks are responsible for more account bans than most operators realize.

What Are DNS Leaks?

How DNS Works

When you visit a website, your browser first asks a DNS server to translate the domain name into an IP address. This DNS query happens before the actual web request.

The Leak

Many proxy configurations only route web traffic through the proxy. DNS queries bypass the proxy and go directly to your ISP DNS server, revealing:

Your real ISP — Identifies your actual internet provider
Your real location — ISP DNS servers are geographically located near you
Your browsing history — Every domain you visit is logged by the DNS server

Without DNS leak:
Browser → Proxy → DNS Server → Website
(Everything goes through proxy)

With DNS leak:
Browser → ISP DNS Server (LEAK!)
Browser → Proxy → Website
(DNS bypasses proxy)

How to Detect DNS Leaks

Visit a DNS leak test site through your proxy. If the results show your real ISP or location instead of the proxy location, you have a DNS leak.

How to Fix DNS Leaks

1. Use proxy-level DNS resolution:

Configure your proxy to handle DNS queries:

import requests

# SOCKS5 proxy with remote DNS resolution
proxy = {
    "http": "socks5h://user:pass@proxy:1080",  # Note the h for DNS through proxy
    "https": "socks5h://user:pass@proxy:1080"
}

The socks5h:// protocol (note the h) tells the client to send DNS queries through the SOCKS5 proxy instead of resolving locally.

2. Configure system DNS:

Point your system DNS to the proxy location:

# Use DNS servers matching your proxy location
# For US proxies:
nameserver 8.8.8.8
nameserver 1.1.1.1

3. Use anti-detect browser DNS settings:

Most anti-detect browsers have built-in DNS leak protection. Enable it in profile settings.

What Are WebRTC Leaks?

How WebRTC Works

WebRTC (Web Real-Time Communication) enables peer-to-peer connections for video calls, voice chat, and file sharing in browsers. To establish these connections, it uses STUN and TURN servers.

The Leak

WebRTC can discover your real IP address — even behind a proxy — by querying STUN servers:

// This runs silently in the browser
const pc = new RTCPeerConnection({
    iceServers: [{urls: "stun:stun.l.google.com:19302"}]
});

pc.createDataChannel("");
pc.createOffer().then(offer => pc.setLocalDescription(offer));

pc.onicecandidate = (event) => {
    if (event.candidate) {
        // This candidate string contains your REAL IP
        const ip = event.candidate.candidate
            .split(" ")[4];
        console.log("Real IP exposed:", ip);
    }
};

Platforms can run this code without your knowledge and compare the WebRTC IP to your proxy IP. If they differ, you are flagged.

Types of WebRTC Leaks

Public IP leak — Your real public IP is exposed
Local IP leak — Your local network IP (192.168.x.x) is exposed, which can be used for fingerprinting
mDNS leak — Some browsers use mDNS to mask local IPs, but implementation varies

How to Fix WebRTC Leaks

Option 1: Disable WebRTC entirely

In Firefox:

about:config → media.peerconnection.enabled → false

Option 2: Use anti-detect browsers

Anti-detect browsers offer WebRTC handling options:

Disabled — No WebRTC functionality
Altered — WebRTC shows the proxy IP instead of real IP
Real — Uses actual WebRTC (only for profiles that need video calls)

Option 3: Browser extensions

Extensions like WebRTC Leak Shield can block WebRTC IP exposure in regular browsers.

Other Hidden Leaks

IPv6 Leaks

Your proxy handles IPv4 traffic, but IPv6 requests bypass it:

# Disable IPv6 on Linux
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

HTTP Header Leaks

Some proxies add headers that reveal proxy usage:

X-Forwarded-For — Contains your real IP
Via — Indicates proxy usage
X-Real-IP — Your actual IP

Use elite/anonymous proxies that strip these headers.

Timezone Leaks

JavaScript can read your system timezone:

Intl.DateTimeFormat().resolvedOptions().timeZone
// Returns: "America/New_York" even if proxy is in London

Always match your system timezone to your proxy location.

Complete Leak Prevention Checklist

[ ] DNS queries route through proxy (use socks5h://)
[ ] WebRTC disabled or showing proxy IP
[ ] IPv6 disabled or routed through proxy
[ ] No X-Forwarded-For or Via headers
[ ] Timezone matches proxy location
[ ] Language settings match proxy location
[ ] GPS coordinates disabled or spoofed (mobile)

For comprehensive proxy leak prevention guides, visit DataResearchTools.

Proxy API Integration: Connecting Proxies to Your Automation Pipeline

Xavier Fok — Sun, 08 Mar 2026 16:20:29 +0000

Modern proxy providers offer APIs that go far beyond simple HTTP/SOCKS connections. You can manage IPs, monitor usage, control sessions, and automate your entire proxy lifecycle programmatically. Here is how to integrate proxy APIs into your pipeline.

What Proxy APIs Offer

Most premium providers expose APIs for:

Session management — Create, extend, and terminate sticky sessions
IP management — Request new IPs, check current IP, rotate on demand
Usage monitoring — Track bandwidth, request counts, and costs
Pool configuration — Set geographic targeting, proxy type, rotation rules
Health checking — Monitor proxy status and availability

Common API Integration Patterns

Pattern 1: Gateway Model

The simplest integration. All requests go through a single gateway endpoint. The provider handles rotation and selection.

import requests

# Single gateway endpoint handles everything
proxy = {
    "http": "http://user:pass@gateway.provider.com:8080",
    "https": "http://user:pass@gateway.provider.com:8080"
}

response = requests.get("https://target.com", proxies=proxy)

Pros: Dead simple, no management needed
Cons: Limited control over IP selection

Pattern 2: Session Control via API

Use the API to create and manage sessions explicitly.

import requests

class ProxySession:
    def __init__(self, api_key, country="US"):
        self.api_key = api_key
        self.base_url = "https://api.provider.com/v1"
        self.country = country
        self.session_id = None

    def create_session(self, duration_minutes=30):
        response = requests.post(
            f"{self.base_url}/sessions",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={
                "country": self.country,
                "duration": duration_minutes,
                "type": "residential"
            }
        )
        data = response.json()
        self.session_id = data["session_id"]
        return data["proxy_url"]

    def extend_session(self, additional_minutes=15):
        requests.patch(
            f"{self.base_url}/sessions/{self.session_id}",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={"extend": additional_minutes}
        )

    def terminate_session(self):
        requests.delete(
            f"{self.base_url}/sessions/{self.session_id}",
            headers={"Authorization": f"Bearer {self.api_key}"}
        )
        self.session_id = None

Pattern 3: Usage Monitoring Integration

Track costs and usage programmatically.

class ProxyUsageMonitor:
    def __init__(self, api_key):
        self.api_key = api_key
        self.base_url = "https://api.provider.com/v1"

    def get_usage(self, period="today"):
        response = requests.get(
            f"{self.base_url}/usage",
            headers={"Authorization": f"Bearer {self.api_key}"},
            params={"period": period}
        )
        return response.json()

    def check_budget(self, daily_limit_gb=10):
        usage = self.get_usage("today")
        used_gb = usage["bandwidth_bytes"] / (1024**3)

        if used_gb > daily_limit_gb * 0.8:
            send_alert(f"Warning: {used_gb:.1f}GB used of {daily_limit_gb}GB daily limit")

        if used_gb > daily_limit_gb:
            send_alert("Daily proxy budget exceeded. Pausing operations.")
            return False

        return True

Pattern 4: Multi-Provider Failover

Use multiple providers with automatic failover.

class MultiProviderProxy:
    def __init__(self, providers):
        self.providers = providers  # List of provider configs
        self.primary = 0

    def get_proxy(self):
        for i, provider in enumerate(self.providers):
            try:
                proxy = provider.get_proxy()
                if self.test_proxy(proxy):
                    return proxy
            except Exception:
                continue

        raise Exception("All proxy providers failed")

    def test_proxy(self, proxy):
        try:
            r = requests.get(
                "https://httpbin.org/ip",
                proxies={"http": proxy, "https": proxy},
                timeout=10
            )
            return r.status_code == 200
        except:
            return False

Webhook Integration

Some providers support webhooks for real-time events:

Session expired
Bandwidth threshold reached
IP flagged or blacklisted
Account balance low

from flask import Flask, request

app = Flask(__name__)

@app.route("/proxy-webhook", methods=["POST"])
def handle_webhook():
    event = request.json

    if event["type"] == "session_expired":
        renew_session(event["session_id"])
    elif event["type"] == "bandwidth_alert":
        send_alert(f"Bandwidth at {event["usage_pct"]}%")
    elif event["type"] == "ip_flagged":
        rotate_proxy(event["account_id"])

    return {"status": "ok"}

Best Practices

Abstract provider specifics — Build a proxy interface so you can switch providers easily
Implement retry logic — API calls can fail; always retry with backoff
Cache responses — Do not call usage APIs on every request
Set budget alerts — Prevent runaway costs with automated monitoring
Log everything — API calls, sessions created, IPs used, for debugging

For proxy API integration guides and automation tutorials, visit DataResearchTools.

Proxy Compliance: Legal Considerations Every Operator Should Know

Xavier Fok — Sun, 08 Mar 2026 16:19:02 +0000

Proxies are legal tools. But how you use them determines whether your operations stay on the right side of the law. Here are the legal considerations every proxy operator should understand.

Disclaimer: This article is informational, not legal advice. Consult a qualified attorney for your specific situation.

The Legal Landscape

Proxies Themselves Are Legal

Using a proxy server is perfectly legal in most jurisdictions. Proxies are standard networking tools used by:

Corporations for security and privacy
Researchers for data collection
Journalists for source protection
Individuals for privacy
Businesses for competitive intelligence

The legality depends on what you do through the proxy, not the proxy itself.

Key Legal Frameworks

Computer Fraud and Abuse Act (CFAA) — United States

The CFFA prohibits "unauthorized access" to computer systems. Key considerations:

Accessing publicly available data is generally not unauthorized access
Circumventing technical barriers (CAPTCHAs, login walls) enters gray area
Violating Terms of Service alone may not constitute unauthorized access (hiQ v. LinkedIn ruling)
Creating fake accounts may constitute unauthorized access on some platforms

General Data Protection Regulation (GDPR) — European Union

If you collect personal data of EU residents:

You need a legal basis for processing (legitimate interest, consent, etc.)
Data subjects have rights (access, deletion, portability)
You must maintain records of processing activities
Data breaches must be reported within 72 hours

California Consumer Privacy Act (CCPA)

Similar to GDPR but for California residents. Requires:

Disclosure of data collection practices
Opt-out mechanisms for data selling
Data deletion upon request

What Is Generally Safe

Scraping publicly available data — Product prices, public business listings, published articles
Competitive intelligence — Monitoring competitor pricing and inventory
Ad verification — Checking that your ads display correctly
SEO monitoring — Tracking search rankings across regions
Academic research — Collecting data for published research
Brand protection — Monitoring for trademark violations

What Carries Risk

Scraping personal data without consent — Collecting email addresses, phone numbers, or personal profiles
Circumventing access controls — Bypassing paywalls, login requirements, or explicit blocking
Violating Terms of Service — While not always illegal, it can lead to civil liability
Creating fake accounts — Especially on platforms where real identity is expected
Excessive scraping that damages infrastructure — DDoS-like request volumes
Reselling copyrighted content — Scraping and republishing protected content

Best Practices for Compliance

1. Check robots.txt

# Always respect robots.txt directives
User-agent: *
Disallow: /private/
Crawl-delay: 10

While robots.txt is not legally binding, ignoring it weakens your position if challenged.

2. Respect Rate Limits

Do not overwhelm target servers. Implement reasonable delays between requests.

3. Identify Your Scraper

Consider using a descriptive User-Agent that includes contact information:

User-Agent: CompanyName-PriceBot/1.0 (contact@company.com)

4. Store Data Responsibly

Minimize personal data collection
Implement data retention policies
Secure stored data with encryption
Honor deletion requests

5. Document Your Purpose

Maintain clear documentation of:

Why you are collecting the data
What data you collect
How long you retain it
Who has access to it

6. Review Terms of Service

While ToS violations are not always illegal, understanding them helps you assess risk.

Industry-Specific Considerations

Industry	Key Concern	Recommendation
E-commerce	Price data, product info	Generally safe for public data
Real estate	Listings, agent info	Be cautious with personal data
Social media	User profiles, posts	High risk, especially personal data
Finance	Stock data, filings	Public data is generally safe
Healthcare	Patient data	Extremely high risk, avoid

For proxy compliance guides and operational best practices, visit DataResearchTools.

E-Commerce Price Monitoring with Proxies: A Complete Guide

Xavier Fok — Sun, 08 Mar 2026 16:18:38 +0000

Competitive pricing is the backbone of e-commerce success. Monitoring competitor prices across thousands of products requires proxy infrastructure that can handle scale without getting blocked. Here is how to build it.

Why Price Monitoring Needs Proxies

E-commerce platforms actively block price scrapers because:

Competitor intelligence — They do not want rivals tracking their pricing strategies
MAP enforcement — Minimum Advertised Price agreements require monitoring
Dynamic pricing — Sites show different prices based on location, device, and browsing history
Server protection — High-volume scraping strains their infrastructure

Without proxies, you will get blocked within minutes on major platforms.

Architecture for Price Monitoring

Product URL Database
        |
        v
Scheduler (cron/Celery)
        |
        v
Scraper Workers (parallel)
        |
        v
Proxy Pool (rotating residential)
        |
        v
Target E-Commerce Sites
        |
        v
Price Parser
        |
        v
Price Database (PostgreSQL/MongoDB)
        |
        v
Alert System (price changes)

Platform-Specific Proxy Requirements

Platform	Proxy Type	Difficulty	Notes
Amazon	Residential	High	Heavy bot detection, geo-specific pricing
eBay	Residential	Medium	Rate limiting focused
Walmart	Residential	High	Akamai Bot Manager
Shopify stores	Datacenter	Low	Basic rate limiting
AliExpress	Residential	Medium	Cloudflare protected
Target	Residential	Medium	PerimeterX protection

Building the Scraper

import requests
from bs4 import BeautifulSoup
import json
import time
import random

class PriceMonitor:
    def __init__(self, proxy_manager):
        self.proxy_manager = proxy_manager
        self.session = requests.Session()

    def get_price(self, url, selectors):
        proxy = self.proxy_manager.get_proxy()
        headers = self.get_headers()

        try:
            time.sleep(random.uniform(1, 3))
            response = self.session.get(
                url,
                proxies={"http": proxy, "https": proxy},
                headers=headers,
                timeout=15
            )

            if response.status_code == 200:
                return self.extract_price(response.text, selectors)
            else:
                self.proxy_manager.report_failure(proxy)
                return None

        except Exception as e:
            self.proxy_manager.report_failure(proxy)
            return None

    def extract_price(self, html, selectors):
        soup = BeautifulSoup(html, "html.parser")

        for selector in selectors:
            element = soup.select_one(selector)
            if element:
                price_text = element.get_text(strip=True)
                # Extract numeric price
                price = float(
                    price_text.replace("$", "")
                    .replace(",", "")
                    .strip()
                )
                return price
        return None

    def get_headers(self):
        user_agents = [
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36",
        ]
        return {
            "User-Agent": random.choice(user_agents),
            "Accept": "text/html,application/xhtml+xml",
            "Accept-Language": "en-US,en;q=0.9",
            "Accept-Encoding": "gzip, deflate, br",
        }

Handling Dynamic Pricing

Many sites show different prices based on:

Location

Use geo-targeted proxies to see prices in different regions:

# Check price from US
us_price = monitor.get_price(url, proxy_geo="US")
# Check price from UK  
uk_price = monitor.get_price(url, proxy_geo="UK")

Device Type

Mobile vs desktop can show different prices. Set appropriate User-Agent headers.

Browsing History

Use clean proxy sessions without cookies to get baseline prices.

Price Change Alerts

def check_price_change(product_id, new_price, db):
    old_price = db.get_latest_price(product_id)

    if old_price is None:
        db.save_price(product_id, new_price)
        return

    change_pct = ((new_price - old_price) / old_price) * 100

    if abs(change_pct) > 5:  # 5% threshold
        send_alert(
            f"Price change detected for {product_id}: "
            f"${old_price} -> ${new_price} ({change_pct:+.1f}%)"
        )

    db.save_price(product_id, new_price)

Scaling Tips

Schedule intelligently — Check prices 2-4 times daily, not every hour
Prioritize products — High-value products get checked more frequently
Use appropriate proxies — Datacenter for easy targets, residential for Amazon/Walmart
Cache product pages — Extract multiple data points per request
Respect rate limits — 1-2 requests per second per target domain

For e-commerce proxy guides and price monitoring tutorials, visit DataResearchTools.

Real-Time Proxy Monitoring: Build a Dashboard with Python and Grafana

Xavier Fok — Sun, 08 Mar 2026 16:18:07 +0000

Flying blind with proxies is expensive. Without monitoring, you do not know which proxies are healthy, which are burned, or how much bandwidth you are wasting on failed requests. Here is how to build a real-time monitoring dashboard.

What to Monitor

Core Metrics

Success rate — Percentage of requests returning HTTP 200
Response time — Average and P95 latency per proxy
Bandwidth usage — Data consumed per proxy and total
Error distribution — Types of errors (timeout, 403, 429, CAPTCHA)
IP uniqueness — How many unique IPs you are actually using

Operational Metrics

Pool health — Percentage of active vs failed proxies
Rotation frequency — How often IPs change
Geographic distribution — Where your exit IPs are located
Cost per successful request — Real cost accounting
Blacklist rate — How many IPs are currently blocked

Architecture

Your Application
    |
    v
Proxy Middleware (collects metrics)
    |
    v
Prometheus (stores time-series data)
    |
    v
Grafana (visualizes dashboards)

Step 1: Metrics Collection

Create a proxy wrapper that logs every request:

import time
import requests
from prometheus_client import Counter, Histogram, Gauge, start_http_server

# Define metrics
REQUEST_COUNT = Counter(
    "proxy_requests_total",
    "Total proxy requests",
    ["proxy", "status", "target_domain"]
)

RESPONSE_TIME = Histogram(
    "proxy_response_seconds",
    "Response time in seconds",
    ["proxy"],
    buckets=[0.1, 0.5, 1, 2, 5, 10, 30]
)

ACTIVE_PROXIES = Gauge(
    "proxy_pool_active",
    "Number of active proxies in pool"
)

BANDWIDTH = Counter(
    "proxy_bandwidth_bytes",
    "Bandwidth consumed in bytes",
    ["proxy", "direction"]
)

class MonitoredProxy:
    def __init__(self, proxy_url):
        self.proxy_url = proxy_url
        self.proxy_dict = {"http": proxy_url, "https": proxy_url}

    def request(self, url, **kwargs):
        start = time.time()
        domain = url.split("/")[2]

        try:
            response = requests.get(
                url,
                proxies=self.proxy_dict,
                timeout=kwargs.get("timeout", 15),
                **kwargs
            )
            duration = time.time() - start

            # Record metrics
            REQUEST_COUNT.labels(
                proxy=self.proxy_url,
                status=str(response.status_code),
                target_domain=domain
            ).inc()

            RESPONSE_TIME.labels(proxy=self.proxy_url).observe(duration)

            BANDWIDTH.labels(
                proxy=self.proxy_url, direction="response"
            ).inc(len(response.content))

            return response

        except Exception as e:
            duration = time.time() - start
            REQUEST_COUNT.labels(
                proxy=self.proxy_url,
                status="error",
                target_domain=domain
            ).inc()
            raise

Step 2: Prometheus Configuration

# prometheus.yml
scrape_configs:
  - job_name: "proxy_monitor"
    scrape_interval: 15s
    static_configs:
      - targets: ["localhost:8000"]

Step 3: Grafana Dashboard Panels

Key panels for your dashboard:

Success Rate Over Time

rate(proxy_requests_total{status="200"}[5m]) /
rate(proxy_requests_total[5m]) * 100

Average Response Time

rate(proxy_response_seconds_sum[5m]) /
rate(proxy_response_seconds_count[5m])

Error Breakdown

sum by (status) (rate(proxy_requests_total{status!="200"}[5m]))

Bandwidth Usage

sum(rate(proxy_bandwidth_bytes[1h])) * 3600

Alert Rules

Set up alerts for critical conditions:

# alert_rules.yml
groups:
  - name: proxy_alerts
    rules:
      - alert: LowSuccessRate
        expr: |
          rate(proxy_requests_total{status="200"}[5m]) /
          rate(proxy_requests_total[5m]) < 0.8
        for: 5m
        annotations:
          summary: Proxy success rate below 80%

      - alert: HighLatency
        expr: |
          rate(proxy_response_seconds_sum[5m]) /
          rate(proxy_response_seconds_count[5m]) > 5
        for: 5m
        annotations:
          summary: Average proxy latency above 5 seconds

Quick Alternative: Simple File-Based Logging

If Prometheus and Grafana are overkill for your setup, a simple CSV logger works:

import csv
from datetime import datetime

def log_request(proxy, url, status, latency, bytes_received):
    with open("proxy_log.csv", "a") as f:
        writer = csv.writer(f)
        writer.writerow([
            datetime.now().isoformat(),
            proxy, url, status,
            round(latency, 3),
            bytes_received
        ])

Analyze with pandas later to identify trends and problem proxies.

For proxy monitoring setups and infrastructure guides, visit DataResearchTools.