DEV Community: Elen Simonian

ChatGPT Proxy Setup: How to Access OpenAI Without Geo-Restrictions in 2026

Elen Simonian — Wed, 26 Aug 2026 15:53:55 +0000

OpenAI restricts ChatGPT access in a growing list of countries. Beyond geographic blocks, developers building on the OpenAI API run into rate limits by IP, multi-account workflows need isolated identities, and AI agents that browse the web need clean residential IPs to interact with external services without getting blocked.

This guide covers the three main reasons to use a proxy with ChatGPT, how to set one up, and the Python code for routing OpenAI API calls through a proxy.

Why Use a Proxy with ChatGPT or the OpenAI API

1. Geo-Restrictions

OpenAI restricts access to ChatGPT and the API from certain countries and regions. The restriction is enforced at the IP level: connections from blocked IPs receive an access error regardless of account status.

A residential proxy with a US or other eligible country IP routes your connection through an address that OpenAI treats as legitimate. Residential IPs are the correct choice here because OpenAI's systems score IP reputation. Datacenter IPs and VPN addresses are routinely flagged. A residential IP from a real household ISP on a clean network passes the check.

2. Multi-Account Testing

Developers and QA teams testing ChatGPT behavior across multiple accounts, API keys, or user profiles need each account to connect from a separate IP. Two accounts sharing the same IP are linked by OpenAI's systems, which can trigger rate limits or restrictions across both.

Residential sticky sessions solve this: one IP per account, held consistently across all requests from that account. NodeMaven's residential proxies support sticky sessions up to 24 hours, which covers a full day of testing per account without IP changes.

3. AI Agents Browsing External Web

AI agents built on ChatGPT or the OpenAI API frequently need to browse external websites, collect data, and interact with third-party services as part of their task execution. Those external services have their own bot detection.

When an agent's web requests come from a clean residential IP, external services treat them the same as organic user traffic. When they come from a datacenter IP or a flagged address, the request fails, the agent retries, and the workflow breaks. NodeMaven proxies for AI agents are built around this use case: sticky sessions for multi-step workflows, 30M+ residential IPs, and a 99.54% average success rate.

Use Case Matrix

Use Case	Proxy Type	Why
Access ChatGPT from restricted country	Residential (US or eligible country)	Household IP matches OpenAI's expected traffic
Multi-account testing	Residential sticky, one per account	Consistent IP per account avoids linking
Agent browsing external web	Rotating residential	IP diversity across many target sites
Long-running authenticated agent	ISP proxy	Fixed IP for stable auth sessions over weeks
API calls through proxy	Residential (HTTPS)	Route API traffic through clean residential IP

Proxy Setup: Browser Access

For accessing ChatGPT through a browser, configure a proxy at the browser or system level. Get your NodeMaven credentials from the dashboard in {host}:{port}:{username}:{password} format. Use port 8080 for HTTP and port 1080 for SOCKS5.

Set the target country to the United States or another eligible OpenAI country in your NodeMaven dashboard before connecting.

Proxy Setup: OpenAI API Calls

The OpenAI Python SDK supports proxy configuration through the http_client parameter, which accepts an httpx.Client with proxy settings. This routes all API calls through the NodeMaven proxy.

pip install openai httpx

import httpx
from openai import OpenAI

PROXY_HOST = "gate.nodemaven.com"
PROXY_PORT = "8080"
PROXY_USER = "your_nodemaven_username"
PROXY_PASS = "your_nodemaven_password"

proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"

# Configure OpenAI client with proxy
client = OpenAI(
    api_key="your_openai_api_key",
    http_client=httpx.Client(
        proxy=proxy_url,
        timeout=30.0,
    )
)

# All API calls now route through the NodeMaven proxy
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)

Async Client

For async workflows using AsyncOpenAI:

import asyncio
import httpx
from openai import AsyncOpenAI

proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"

async_client = AsyncOpenAI(
    api_key="your_openai_api_key",
    http_client=httpx.AsyncClient(
        proxy=proxy_url,
        timeout=30.0,
    )
)

async def call_gpt(prompt: str) -> str:
    response = await async_client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

result = asyncio.run(call_gpt("Summarize the benefits of residential proxies"))
print(result)

Routing Agent Web Requests Through Proxy

When a ChatGPT-based agent needs to browse external URLs, route those requests through the same proxy. Using httpx directly for agent web browsing:

import httpx

proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"

def agent_fetch(url: str) -> str:
    """Fetch a URL through residential proxy for agent web browsing."""
    headers = {
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/124.0.0.0 Safari/537.36"
        )
    }
    with httpx.Client(proxy=proxy_url, timeout=20.0) as client:
        response = client.get(url, headers=headers, follow_redirects=True)
        response.raise_for_status()
        return response.text

# Agent browses external source through clean residential IP
html = agent_fetch("https://example.com/data")
print(f"Fetched {len(html)} characters")

Choosing the Right Proxy Type

Accessing ChatGPT from a restricted country: Residential proxy, US or eligible country, sticky session. One clean household IP presents as a genuine American user.
Multi-account API testing: Residential sticky sessions, one per account, up to 24 hours. Each account maintains its own IP identity across all requests.
Agent browsing external sites: Rotating residential. IP diversity across the 30M+ pool reduces reuse on any single external target.
Long-running authenticated agent sessions: ISP proxy. Fixed IP for weeks covers authentication flows that need consistent identity over time. NodeMaven's ISP proxies are specifically recommended for long-lived agent sessions and authentication flows.

For Claude-based agents with similar requirements, the same proxy infrastructure applies. Details at nodemaven.com/websites/claude-proxy/.

NodeMaven for AI Agent Workflows

NodeMaven's AI agent proxy infrastructure covers three requirements: session stability, IP quality, and scale. Residential and mobile proxies include sticky sessions up to 24 hours, a 30M+ IP pool with real-time quality filtering, and a 99.54% average success rate on protected platforms including Google, Amazon, LinkedIn, and TikTok.

For AI agent workflows: residential proxies are recommended for distributed agents and large-scale web scraping. Mobile proxies for login-based workflows and sensitive data collection. ISP proxies for long-lived agent sessions and authentication flows.

Pricing from $2.20/GB for residential and mobile, $2.99/IP for ISP. Trial at $3.50 for 750MB. Details at nodemaven.com/use-cases/ai-agent-proxy/.

How to Scrape LinkedIn Data Legally and Without Bans in 2026

Elen Simonian — Wed, 26 Aug 2026 15:44:09 +0000

LinkedIn is one of the most valuable data sources for B2B research, lead generation, talent intelligence, and market analysis. It is also one of the most aggressively protected. LinkedIn's bot detection has matured significantly over the past two years, and the gap between setups that work and setups that get banned immediately comes down to a few specific technical decisions.

This guide covers the legal and technical landscape of LinkedIn scraping, how to choose between ISP and rotating residential proxies depending on whether you're logging into an account or pulling public pages, and a complete working Playwright setup with rate limiting built in.

Legal Context: What You Can and Cannot Scrape on LinkedIn

This section is not legal advice. If you are scraping LinkedIn at scale for commercial purposes, consult a lawyer familiar with data protection law in your jurisdiction.

The legal landscape around LinkedIn scraping has been shaped significantly by the hiQ Labs v. LinkedIn case in the United States. In that case, the Ninth Circuit Court of Appeals held that scraping publicly available data from LinkedIn does not violate the Computer Fraud and Abuse Act (CFAA), because accessing public pages does not constitute unauthorized access to a protected computer.

The practical boundaries are:

Public profile data: name, title, employer, location, skills listed publicly. Generally considered accessible under the hiQ ruling.
Private data behind a login wall: contact information, connection lists, private posts. Accessing this requires authentication, which creates different legal exposure.
Rate and volume: even legally accessible data can trigger ToS violations and account bans if scraped aggressively. LinkedIn's ToS prohibits automated access regardless of the data's public nature.
GDPR and equivalent laws: in the EU, even scraping publicly available personal data may require a lawful basis. Data minimization and purpose limitation apply.

The safest approach for most use cases: scrape public profile data, do not scrape behind login, keep request rates well below what a human could produce, and do not store personal data beyond what your use case requires.

Why LinkedIn Bans Scrapers and How Detection Works

LinkedIn runs one of the most sophisticated bot detection systems among professional platforms. Detection operates on several layers simultaneously:

IP reputation scoring. LinkedIn scores every incoming IP against fraud databases. Datacenter IPs are flagged immediately. Many residential IPs from unfiltered pools carry blacklist history from prior abuse and fail the same check.
Request velocity. LinkedIn monitors requests per IP per time window. Even clean IPs get flagged if they send requests faster than a human could browse.
Session behavior. Real users do not visit profiles in alphabetical order at 1-second intervals. Behavioral anomalies in navigation patterns trigger flags.
Login IP consistency. Accounts that log in from different IPs across sessions or from IPs that do not match their registered location get security prompts and eventual restrictions.
Browser fingerprint. Headless browsers without proper fingerprint configuration are detectable through Canvas, WebGL, and JavaScript environment checks.

ISP vs. Rotating Residential: Which Proxy Fits Your LinkedIn Workflow

Not every proxy type works equally well here. LinkedIn's detection weighs IP consistency over time more heavily than most platforms, so the right choice comes down to what you're actually doing: logging into an account, or pulling public pages without one.

A rotating residential proxy assigns a new IP on each request or session. Every time you log in from a different IP, LinkedIn's security system registers an anomaly. Enough anomalies and the account gets a verification prompt, a temporary restriction, or a permanent ban.

An ISP proxy holds the same IP for 30 or 90 days. Every login, every profile visit, every search from that account comes from the same ISP-registered address. To LinkedIn's security system, this looks exactly like a real user on a consistent home internet connection.

Proxy Type	Session	LinkedIn Fit
ISP (NodeMaven)	30-90 days fixed	Best: same IP every login, builds account trust history
Residential sticky	Up to 24 hours	Good: day-long sessions, city-level targeting
Rotating residential	Per request	Scraping public pages at scale, no login required
Datacenter	Variable	Not suitable: flagged immediately by LinkedIn

For scraping public LinkedIn pages without authentication, rotating residential proxies work well and give you more IP diversity across a large volume of requests — that's the setup used in the Playwright example below. For anything involving a LinkedIn account, ISP proxies are the correct choice.

NodeMaven ISP proxies are quality-checked using Scamalytics before assignment, with a 95.40% IP quality rate. Every order includes one free IP pack swap. Plans are 30 or 90 days, from $2.99/IP, with unlimited traffic, HTTP/HTTPS, SOCKS5, and UDP support, and 99.9% uptime. Available in 9 countries: United States, United Kingdom, Germany, France, Romania, Italy, Brazil, Hong Kong, and Poland (worth double-checking the current list on the dashboard, since coverage changes).

Rate Limiting: LinkedIn-Specific Patterns

LinkedIn's rate limiting is not a simple request count. It combines IP-level, account-level, and behavioral signals. These are the patterns that work:

Requests per session: 20-30 profile views. Real LinkedIn users do not browse 200 profiles in one sitting. Keep individual sessions short and spread them across a longer time window.
Inter-request delay: 8-15 seconds minimum. Human reading speed on a LinkedIn profile is roughly 30-60 seconds. A 2-second delay between profiles is an obvious bot signal. Use random delays in the 8-15 second range as a minimum.
Session length: 30-45 minutes maximum. Real browsing sessions are not 6 hours of continuous profile scraping. Cap session length and introduce breaks between sessions.
Daily volume per account: under 100 profiles. LinkedIn's own limits for Sales Navigator users are in the hundreds per day. Manual accounts scraping thousands of profiles per day are anomalous.
Vary navigation patterns. Do not visit profiles in a predictable sequence. Mix in searches, feed browsing, and company page visits between profile visits.

Complete Playwright Setup for LinkedIn

Installation

pip install playwright
playwright install chromium

Proxy Configuration with Rotating Residential Credentials

Get your rotating residential proxy credentials from the NodeMaven dashboard in {host}:{port}:{username}:{password} format. NodeMaven's gateway runs on port 8080 for HTTP, and each new session ID pulls a fresh IP — that's what gives this scraper its IP diversity.

import asyncio
import random
import time
from playwright.async_api import async_playwright

PROXY_HOST = "gate.nodemaven.com"
PROXY_PORT = "8080"
PROXY_USER = "your_nodemaven_username"
PROXY_PASS = "your_nodemaven_password"

# Realistic user agents
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
]

Rate-Limited Profile Scraper

class LinkedInScraper:
    def __init__(self):
        self.request_count = 0
        self.session_start = time.time()
        self.max_requests_per_session = 25
        self.session_duration_limit = 35 * 60  # 35 minutes

    def should_take_break(self) -> bool:
        session_age = time.time() - self.session_start
        return (
            self.request_count >= self.max_requests_per_session
            or session_age >= self.session_duration_limit
        )

    def reset_session(self):
        self.request_count = 0
        self.session_start = time.time()

    async def human_delay(self, min_s: float = 8.0, max_s: float = 15.0):
        """Randomized delay mimicking human reading speed."""
        delay = random.uniform(min_s, max_s)
        await asyncio.sleep(delay)

    async def scrape_public_profile(self, page, profile_url: str) -> dict:
        """Scrape a single public LinkedIn profile."""
        try:
            await page.goto(profile_url, wait_until="domcontentloaded", timeout=30000)
            await asyncio.sleep(random.uniform(2, 4))  # initial load pause

            # Check for login wall or CAPTCHA
            if await self._is_blocked(page):
                print(f"Blocked on: {profile_url}")
                return {"url": profile_url, "status": "blocked"}

            # Extract public data
            data = {
                "url": profile_url,
                "status": "ok",
                "name": await self._safe_text(page, "h1"),
                "headline": await self._safe_text(page, ".text-body-medium"),
                "location": await self._safe_text(page, ".text-body-small.inline.t-black--light"),
            }
            self.request_count += 1
            return data

        except Exception as e:
            print(f"Error scraping {profile_url}: {e}")
            return {"url": profile_url, "status": "error", "error": str(e)}

    async def _is_blocked(self, page) -> bool:
        content = (await page.content()).lower()
        block_signals = [
            "authwall",
            "sign in to view",
            "join to see",
            "challenge",
            "security verification",
        ]
        return any(s in content for s in block_signals)

    async def _safe_text(self, page, selector: str) -> str:
        try:
            el = page.locator(selector).first
            return (await el.inner_text()).strip()
        except Exception:
            return ""

Main Scraping Loop with Session Management

async def run_scraper(profile_urls: list):
    scraper = LinkedInScraper()
    results = []

    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={
                "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
                "username": PROXY_USER,
                "password": PROXY_PASS,
            },
            headless=True,
        )
        context = await browser.new_context(
            user_agent=random.choice(USER_AGENTS),
            viewport={"width": 1440, "height": 900},
            locale="en-US",
        )
        page = await context.new_page()

        # Block images and media to reduce bandwidth
        await page.route(
            "**/*",
            lambda route: route.abort()
            if route.request.resource_type in ["image", "media", "font"]
            else route.continue_()
        )

        for url in profile_urls:
            # Check if session limits are hit
            if scraper.should_take_break():
                break_duration = random.uniform(300, 600)  # 5-10 min break
                print(f"Session limit reached. Breaking for {break_duration:.0f}s...")
                await browser.close()
                await asyncio.sleep(break_duration)

                # New browser instance = fresh connection context
                browser = await p.chromium.launch(
                    proxy={
                        "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
                        "username": PROXY_USER,
                        "password": PROXY_PASS,
                    }
                )
                context = await browser.new_context(
                    user_agent=random.choice(USER_AGENTS),
                    viewport={"width": 1440, "height": 900},
                )
                page = await context.new_page()
                scraper.reset_session()

            result = await scraper.scrape_public_profile(page, url)
            results.append(result)
            print(f"Scraped: {result.get('name', 'unknown')} | {result['status']}")

            # Human-speed delay between profiles
            await scraper.human_delay()

        await browser.close()
        return results

# Usage
profiles = [
    "https://www.linkedin.com/in/example-profile-1/",
    "https://www.linkedin.com/in/example-profile-2/",
]
results = asyncio.run(run_scraper(profiles))
print(f"Scraped {len(results)} profiles")

What This Setup Does Right

Rotating residential IPs. Each new session pulls a fresh residential IP, so request volume is spread across a large pool instead of hammering LinkedIn from one address. With no account to protect, diversity beats a fixed identity.
Session length caps. The scraper stops after 25 requests or 35 minutes, whichever comes first, and takes a 5-10 minute break. This mimics real browsing behavior.
Human-speed delays. 8-15 seconds between requests is well within the range of human reading speed on a profile page.
Block detection. The scraper checks for authwall and challenge signals on every page before parsing. A blocked response is logged, not parsed.
Bandwidth saving. Images, media, and fonts are blocked via page.route(). Because rotating residential proxies are billed by the gigabyte, this saves cost as well as speed, and it also reduces fingerprinting surface.
Fresh browser context after breaks. A new browser instance creates a fresh connection context, and a new session ID pulls a new residential IP along with it — so both the fingerprint and the IP look like a first-time visitor after every break.

What to Avoid

Logging into LinkedIn through the scraper. Authentication creates legal exposure and triggers much stricter detection. Stick to public pages — and if your workflow does need a login, switch to ISP proxies for the fixed-IP identity an authenticated session needs.
Scraping faster than human speed. Even 3-second delays are detectable. 8 seconds minimum.
Running the scraper 24 hours a day. Real users do not browse LinkedIn continuously. Add realistic off-hours in the schedule.
Using datacenter IPs. LinkedIn flags these immediately. NodeMaven's rotating residential proxies use real residential addresses, not datacenter ranges.
Reusing the same session ID for too many requests. Let the scraper rotate sessions the way it's built to — a fresh IP every 25 requests or 35 minutes keeps the pool diverse instead of hammering LinkedIn from one address disguised as many.

Rotating Residential Proxies for Public Page Scraping

If your use case is scraping public LinkedIn company pages, job listings, or search results without authentication, rotating residential proxies are the better fit — not ISP. They provide more IP diversity across high-volume requests, which is exactly what this workflow needs instead of a single fixed identity.

NodeMaven's residential proxies cover 30M+ IPs across 190+ countries with city-level targeting and a 99.54% average success rate. For large-scale public page collection where session continuity is not required, they are a cost-effective option. Details at nodemaven.com/proxies/residential-proxies/.

Getting Started

For the scraping setup above, get rotating residential proxy credentials from the NodeMaven dashboard — 30M+ IPs across 190+ countries, with a 99.54% average success rate. If your workflow logs into a LinkedIn account instead, switch to ISP proxies: available in 9 countries, from $2.99/IP on 30 or 90-day plans, with unlimited traffic, HTTP/HTTPS, SOCKS5, and UDP support. Each IP is quality-checked using Scamalytics before assignment, and every order includes one free IP pack swap.

Paste PROXY_USER and PROXY_PASS into the setup above, and the rate limiter and session manager handle the rest. Details at nodemaven.com/proxies/residential-proxies/ and nodemaven.com/proxies/isp-proxies/.

Playwright with Proxies: Full Integration Guide for 2026

Elen Simonian — Thu, 20 Aug 2026 10:04:42 +0000

Playwright is the standard tool for browser automation in 2026. It handles Chromium, Firefox, and WebKit from a single API, supports async/await natively, and integrates cleanly with modern Python and Node.js workflows. But without proxies, Playwright automation hits rate limits and blocks quickly on any platform with bot detection.

This guide covers every aspect of integrating NodeMaven residential proxies with Playwright: proxy configuration for all three browser engines, authentication handling, rotating and sticky session patterns, and a CAPTCHA detection strategy. All code is production-ready and tested against real-world targets.

Prerequisites

Install Playwright and its browser binaries:

pip install playwright
playwright install

You will need NodeMaven proxy credentials from your dashboard. Copy them in {host}:{port}:{username}:{password} format. NodeMaven supports both HTTP and SOCKS5 protocols. Use port 8080 for HTTP and port 1080 for SOCKS5.

Basic Proxy Setup: Chromium

Playwright accepts proxy configuration at the browser level via the proxy parameter in launch(). The server field takes the full proxy URL including protocol.

import asyncio
from playwright.async_api import async_playwright

PROXY_HOST = "gate.nodemaven.com"
PROXY_PORT = "8080"
PROXY_USER = "your_nodemaven_username"
PROXY_PASS = "your_nodemaven_password"

async def basic_chromium():
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={
                "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
                "username": PROXY_USER,
                "password": PROXY_PASS,
            }
        )
        page = await browser.new_page()
        await page.goto("https://httpbin.org/ip")
        content = await page.content()
        print(content)
        await browser.close()

asyncio.run(basic_chromium())

The proxy applies to all requests made by the browser instance. Every page opened under this browser uses the same proxy configuration.

Proxy Setup: Firefox and WebKit

The proxy configuration syntax is identical across all three engines. Swap p.chromium for p.firefox or p.webkit:

async def multi_engine_example():
    async with async_playwright() as p:
        proxy_config = {
            "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
            "username": PROXY_USER,
            "password": PROXY_PASS,
        }

        browser_cr = await p.chromium.launch(proxy=proxy_config)
        browser_ff = await p.firefox.launch(proxy=proxy_config)
        browser_wk = await p.webkit.launch(proxy=proxy_config)

        for browser, name in [
            (browser_cr, "Chromium"),
            (browser_ff, "Firefox"),
            (browser_wk, "WebKit"),
        ]:
            page = await browser.new_page()
            await page.goto("https://httpbin.org/ip")
            print(f"{name}: {await page.inner_text('body')}")
            await browser.close()

asyncio.run(multi_engine_example())

For scraping workflows targeting sites that fingerprint browser engine type, use Chromium. For sites that explicitly check for non-Chrome browsers, Firefox gives you a different engine fingerprint. WebKit is useful for testing how iOS Safari users see content.

SOCKS5 Proxy Configuration

NodeMaven supports SOCKS5 proxies, which handle all traffic types, not just HTTP. SOCKS5 is available at port 1080. For more on the SOCKS5 protocol and use cases, see nodemaven.com/proxies/socks5-proxy-server/.

async def socks5_example():
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={
                "server": f"socks5://{PROXY_HOST}:1080",
                "username": PROXY_USER,
                "password": PROXY_PASS,
            }
        )
        page = await browser.new_page()
        await page.goto("https://httpbin.org/ip")
        print(await page.inner_text("body"))
        await browser.close()

asyncio.run(socks5_example())

Rotating Sessions: New IP Per Browser Instance

For scraping tasks that need a different IP on each run, instantiate a new browser for each request batch. NodeMaven assigns a new IP from the 30M+ residential pool on each new connection when no session parameter is set.

import asyncio
import random
from playwright.async_api import async_playwright

TARGETS = [
    "https://www.amazon.com/s?k=laptop",
    "https://www.amazon.com/s?k=monitor",
    "https://www.amazon.com/s?k=keyboard",
]

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0",
]

async def scrape_with_rotation(url: str):
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={
                "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
                "username": PROXY_USER,
                "password": PROXY_PASS,
            }
        )
        context = await browser.new_context(
            user_agent=random.choice(USER_AGENTS),
            viewport={"width": 1366, "height": 768},
        )
        page = await context.new_page()
        try:
            await page.goto(url, wait_until="domcontentloaded", timeout=30000)
            title = await page.title()
            print(f"OK: {title[:60]} | {url}")
            return await page.content()
        except Exception as e:
            print(f"Failed: {url} | {e}")
            return None
        finally:
            await browser.close()

async def main():
    for url in TARGETS:
        html = await scrape_with_rotation(url)
        await asyncio.sleep(random.uniform(2, 5))

asyncio.run(main())

Each call to scrape_with_rotation() opens a new browser and gets a fresh IP from the pool. NodeMaven's 95% clean IP rate and 99.54% average success rate mean the vast majority of these fresh IPs succeed on the first attempt.

Sticky Sessions: Same IP Across Multiple Pages

For workflows requiring session continuity — login flows, checkout sequences, account management — configure sticky sessions in your NodeMaven dashboard before starting. The same IP holds for up to 7 days on residential proxies.

async def login_and_scrape():
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={
                "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
                "username": PROXY_USER,
                "password": PROXY_PASS,
            },
            headless=True,
        )
        context = await browser.new_context(
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/124.0.0.0 Safari/537.36"
            ),
            viewport={"width": 1440, "height": 900},
        )
        page = await context.new_page()

        # Step 1: login page
        await page.goto("https://example-site.com/login", wait_until="networkidle")
        await page.fill("#username", "your_account_username")
        await page.fill("#password", "your_account_password")
        await page.click("#login-button")
        await page.wait_for_load_state("networkidle")

        # Step 2: navigate to protected page - same IP throughout
        await page.goto("https://example-site.com/dashboard")
        print("Dashboard loaded, IP consistent across all steps")

        cookies = await context.cookies()
        print(f"Saved {len(cookies)} cookies")
        await browser.close()

asyncio.run(login_and_scrape())

Context-Level Proxy Override

Playwright supports setting proxy at the context level, which lets you run multiple contexts within one browser instance, each with a different proxy:

async def multi_context_proxies():
    async with async_playwright() as p:
        browser = await p.chromium.launch()

        context_us = await browser.new_context(
            proxy={
                "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
                "username": f"{PROXY_USER}-country-us",
                "password": PROXY_PASS,
            }
        )
        context_uk = await browser.new_context(
            proxy={
                "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
                "username": f"{PROXY_USER}-country-gb",
                "password": PROXY_PASS,
            }
        )

        page_us = await context_us.new_page()
        page_uk = await context_uk.new_page()

        await page_us.goto("https://httpbin.org/ip")
        await page_uk.goto("https://httpbin.org/ip")

        print("US IP:", await page_us.inner_text("body"))
        print("UK IP:", await page_uk.inner_text("body"))
        await browser.close()

asyncio.run(multi_context_proxies())

This pattern is useful for geo-comparison tasks: checking how a page renders for US vs UK users in parallel, or running ad verification across multiple markets simultaneously.

CAPTCHA Detection and Handling

NodeMaven's 95% clean IP rate reduces CAPTCHA frequency significantly. But CAPTCHAs still appear occasionally. Detect them and rotate to a fresh IP rather than retrying the same address:

CAPTCHA_SIGNALS = [
    "captcha", "recaptcha", "hcaptcha",
    "are you a robot", "verify you are human",
    "unusual traffic", "automated queries", "g-recaptcha",
]

async def is_captcha_page(page) -> bool:
    try:
        content = (await page.content()).lower()
        title   = (await page.title()).lower()
        return any(s in content or s in title for s in CAPTCHA_SIGNALS)
    except Exception:
        return False

async def scrape_with_captcha_handling(url: str, max_retries: int = 3):
    for attempt in range(max_retries):
        async with async_playwright() as p:
            browser = await p.chromium.launch(
                proxy={
                    "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
                    "username": PROXY_USER,
                    "password": PROXY_PASS,
                }
            )
            page = await (await browser.new_context()).new_page()
            try:
                await page.goto(url, wait_until="domcontentloaded", timeout=30000)
                if await is_captcha_page(page):
                    print(f"CAPTCHA on attempt {attempt + 1}, rotating IP...")
                    await browser.close()
                    await asyncio.sleep(random.uniform(2, 5))
                    continue
                content = await page.content()
                await browser.close()
                return content
            except Exception as e:
                print(f"Error on attempt {attempt + 1}: {e}")
                await browser.close()
                await asyncio.sleep(random.uniform(1, 3))
    return None

Each retry opens a new browser and draws a fresh IP from the pool. Since the pool is 30M+ pre-filtered IPs, successive retries are unlikely to encounter the same flagged address.

Bandwidth Saving: Blocking Unnecessary Assets

Playwright's route() API lets you intercept requests before they go out through the proxy. Block images, fonts, and stylesheets to reduce bandwidth consumption on text-heavy scraping targets:

async def scrape_with_request_filtering(url: str):
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={
                "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
                "username": PROXY_USER,
                "password": PROXY_PASS,
            }
        )
        page = await browser.new_page()
        await page.route(
            "**/*",
            lambda route: route.abort()
            if route.request.resource_type in ["image", "font", "media", "stylesheet"]
            else route.continue_()
        )
        await page.goto(url, wait_until="domcontentloaded")
        content = await page.content()
        await browser.close()
        return content

Since NodeMaven is priced per GB at $2.20/GB, reducing unnecessary asset downloads translates directly to lower costs on large-scale runs.

Quick Reference: Playwright Proxy Patterns

Pattern	Config level	Use case
Single browser proxy	`launch(proxy=...)`	All pages use same proxy
Context-level proxy	`new_context(proxy=...)`	Different proxy per context in one browser
Rotating IP	New browser per task	SERP scraping, price monitoring
Sticky session	Dashboard session config	Login flows, account management
SOCKS5	`server: socks5://` port 1080	All traffic types, not just HTTP
CAPTCHA retry	Close browser, open new	Any anti-bot target
Bandwidth saving	`page.route()`	Text scraping at scale

Getting Started

NodeMaven residential proxies integrate with Playwright's launch() and new_context() proxy parameters directly. 30M+ IPs across 190+ countries, 95% clean IP rate, 99.54% average success rate, sticky sessions up to 24 hours, HTTP and SOCKS5 support. From $2.20/GB with traffic rollover and cashback on used bandwidth. Trial at $3.50 for 750MB.

Get credentials from the dashboard at nodemaven.com/proxies/residential-proxies/ and paste into the PROXY_USER and PROXY_PASS variables in any of the examples above.

Proxy Rotation Strategies: When to Use Sticky vs Rotating Sessions in 2026

Elen Simonian — Mon, 17 Aug 2026 16:26:55 +0000

Choosing between sticky and rotating sessions is one of the most consequential proxy configuration decisions you will make. Get it wrong and you either break workflows that need session continuity, or waste IP diversity on tasks that do not need it. This guide covers the technical difference between the two modes, which use case requires which, and how to implement a Python session manager that switches between them programmatically.

What Sticky Sessions and Rotating Sessions Actually Do

A rotating session assigns a new IP address on every request, or at a set interval. The proxy infrastructure draws from the pool each time, so no single IP accumulates multiple requests to the same target. This maximizes IP diversity and is the standard mode for high-volume scraping.

A sticky session locks your traffic to the same IP address for a defined duration. The same IP handles every request within that session window, regardless of how many requests you make. This is essential for any workflow where the target site tracks session state across requests.

For a more detailed definition, NodeMaven’s glossary covers the concept at https://nodemaven.com/blog/rotating-proxy/.

NodeMaven residential proxies support both modes from the same pool of 30M+ IPs across 190+ countries. Sticky sessions last up to 24 hours on residential proxies. Mobile proxies support sticky sessions up to 24 hours. You switch between modes by changing the session parameters in your proxy credentials, not by using a different endpoint.

The Core Decision: Does Your Workflow Need Session Continuity?

The decision tree is straightforward:

When to Use Rotating Sessions

SERP Scraping

Search engine scraping is the canonical use case for rotating proxies. Sending hundreds of Google or Bing queries from the same IP triggers rate limiting quickly. Rotating per-request distributes the load across the IP pool, keeping individual IPs below the threshold that triggers blocks or CAPTCHAs.

NodeMaven’s residential pool of 30M+ IPs with a 95% clean IP rate means the IPs being rotated through are consistently low-fraud-score addresses. On SERP scraping, this directly affects how many requests succeed without triggering verification challenges.

Large-Scale E-commerce Scraping

Collecting product data, prices, or availability from Amazon, eBay, or retail sites at scale requires IP diversity. These platforms monitor request frequency per IP. Per-request rotation distributes the workload so no single IP builds up a suspicious request pattern.

Price Monitoring Across Many Pages

If you are checking thousands of product pages in a single run, rotating per-request keeps individual IPs from accumulating too many requests against the same domain in a short window.

Market Research Data Collection

Collecting public data from review platforms, job boards, or news sites at volume is a rotating proxy use case. The goal is throughput and coverage, not session continuity.

When to Use Sticky Sessions

Checkout Flows and Cart Abandonment Testing

E-commerce checkout flows are the clearest sticky session use case. From the moment you add a product to a cart to the point of completing a purchase, the platform tracks session state. A mid-flow IP change looks like a new session: the cart clears, authentication breaks, and the flow starts over.

A sticky session holds the same IP from the first page load through checkout completion. For testing checkout flows across different geo-locations, configure a sticky session per target region and run each test within the same session window.

Multi-Step Form Submission and Login Flows

Any workflow involving login, form submission across multiple pages, or state that persists between requests needs sticky sessions. The platform associates the session with the originating IP. An IP change mid-flow either breaks the session or triggers re-authentication.

Account Management and Social Media Workflows

Platforms like Instagram, Facebook, LinkedIn, and Google track login IP history. An account that logs in from the same residential IP every session looks like a real user. An account whose IP changes on every login looks automated.

For multi-account management, assign one sticky session per account and hold it for the full duration of that account’s activity cycle. NodeMaven’s 24 hours sticky sessions mean a single residential IP can cover a full work week of account activity without changing.

AI Agent Workflows with Authentication

AI agents that need to stay logged in to a service across multiple steps require sticky sessions. A rotating IP breaks authentication mid-task. NodeMaven’s residential proxies are used for AI agent workflows including ChatGPT, Claude, DeepSeek, and Gemini integrations, with sticky sessions enabling consistent task execution across multi-step runs.

Browser Automation with Selenium or Playwright

Browser automation that manages sessions, cookies, and authenticated state across multiple page visits needs sticky sessions. The browser maintains session state internally, but if the IP changes between requests, the server may invalidate the session.

NodeMaven residential proxies support integration with Selenium, Playwright, and Puppeteer. Configure a sticky session before starting the browser automation run and hold it for the full run duration.

Python Session Manager: Switching Between Modes

The following session manager handles both sticky and rotating sessions using NodeMaven's credential format. It manages session IDs, tracks active sessions, and handles rotation logic.


import requests

import time

import random

import string

from dataclasses import dataclass, field

from typing import Optional

 

PROXY_USER = "your_nodemaven_username"

PROXY_PASS = "your_nodemaven_password"

PROXY_HOST = "gate.nodemaven.com"

PROXY_PORT = "8080"

 

 

def random_session_id(length=8):

    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length))

 

 

@dataclass

class ProxySession:

    session_id: str

    country: str

    created_at: float = field(default_factory=time.time)

    request_count: int = 0

 

 

class ProxySessionManager:

    def __init__(self, country: str = "us", sticky_duration_hours: int = 24):

        self.country = country

        self.sticky_duration_seconds = sticky_duration_hours * 3600

        self.active_session: Optional[ProxySession] = None

 

    def _build_proxy_url(self, session_id: Optional[str] = None) -> str:

        user = f"{PROXY_USER}-country-{self.country}"

        if session_id:

            user += f"-session-{session_id}"

        return f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"

 

    def get_rotating_proxy(self) -> dict:

        """New IP on every call - no session ID."""

        proxy_url = self._build_proxy_url()

        return {"http": proxy_url, "https": proxy_url}

 

    def get_sticky_proxy(self) -> dict:

        """Same IP held for sticky_duration_hours."""

        now = time.time()

 

        # Create new session if none exists or current one has expired

        if (

            self.active_session is None

            or (now - self.active_session.created_at) > self.sticky_duration_seconds

        ):

            session_id = random_session_id()

            self.active_session = ProxySession(

                session_id=session_id,

                country=self.country,

            )

            print(f"New sticky session: {session_id}")

 

        self.active_session.request_count += 1

        proxy_url = self._build_proxy_url(self.active_session.session_id)

        return {"http": proxy_url, "https": proxy_url}

 

    def force_rotate(self):

        """Force a new sticky session on next get_sticky_proxy call."""

        self.active_session = None

        print("Session rotated.")

 

    def session_info(self) -> dict:

        if not self.active_session:

            return {"active": False}

        age_minutes = (time.time() - self.active_session.created_at) / 60

        return {

            "active": True,

            "session_id": self.active_session.session_id,

            "age_minutes": round(age_minutes, 1),

            "requests": self.active_session.request_count,

        }

 

 

def make_request(url: str, proxies: dict, headers: Optional[dict] = None) -> Optional[str]:

    default_headers = {

        "User-Agent": (

            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "

            "AppleWebKit/537.36 (KHTML, like Gecko) "

            "Chrome/124.0.0.0 Safari/537.36"

        )

    }

    if headers:

        default_headers.update(headers)

    try:

        response = requests.get(url, proxies=proxies, headers=default_headers, timeout=15)

        response.raise_for_status()

        return response.text

    except requests.exceptions.RequestException as e:

        print(f"Request failed: {e}")

        return None

 

 

# --- Usage examples ---

 

# 1. Rotating session: SERP scraping

rotating_manager = ProxySessionManager(country="us")

keywords = ["best project management software", "asana alternatives 2026"]

 

for kw in keywords:

    proxies = rotating_manager.get_rotating_proxy()

    # Each request gets a different IP

    print(f"Scraping: {kw}")

    # html = make_request(f"https://www.google.com/search?q={kw}", proxies)

    time.sleep(random.uniform(2, 5))

 

# 2. Sticky session: checkout flow testing

checkout_manager = ProxySessionManager(country="us", sticky_duration_hours=1)

checkout_steps = [

    "https://example-shop.com/product/123",

    "https://example-shop.com/cart/add",

    "https://example-shop.com/checkout",

    "https://example-shop.com/checkout/payment",

]

 

for step in checkout_steps:

    proxies = checkout_manager.get_sticky_proxy()

    # All steps use the same IP - session state preserved

    print(f"Step: {step} | Session: {checkout_manager.session_info()}")

    # html = make_request(step, proxies)

    time.sleep(random.uniform(1, 3))

 

# 3. Sticky session: account management (7-day window)

account_manager = ProxySessionManager(country="us", sticky_duration_hours=168)  # 7 days

# Each account gets its own manager instance with a unique session

account_sessions = {

    "account_1": ProxySessionManager(country="us", sticky_duration_hours=168),

    "account_2": ProxySessionManager(country="us", sticky_duration_hours=168),

    "account_3": ProxySessionManager(country="uk", sticky_duration_hours=168),

}

 

for account_name, manager in account_sessions.items():

    proxies = manager.get_sticky_proxy()

    print(f"Account: {account_name} | {manager.session_info()}")

    # Each account maintains its own IP throughout the week

Configuring Session Duration in the Dashboard

NodeMaven’s session configuration is handled through the dashboard rather than proxy string parameters. Log in, select your proxy type (residential or mobile), set your target location, and configure your session type and duration before copying your credentials in {host}:{port}:{username}:{password} format.

For residential proxies, sticky sessions are available up to 24 hours. For mobile proxies, sticky sessions are available up to 24 hours. For sessions beyond 24 hours on mobile or beyond 7 days on residential, ISP proxies with fixed 30 or 90-day IPs are the appropriate choice.

Practical Notes on Session Management

Sticky session expiry. When a sticky session expires, the next request automatically gets a new IP from the pool. In account management workflows, this means re-authentication. Plan session duration to cover your full activity cycle, or use ISP proxies for workflows that need identity consistency across weeks.

Rotating too aggressively on stateful sites. Sites that use CSRF tokens, multi-step forms, or server-side sessions will break if the IP changes mid-flow. Even if no login is involved, some sites tie session tokens to IP. When in doubt, use sticky sessions for any multi-step workflow.

One sticky session per account. For multi-account operations, never share a sticky session between two accounts on the same platform. Each account needs its own session ID and its own IP. The Python example above demonstrates this pattern with separate manager instances per account.

Quality guarantee. NodeMaven issues $1 in bonus traffic every time a proxy fails to perform. For sticky sessions where a failed IP disrupts a multi-step workflow, this guarantee reduces the cost of rare failures.

Getting Started

NodeMaven residential proxies support both rotating and sticky sessions from a pool of 30M+ IPs with a 95% clean IP rate, 99.54% average success rate, and under 0.6s average speed. Pricing from $2.20/GB with traffic rollover and cashback on used bandwidth. Trial at $3.50 for 750MB.

Configure session type and duration in the dashboard before starting your workflow. Details at nodemaven.com/proxies/residential-proxies/.

Bing Proxies in 2026: How to Scrape Bing SERP Without Getting Banned

Elen Simonian — Fri, 26 Jun 2026 13:22:22 +0000

Scraping Bing at scale sounds straightforward until you hit your first block. Unlike Google, Bing is less discussed in the scraping community, so people underestimate how seriously Microsoft takes automated traffic. The detection logic is different from Google, the rate limits behave differently, and the mistakes that get you blocked are specific to Bing’s infrastructure.
This guide covers why Bing blocks scrapers, how its detection differs from Google, and how to build a Python scraping pipeline with rotating residential proxies that actually holds up under load.

Why Bing Blocks Scrapers

Bing evaluates every incoming request against several signals simultaneously:
• IP reputation and ASN classification (residential vs datacenter vs hosting)
• Request frequency per IP over rolling time windows
• User-agent consistency and header patterns
• Cookie and session behavior across requests
• JavaScript challenge completion (on certain result pages)

The most common failure point is IP type. Datacenter IPs are classified immediately by their ASN - Microsoft’s infrastructure knows which IP ranges belong to cloud providers. Sending SERP requests from an AWS or Azure IP range gets you blocked within a handful of requests, often silently returning empty or altered results rather than an explicit error.
Residential IPs avoid this because they belong to real ISP-assigned addresses on consumer networks. Bing treats them the same way it treats organic traffic. The second most common failure point is request pacing - even residential IPs get flagged if they fire queries at machine speed with no variation.

Bing vs Google: Key Detection Differences

If you have experience scraping Google SERPs, a few things about Bing are worth knowing before you start:
Rate limits are less aggressive early, harder later. Bing tends to allow more requests before the first block, but once an IP is flagged it tends to stay flagged longer than Google’s equivalent cooldown periods.
CAPTCHAs appear less frequently. **Bing more often returns degraded or empty results without triggering a visible CAPTCHA. This means you can silently collect bad data if you are not validating response content.
**User-agent matters more. Bing’s bot detection is more sensitive to user-agent strings than Google’s. Stale or clearly automated user-agents get flagged faster. Rotating realistic user-agents alongside IPs is more important here.
Geo-targeting affects results significantly. Bing returns noticeably different results based on the cc and setlangparameters and the IP’s geographic location. For accurate local SERP data, the proxy location and the query parameters need to match.

Setting Up the Proxy Layer

For Bing SERP scraping, NodeMaven Bing proxies are residential IPs with a 95%+ clean rate across a pool of 30M+ addresses. The key spec for SERP work is the IP Quality Filter, which screens IPs before assignment. Typical unfiltered pools have a large share of IPs with blacklist history - those get caught immediately on Bing.
Relevant NodeMaven specs for this use case:
• 30M+ residential IPs across 190+ countries
• IP Quality Filter: roughly 95% of pool at low fraud scores
• Sticky sessions up to 7 days (useful for multi-page result crawls)
• ZIP-level and city-level geo-targeting in 190+ locations
• Success rate: 99.54% average
• Pricing from $2.20/GB, trial at $3.50 for 750MB

The proxy connection format uses HTTP or SOCKS5 with username/password auth. For rotating residential proxies, each new session string generates a new IP. For sticky sessions, the same session string holds the same IP for the duration you set.

Python Scraping Pipeline: Basic Setup

Install the required packages:
pip install requests beautifulsoup4 fake-useragent

Basic rotating proxy request:
import requests
import time
import random
from bs4 import BeautifulSoup

PROXY_USER = "your_nodemaven_username"
PROXY_PASS = "your_nodemaven_password"
PROXY_HOST = "gate.nodemaven.com"
PROXY_PORT = "8080"

def get_proxy(country="us", session_id=None):
user = PROXY_USER
if country:
user += f"-country-{country}"
if session_id:
user += f"-session-{session_id}-sesstime-60"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}

def get_useragent():
agents = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0",
]
return random.choice(agents)

def scrape_bing(keyword, country="us", page=1):
first = (page - 1) * 10
params = {
"q": keyword,
"first": first,
"cc": country.upper(),
"setlang": "en",
}
headers = {
"User-Agent": get_useragent(),
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
}
proxies = get_proxy(country=country)

try:
    response = requests.get(
        "https://www.bing.com/search",
        params=params,
        headers=headers,
        proxies=proxies,
        timeout=15,
    )
    response.raise_for_status()
    return response.text
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")
    return None

Parsing Bing SERP Results
Bing’s HTML structure is more stable than Google’s and less frequently updated. Organic results sit inside li elements with the class b_algo:
def parse_results(html):
if not html:
return []

soup = BeautifulSoup(html, "html.parser")
results = []

for item in soup.select("li.b_algo"):
    title_el = item.select_one("h2 a")
    snippet_el = item.select_one(".b_caption p")

    if not title_el:
        continue

    result = {
        "title": title_el.get_text(strip=True),
        "url": title_el.get("href", ""),
        "snippet": snippet_el.get_text(strip=True) if snippet_el else "",
    }
    results.append(result)

return results

def validate_response(html):
if not html:
return False
if len(html) < 5000:
return False
if "b_algo" not in html:
return False
return True
The validate_response function catches the silent failure case: Bing returning a short or empty page instead of a CAPTCHA. Always check that the response contains actual result elements before treating the request as successful.

*Adding Rate Control and Retry Logic
*
Request pacing is as important as IP rotation. Even clean residential IPs get flagged if they fire at machine speed.
import time
import random

def scrape_with_retry(keyword, country="us", page=1, max_retries=3):
for attempt in range(max_retries):
# Randomized delay between requests: 2 to 6 seconds
if attempt > 0:
wait = random.uniform(3, 8)
print(f"Retry {attempt}, waiting {wait:.1f}s")
time.sleep(wait)

    html = scrape_bing(keyword, country=country, page=page)

    if validate_response(html):
        return parse_results(html)

    print(f"Invalid response on attempt {attempt + 1}")

return []

def scrape_keyword_pages(keyword, country="us", pages=3):
all_results = []

for page in range(1, pages + 1):
    print(f"Scraping page {page} for: {keyword}")
    results = scrape_with_retry(keyword, country=country, page=page)
    all_results.extend(results)

    # Delay between pages: 3 to 7 seconds
    if page < pages:
        time.sleep(random.uniform(3, 7))

return all_results

*Geo-Targeting Bing Results by Location
*

Bing’s geo-targeting works through two mechanisms: the cc parameter in the query and the geographic location of the requesting IP. For accurate local results, both need to match.
NodeMaven supports targeting down to city and ZIP level. To get Bing results as they appear to a user in Seattle:
def get_proxy_geo(country="us", city=None, zip_code=None, session_id=None):
user = PROXY_USER + f"-country-{country}"
if city:
user += f"-city-{city.lower().replace(' ', '')}"
if zip_code:
user += f"-zip-{zip_code}"
if session_id:
user += f"-session-{session_id}-sesstime-30"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}

Seattle-specific results

proxies = get_proxy_geo(country="us", city="seattle", session_id="bing-seattle-01")
Pair this with cc=US&setlang=en in the query parameters and the response reflects what a Seattle user would see in Bing, including localized ad placements and map pack results.

*Scraping Bing vs Google: Practical Differences
*

*Running a Full Keyword Batch
*

Putting it together for a multi-keyword rank tracking run:
import json

keywords = [
"best project management software",
"project management tools 2026",
"asana vs monday comparison",
]

all_data = {}

for kw in keywords:
print(f"\nKeyword: {kw}")
results = scrape_keyword_pages(kw, country="us", pages=2)
all_data[kw] = results
print(f" Collected {len(results)} results")

# Delay between keywords: 5 to 12 seconds
time.sleep(random.uniform(5, 12))

with open("bing_serp_results.json", "w") as f:
json.dump(all_data, f, indent=2)

print(f"\nDone. Results saved to bing_serp_results.json")

*What to Watch For
*

Empty result pages. Always run validate_response() before parsing. Bing returns 200 status codes on blocked requests - the response just has no organic results.
Session reuse across keywords. For rank tracking where you want consistent geo, use the same sticky session string per location target. For bulk scraping where you want IP diversity, generate a fresh session ID per request.
Pagination limits. Bing typically returns results up to around page 10 before results degrade significantly. For most rank tracking use cases, pages 1-3 cover what matters.
Bing News and Bing Shopping. Different result types use different CSS selectors. The b_algo class covers standard organic results. News results use div.news-card, shopping results use li.b_shpItem. Scope your parser to the result type you need.

Getting Started
The NodeMaven residential proxies page covers pool specs and pricing. The $3.50 trial gives 750MB to validate the setup against your actual keyword targets before committing to a plan. For Bing SERP work, residential proxies from $2.20/GB with city-level geo-targeting give you everything the pipeline above needs.

How to Test if Your Proxy is Leaking DNS: 2026 Setup Guide

Elen Simonian — Sat, 30 May 2026 14:18:46 +0000

How to Test if Your Proxy is Leaking DNS: 2026 Setup Guide

You configured a proxy. Traffic is routing through it. IP check passes. Job done?

Not necessarily.

DNS leaks are one of the most common ways a proxy setup can expose your real location and ISP while appearing to work correctly. Your IP changes, but your DNS queries still go home.

This guide covers:

What a DNS leak actually is at the network level
How to detect one reliably
How to fix it across different proxy configurations
Programmatic detection methods with code examples

What Is a DNS Leak?

When you type a domain into your browser, a DNS resolver translates it into an IP address.

Normally this happens through your ISP's DNS server. When you use a proxy, the expectation is that DNS resolution also happens inside the proxy network rather than through your local ISP.

A DNS leak occurs when your operating system or application bypasses the proxy and sends DNS queries directly to your ISP's resolver.

The result:

The proxy hides your IP from the destination website
Your ISP still sees every domain you're resolving
Anyone monitoring DNS traffic can reconstruct your browsing activity regardless of what proxy you're using

This matters most for

Automation and scraping jobs where geo-accurate DNS is required for localized responses
Multi-account workflows where a mismatched DNS origin can trigger platform-side identity checks
Privacy-sensitive workflows where DNS history is the actual attack surface

Why DNS Leaks Happen

DNS leaks are usually not a misconfiguration on the proxy provider's side.

They happen because of how OS-level DNS resolution works.

Most operating systems have a DNS resolver that runs independently of application-level proxy settings.

When you configure an HTTP or SOCKS5 proxy in a browser or script, the proxy handles connection routing, but the OS resolver may still handle DNS separately depending on the application and proxy type.

Common causes

HTTP proxies handle DNS on the proxy server by default, but some clients resolve DNS locally first and then send the IP to the proxy
SOCKS4 does not support remote DNS resolution
SOCKS5 supports remote DNS resolution, but only if the client explicitly requests it
WebRTC in browsers can expose local DNS even when a proxy is active
Some automation frameworks use the system DNS resolver instead of routing through the proxy
Split DNS configurations on corporate or managed networks

How to Test for a DNS Leak

There are several reliable ways to verify whether your DNS requests are leaving the proxy tunnel.

Method 1: Manual Test with a DNS Leak Test Tool

The fastest way to check is to use NodeMaven's DNS leak test tool while your proxy is active.

It works by making multiple DNS requests to unique subdomains and checking which DNS server resolved them.

Steps

Configure your proxy in your browser or system settings.
Navigate to the DNS leak test tool with the proxy active.
Run the extended test.
Check the resolver IPs in the results.

Result interpretation

If any resolver belongs to your ISP or home network rather than the proxy provider's infrastructure, you have a DNS leak.

Method 2: dig or nslookup via the Proxy

For terminal-based testing, you can route a DNS query through the proxy using dig with a SOCKS5 proxy:

# Test DNS resolution through a SOCKS5 proxy
# Replace with your NodeMaven proxy credentials and host

dig @8.8.8.8 whoami.akamai.net +short

# Route the query through the proxy using proxychains
proxychains dig whoami.akamai.net +short

# Compare the two outputs
# If they return different IPs, your DNS is routing through the proxy
# If they return the same IP, you have a potential DNS leak

You can also use curl to hit a DNS echo service:

# Check your apparent DNS resolver through the proxy

curl --proxy socks5h://username:password@proxy.nodemaven.com:9999 \
  https://whoami.akamai.net

# Note the 'h' in socks5h - this forces remote DNS resolution
# socks5:// resolves DNS locally
# socks5h:// resolves DNS on the proxy server

Important

The socks5h:// vs socks5:// distinction is one of the most common sources of DNS leaks in automation scripts.

Method 3: Python Script for Programmatic Detection

If you're running automated workflows, you can build DNS leak detection into your setup verification:

import requests
import json

def check_dns_leak(proxy_url: str) -> dict:
    """
    Check for DNS leaks by comparing DNS resolvers visible
    with and without the proxy.
    """

    proxies = {
        "http": proxy_url,
        "https": proxy_url,
    }

    ...