DEV Community: Proxy-Seller

How to Route Your Python Requests Through a Proxy for Reliable Scraping

Proxy-Seller — Wed, 25 Mar 2026 14:06:30 +0000

Have you ever thought of how much stays hidden in the Web? The challenge of dragging out the publicly available information is enormous to the developers nowadays. When you fire off multiple queries from one machine, servers quickly flag and block your actual IP address. Using Python requests through a proxy works like a middleman to help you operate smoothly. This guide walks you through how to import the necessary tools and put together a stable script.

TL;DR Summary of Quick Integration

To get moving with Python requests through a proxy, just pass a proxies dictionary into your requests.get() call. Use a Session() object to keep a persistent connection active and boost your speed. Always provide authenticated credentials for any private nodes you use. If you want to avoid the "too many requests" (HTTP 429) error during heavy web scraping, implement the requests + rotating proxy logic using a custom list or a backconnect provider.

Prerequisites

Ready-to-use Python 3.8.
The requests library (pip install requests).
A live URL for testing (e.g., https://httpbin.org/ip).
Access to a private proxy or a shared node list.

Basic Setup: How to Use Python Requests to Set a Proxy

Most of the time, when learning about how Python requests use proxy parameters, developers begin with a simple dictionary. This tells the library exactly which gateway to route HTTP or HTTPS traffic through.

import requests

# Dictionary mapping protocols to proxy URLs
proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "http://10.10.1.10:1080",
}

# Standard GET request
response = requests.get("https://httpbin.org/ip", proxies=proxies)
print(response.json())

Is such a simple matter all the time? While this works for quick tasks, it forces a new TCP handshake for every single call. That puts a lot of delay in your program.

Performance Tuning Using a Python Requests Session Proxy

Would you rather things should run faster? A session object keeps the same connection open for every following call. This is a vital settings tweak for anyone serious about gathering data at scale.

import requests

session = requests.Session()
session.proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "http://10.10.1.10:1080",
}

# The session persists proxy settings automatically
resp = session.get("https://httpbin.org/ip")
print(resp.text)

Configuring Python Requests Proxy Authentication

The majority of the quality sources require a username and password. You can bake these details right into the string you pass. This is the standard Python requests proxy example you'll see in professional codebases.

The format is http://user:pass@host:port.

import requests

# Example with user credentials
proxies = {
    "http": "http://user123:pass456@192.168.1.1:8080",
    "https": "http://user123:pass456@192.168.1.1:8080",
}

# Sending the authenticated request
response = requests.get("https://httpbin.org/ip", proxies=proxies)

If your password uses special characters, make sure to use URL encoding. Plenty of developers overlook this and end up with Python requests proxy authentication errors. You can also lean on environment variables to keep your login info out of the main script.

Comparison of Proxy Types

Automated traffic constitutes 37% of web traffic. Websites are always in search of nonhuman behavior.

Type	Success Rate	Cost	Best For
Datacenter	45–60%	$0.50–$1.50/IP	Basic sites and testing
Residential	95–99%	$3.00–$15.00/GB	High-protection sites
ISP / Static	90–97%	$2.00–$5.00/IP	Social media

Datacenter IPs are cheap yet are easily identified. Residential IPs reach a 99% success rate, but they are relatively costly. For a sweet spot in the middle, look into ISP proxies — they provide the appearance of a domestic user and the processing speed of a server farm. If your target supports newer protocols, IPv6 proxies can drop your costs while giving you a massive pool of addresses to work with.

Rotation Logic for Higher Success Rates

It is not wise to stay on the same IP. You need Python requests and a rotating proxy. You can cycle through a local list or let a rotating provider handle the heavy lifting.

Manual rotation: Have an array of IPs and use random.choice().
Backconnect rotation: Use a single entry point that replaces the exit IP automatically.
ISP: These appear very authentic to sensitive targets.
IPv6: A cheap source of thousands of new IPs for modern sites.

Troubleshooting Your Python Requests Proxy Connection

Running into a ProxyError? Usually, that means the server is offline or the URL you typed has a typo.

Check the protocol: Does the node actually support HTTPS? Many free ones are HTTP-only.
Check credentials: Ensure that your usernames and passwords have no missing characters.
Fine-tune timeouts: There should always be a timeout=10 argument. Without it, your script might just sit there forever on a dead link.
Local Firewall: Ensure your own network settings aren't blocking the port the proxy uses.

Why Python Requests Through a Proxy Fail

Occasionally a new IP address is not sufficient. Servers also peek at your user agent and your TLS fingerprint. If your headers look like they came from a library, you'll get kicked even with a private proxy. Always drop a real-looking user agent into your settings.

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}
response = requests.get(url, proxies=proxies, headers=headers)

Summary of Best Practices

Using Python requests through a proxy the right way keeps your data flowing without a hitch. Steer clear of free lists — they're incredibly slow and often snoop on your traffic. Invest in a solid provider if you're doing serious work.

Use requests.Session() for better speed.
Rotate IP addresses to stay below the radar.
Associate your proxies with non-toxic-looking headers.

Things to Read Next

Ready to dive deeper? Review the official Requests documentation for edge cases. If you're hitting sites loaded with JavaScript, you might want to switch gears to Playwright or Selenium. Refer to the Requests GitHub repository to see how the library handles proxy logic internally. You could even look into how urllib3 manages the nuts and bolts if you need to build something totally custom.

Selenium and a Proxy in Python: A Practical Automation Guide

Proxy-Seller — Mon, 09 Mar 2026 12:16:55 +0000

Your script runs, gets to a point, and stops in mid-flow. Why does this happen? Systems limit repeated sessions originating within the same source, and they actively inspect your hardware to avoid overloading your server when it is at peak times. Implementing Selenium and a proxy in Python fixes this exact problem. It routes your commands via alternative hardware. You get better link stability and lower ping to far endpoints.

The Cloudflare Radar Report states that about 30 percent of all internet requests are automated bots. This is expected on platforms. However, they drop sessions that congest their infrastructure. A proxy distributes the load efficiently, optimizing your network pathways so that remote servers perceive your machine as a regular human visitor. This guide shows how to set a proxy in Selenium WebDriver Python correctly.

Rapid Code to Rapid Integration

Need an immediate solution? Use the selenium-wire add-on. It handles authentication better than the default WebDriver. The basic Python Selenium proxy approach requires just a few lines.

from seleniumwire import webdriver

options = {
    'proxy': {
        'http': 'http://user:pass@192.168.1.1:8080',
        'https': 'https://user:pass@192.168.1.1:8080',
    }
}
driver = webdriver.Chrome(seleniumwire_options=options)
driver.get("https://example.com")

Reasons Why Standard Frameworks Abandon Sessions

Simple automation transmits all the commands using a single local identifier. Hosts notice this at once. They choke your pace of execution.

You need a different IP address to distribute requests. A well-built architecture using Selenium and a proxy in Python mimics normal human pathways. It prevents timeouts.

But what about the right alternate hardware to choose? There are a number of levels available in the market. Each has distinct traits for automation testing.

Hardware Type	Average Cost	Best Use Case
Datacenter	$0.50–$2.00/month	High-speed requests
Residential	$3.00–$15.00/GB	Complex platforms
Mobile	$30.00–$60.00/month	Social applications

Selecting the Right Proxy Type

Datacenter addresses are inexpensive and fast. They are a part of cloud hosts. Some platforms drop them. If you want stable performance, residential proxies offer links from real home internet providers. They resemble ordinary consumer lines.

Sometimes you need cellular connections. Mobile IPs provide excellent path stability. They are identifiable on most cellular devices and are very difficult to throttle when they are being loaded heavily.

For high-speed corporate jobs, ISP options give you datacenter speeds with residential trust scores. And if you target modern infrastructure, IPv6 proxies improve packet efficiency for newer web pages.

Integrating With Chrome

The market is dominated by the Google client. The engine is quick and is well supported. You can add a proxy to Selenium Python configurations using the ChromeOptions class. This specific proxy Selenium Python Chrome method works for IP-authenticated nodes.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

target_node = "1.2.3.4:8080"
chrome_options = Options()
chrome_options.add_argument(f'--proxy-server={target_node}')

driver = webdriver.Chrome(options=chrome_options)
driver.get("https://example.com")

This snippet directs all network traffic through the designated IP. The core of any project based on Selenium and a proxy in Python relies on this precise routing mechanism.

Configuring Firefox

Mozilla has another internal architecture. The Firefox browser requires profile modifications instead of command-line arguments. There are certain capability keys you have to set.

from selenium import webdriver

profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", "1.2.3.4")
profile.set_preference("network.proxy.http_port", 8080)
driver = webdriver.Firefox(firefox_profile=profile)

This forces the client to route web traffic correctly. It works with simple IP authentication with ease.

Managing User Credentials

Simple automation tools lack native support of credentials in the URL format. A popup usually comes up requesting a username. To fix this, you need a Selenium proxy authentication Python workaround. A temporary extension is constructed by many engineers on the spot.

You can also use an API key with third-party extensions. The selenium-wire package mentioned earlier intercepts HTTP requests. It inserts the credentials in the header. This use of a proxy in the Selenium Python technique saves hours of debugging.

Establishing a Swapping System

The static nodes ultimately reach rate limits. A Python Selenium and rotating proxy strategy will switch your outward-facing identifier every so often. This ensures stability in execution when doing large jobs.

A rotating gateway is available as a purchase. This one endpoint automatically changes the primary IP. It makes Selenium rotating proxy Python integration incredibly simple. You just point your scripts at one address.

Creating The Rotation Logic

Suppose you have a list of individual numbers? You have to turn them yourself. The script with rotating proxies and Selenium in Python picks a new address for every session. This distributes the load perfectly for heavy web scraping tasks.

import random
from selenium import webdriver

node_list = ["ip1:port", "ip2:port", "ip3:port"]
selected_node = random.choice(node_list)

options = webdriver.ChromeOptions()
options.add_argument(f'--proxy-server={selected_node}')
driver = webdriver.Chrome(options=options)

This Python Selenium rotating proxies tactic requires a loop. You start a new client with a new address every time.

Pre-Launch Checklist

Before executing your scraping tasks, verify your environment. Minor mistakes lead to huge memory leaks.

Test the endpoint manually.
Determine whether the port needs SOCKS5 or HTTP protocols.
Verify your third-party library versions.
Check your subscription bandwidth.

These steps keep your framework on Selenium and a proxy in Python stable.

Dealing with Common Process Errors

The ERR_PROXY_CONNECTION_FAILED message is very common. This occurs when the node becomes unavailable. The port is sometimes blocked by the firewall. You have to catch these exceptions.

The other common error is the TimeoutException. The remote host may be too slow. You need to raise the page load time.

driver.set_page_load_timeout(30)

Sometimes the standard requests library connects fine, but the client fails. JavaScript is more bandwidth-consuming to render. A Selenium Python proxy setup always consumes more data than simple text commands.

Setting up the Development Environment

Isolate your dependencies before you write any code. A virtual environment eliminates the conflicts among various projects. It keeps your packages clean.

python -m venv myenv
source myenv/bin/activate
pip install selenium selenium-wire

It is a common best practice to work within a virtual wrapper. It ensures that your codebase is reproducible. When you share your repository with another engineer, they can install the same dependencies immediately.

Finding Page Elements Effectively

After your page has loaded successfully, you have to extract the text. Avoid using absolute XPaths. They are broken each time the site changes its layout.

Instead, use CSS selectors or ID attributes. They offer far greater resistance to front-end changes.

from selenium.webdriver.common.by import By

try:
    title_element = driver.find_element(By.CSS_SELECTOR, "h1.main-title")
    print(title_element.text)
except Exception:
    print("Element not found.")

This is a focused method that accelerates your data gathering. It does not allow your logic to crash when making small visual changes.

Exporting Collected Results to Local Storage

It is okay to print outputs to the console to test. But production work needs to be organized in storage. CSV files provide a simple text-based storage format.

import csv

data_row = ["Product A", "$19.99", "In Stock"]

with open('output.csv', mode='a', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(data_row)

Appending rows eliminates data loss instantly. In case of a failure during execution, you retain the information gathered before.

Dealing with Automated Visual Challenges

Numerous sites offer visual riddles to authenticate human users. These are not solvable with standard frameworks. But you can detect them.

if "captcha" in driver.page_source.lower():
    print("Challenge detected. Pausing execution.")
    driver.save_screenshot("challenge.png")

A screenshot will assist you in knowing what caused the block. You may then change the frequency of your request or rotate your identifier.

Timing Your DOM Interactions

Never use time.sleep(). Hardcoded pauses consume huge amounts of execution time. They also fail when the remote server is slow.

Explicit waits only hold the logic until a certain component is visible.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "target-data"))
)

This is the best way to maximize your processing speed. The client proceeds to the next step the very moment the element is available.

Installing Advanced Console Logging

Remote debugging is a challenge. The screen is not visible in headless mode. You have to depend on console outputs. Install simple logging to monitor your progress.

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

try:
    driver.get("https://example.com")
    logger.info("Page loaded successfully.")
except Exception as e:
    logger.error(f"Failed to load: {e}")

This monitors failed addresses. Then you can automatically remove bad nodes from your pool.

Memory Management Techniques

Constant execution leads to memory leakage. The driver process slowly consumes RAM. If you run Selenium and a proxy in Python for days, your machine might crash.

You have to kill the driver process. Do not just close the window.

driver.quit()

The quit() method ends the background service. It frees up available RAM. Reboot your complete logic after every 24 hours. This maintains a clean environment.

Header Information Management

Endpoints do not only check your IP. They look at your user agent. They analyze your language preferences. A default tool transmits different headers — it is almost a proclamation of being a machine.

These headers need to be changed. Compare them to a typical consumer browser. Combine this trick with Selenium and a proxy in Python for maximum stability.

options.add_argument('--user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)..."')
options.add_argument('--accept-language="en-US,en;q=0.9"')

This forms a plausible online presence. It reduces the possibility of throttled sessions.

Developing a Pre-Flight Health Check System

Do not pass important tasks over an untested node. Establish a little pre-flight ritual. Ping a basic endpoint that echoes your active address.

driver.get("https://api.ipify.org")
current_ip = driver.find_element("tag name", "body").text
print(f"Current active address: {current_ip}")

In case this output equals your local machine's IP, the configuration was unsuccessful. Abort the execution. Do not continue with the main job.

Scaling Your Automated Operations

Having ten browsers running simultaneously kills CPU performance. You need to optimize the environment. The graphical interface is disabled in headless mode. It conserves large quantities of RAM.

options.add_argument("--headless=new")

In the case of enterprise work, engineers package their code. Docker isolates every client instance. It allows you to run a Selenium and a proxy in Python script across multiple virtual machines.

You are able to map certain nodes to certain containers. This isolates system failures. When one container falls, the rest continue running.

Advantages and Disadvantages of Proxy Integration

A private proxy makes your codebase more complicated. However, the advantages tend to prevail over the disadvantages.

System Feature	Major Pros	Notable Cons
Path Stability	Optimizes local network paths	Adds slight latency
Traffic Distribution	Balances server loads	Increases code complexity
Execution Speed	Allows parallel processing	Requires paid subscriptions

Make your architecture as simple as you can. Unless it is necessary, do not over-engineer the rotation logic.

Concluding Remarks on the Framework

Serious engineers must optimize their networks. Using Selenium and a proxy in Python improves connection quality drastically. Begin with basic datacenter IPs. Then upgrade to advanced networks as your project expands.

Always handle exceptions well. Clear your cache between sessions. And keep track of your memory.

What are IPv6 proxies?

Proxy-Seller — Thu, 05 Mar 2026 13:30:58 +0000

Why Every Developer Should Understand the IPv6 Proxy

The issue of IP exhaustion is common among developers when they are scaling their web scrapers or other automated tools. However, with the shift away from older technologies, the IPv6 proxy has become an essential component for backend programmers and data scientists. This article will walk you through how these tools operate within the new paradigm of the network stack.

TL;DR

An IPv6 proxy server uses the 128-bit internet protocol for routing traffic.
There is a significantly larger pool of IP addresses provided compared to the older IPv4 protocol.
These servers are best used for tasks like web scraping, SEO monitoring, and even social media automation.
The deployment is highly cost-effective because providers offer large IP blocks like /64 for cheap.

Prerequisites

To keep up with the technical examples given in the article, you must have the following:

A VPS based on Linux such as Ubuntu 22.04 or Debian 11/12.
General understanding of the terminal and the command line.
Familiarity with either Node.js or Python development.
A dedicated subnet from a provider.

What is IPv6 and why is it necessary?

The older IPv4 protocol allows for only 4.3 billion IPs. On the other hand, the new 128-bit protocol allows for 3.4 × 10³⁸ possible IPs. This is required since the number of devices that are connected to the internet is increasing exponentially.

As per the official Google statistics, IPv6 has already been adopted by 49 percent of the world as of late 2025. That is why programmers have to adjust to the new paradigm. IPv6 proxy servers utilize the new space to avoid the problem of IP exhaustion.

The protocol header structures have been made more efficient by the new protocol. The reason is that the older protocol uses a checksum per hop. The new protocol is configured differently.

Comparing IPv4 and IPv6

The choice of the best protocol will be based on your infrastructure needs. The table below compares the two protocols to help you select the most suitable one for your current project.

	IPv4	IPv6
Space	32-bit (limited)	128-bit (nearly infinite)
Availability (supply)	Scarce	Vast
Transfer speed	Low speed	High speed
Price	High cost	Low cost
Protocol Logic	Dependence on NAT for complete functionality	Direct connections between two endpoints
Compatibility	Universal site integration	Support for IPv6-enabled sites
Risk of Blocking	High (shared IPs)	Low (unique /64 subnets)

Practical Implementation

Most developers begin with IPv6 private proxies, ensuring that their automation scripts remain completely isolated from the rest of the system. This gives the benefit of total control as opposed to other IPs which can be shared with other users.

Squid may be the best tool to use in case your application is the creation of a custom caching system. This gives you the opportunity to attach your outgoing connections to particular endpoints in your allocated IP block.

Configuring a Rotating IPv6

The following squid.conf configuration demonstrates the process of rotating requests to various unique IPs:

# Define authentication
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwords
acl authenticated proxy_auth REQUIRED
http_access allow authenticated

# Port for incoming requests
http_port 3128

# Bind outgoing requests to unique IPv6 addresses
acl dev_set_1 localip 2001:db8:1::10
tcp_outgoing_address 2001:db8:1::10 dev_set_1

acl dev_set_2 localip 2001:db8:1::11
tcp_outgoing_address 2001:db8:1::11 dev_set_2

# Filter out identifying headers
request_header_access X-Forwarded-For deny all
request_header_access Via deny all

If your application does not require the complexity of setting up your own servers, acquiring a dedicated IPv6 proxy can provide the most appropriate solution. These are often the best IPv6 proxies for teams that prefer to focus on the application logic rather than the server setup.

Scaling with Microservices and Docker

SaaS teams tend to run their scrapers in containers, and to get 128-bit routing, the Docker daemon will need to be changed.

Edit /etc/docker/daemon.json to include the following:

{
  "ipv6": true,
  "fixed-cidr-v6": "2001:db8:abc:1::/64"
}

There is a possibility of unlimited bandwidth, particularly when paired with premium hosting. This leads to reliable performance potential during peak data collection cycles. Some setups also use residential proxies for higher trust scores.

Working with Node.js

If your application involves the development of a custom backend system, using an IPv6 proxy will require specific configurations on the agent. The most popular configuration uses the axios library and the https-proxy-agent package.

const axios = require('axios');
const { HttpsProxyAgent } = require('https-proxy-agent');

// Standard 128-bit address format for the endpoint
const proxyUrl = 'http://user:pass@[2001:db8:85a3::8a2e:370:7334]:8080';
const agent = new HttpsProxyAgent(proxyUrl);

async function checkConnection() {
  try {
    const res = await axios.get('https://api64.ipify.org?format=json', {
      httpsAgent: agent
    });
    console.log('Active IP:', res.data.ip);
  } catch (err) {
    console.error('Request failed:', err.message);
  }
}

checkConnection();

The square brackets in the address are compulsory in the URI string. In their absence, the colons in the identifier will probably be interpreted by the parser as a port separator.

Operational Strengths and Weaknesses

The use of an IPv4 proxy is sometimes required, especially for older websites. Nonetheless, the 128-bit subnet transition has its benefits particularly in managing multiple accounts.

The Benefits

Cost Efficiency: A subnet of /64 (millions of IPs) can be purchased at less than $20/month.
Scale: High-volume web scraping benefits greatly from the virtually limitless IPs available.
Low Interference: The activities of other users will not affect reputation because the subnet is owned.
Fewer Blocks: The majority of platforms will consider a /64 subnet as one residential-type entity.

The Challenges

Target Support: Not every target is currently enabled with the new protocol.
Routing Problems: When a 128-bit subnet is not configured correctly, it is likely to cause Destination Unreachable errors.
Hardware Limitations: 128-bit traffic will not be natively supported by some older routers and VPS nodes.

Community Utilities and Resources

A number of repositories have been developed to facilitate this process:

Script on Linux distribution: Script to immediately convert a VPS into a huge pool of gateways.
TREVORproxy: Best suited to Python developers who require rotating source IPs.
Squid Documentation: The ultimate guide to server optimization.

Final Considerations

Is an IPv6 proxy suitable for your project? The answer is most likely yes, as long as the websites that you are targeting have this modern protocol. The cost savings alone make this a great option for startups or large-scale data teams.

It is a good thing to test your code as soon as you can. Use a small subnet to make sure that your code correctly interprets 128-bit headers.

Next Steps

Check the presence (or absence) of AAAA record support on your site with a tool like dig.
Install a simple Squid server to test local routing.
Test your automation scripts to have 128-bit support.