DEV Community

ProxyMaster
ProxyMaster

Posted on

Setting Up Proxies in Python Requests and httpx

#webscraping #python #proxy

When your script sends every request from the same address, servers notice. A scraper, a monitoring job, or a bulk API caller that fires hundreds of requests from one IP quickly hits rate limits, CAPTCHAs, or a flat 403. The fix is to route traffic through a proxy so the target sees a different address, and to rotate that address so no single one carries the whole load. This post shows how to do that in both requests and httpx, how to check it worked, and how to keep your jobs from getting blocked.

Setting proxies in requests

The requests library reads proxies from a dictionary keyed by scheme. Each value is a full proxy URL, and you can embed credentials directly in that URL as user:pass@host:port.

import requests

proxies = {
    "http": "http://USER:PASS@proxy.host:8080",
    "https": "http://USER:PASS@proxy.host:8080",
}

r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=15)
print(r.json())
Enter fullscreen mode Exit fullscreen mode

Note that both keys point at the same HTTP proxy. That is normal: the https key describes the scheme of the destination, not the proxy itself. For SOCKS5, install the extra dependency with pip install requests[socks], then use the socks5h:// scheme so DNS resolves on the proxy side rather than locally.

Setting proxies in httpx

httpx uses a proxy argument (or a mounts mapping for finer control). The auth format is the same inline user:pass@host:port. Because httpx supports both sync and async clients, the pattern below works in scripts and inside async workers alike.

import httpx

proxy = "http://USER:PASS@proxy.host:8080"

with httpx.Client(proxy=proxy, timeout=15) as client:
    r = client.get("https://api.ipify.org?format=json")
    print(r.json())
Enter fullscreen mode Exit fullscreen mode

For SOCKS5 in httpx, install pip install httpx[socks] and pass socks5://USER:PASS@proxy.host:1080 as the proxy value. The client handles the handshake for you.

Verifying the IP actually changed

Never assume the proxy took effect. Send one request without the proxy and one with it to the same echo endpoint, then compare. Services like api.ipify.org, ifconfig.me/ip, or httpbin.org/ip return the address the server sees. If both calls return your real address, the proxy dictionary key, the scheme, or the credentials are wrong. A quick assertion in your setup code catches this before a long job wastes hours pulling from the wrong address.

Handling rotation and avoiding blocks

Blocks usually come from volume per address, not from proxies as a concept. A few practices keep jobs healthy:

  • Rotate addresses across requests so no single IP carries the full request count.
  • Add small randomized delays and cap concurrency instead of hammering at full speed.
  • Set a realistic User-Agent and reuse a session so cookies and connections stay consistent.
  • Retry on 429 and 5xx with backoff rather than looping instantly.

With a rotating pool, each call can leave from a fresh address automatically, which spreads load and sidesteps per-IP thresholds without you managing a list by hand.

Where to get proxies

or this to work you need reliable upstream addresses. WinGate proxies for Python provide private IPv4 and SOCKS5 with rotation, a worldmix pool, unlimited traffic, and support for HTTP, HTTPS, and SOCKS5, which covers every setup shown above. High concurrency jobs are supported with up to 5,000 threads, so bulk scraping and monitoring stay fast.

You can confirm the code works against real endpoints using the free 2-hour test before committing to anything. Wire the credentials into the snippets above, run the IP check, and you have a proxied Python client ready for production.

Top comments (0)