DEV Community

Shifter
Shifter

Posted on AI-assisted

Rotating vs sticky residential proxy sessions: how to choose, with Python examples

#webscraping #python #tutorial #networking

Disclosure: this post comes from the Shifter team. Shifter is a residential proxy provider, and the examples use its gateway, but the ideas apply to any provider that lets you choose how long you keep an exit IP.

When you send traffic through a residential proxy gateway, you get an exit IP from a large pool. The first decision is simple: should the next request come from a new IP, or from the same one as the last request? That is the difference between a rotating session and a sticky session.

When to rotate

Use a fresh IP for every request when requests do not depend on each other. Typical cases:

  • Fetching many independent pages, such as product pages or search result pages.
  • Collecting public data across a long list of URLs.
  • Any job where no cookie or login state has to survive between requests.

Spreading independent requests over many IPs keeps the request rate per IP low, which is what most sites look at first.

When to stick

Keep the same IP for a while when the site ties state to your connection. Typical cases:

  • Logging in, then reading pages that need the login.
  • Carts, checkouts and multi-step forms.
  • Pagination that depends on a session cookie or token.

A sticky session holds one IP for a time to live (TTL). Pick a TTL a little longer than the whole flow, so the IP does not change halfway through.

How it looks in practice

With the Shifter gateway, the options go into the proxy username. The gateway is p.shifter.io:443, and the username takes these parameters:

Parameter Example Effect
country-XX customer-USERNAME-country-de Route through Germany
region-NAME customer-USERNAME-region-california Target a specific region
city-NAME customer-USERNAME-city-new_york Target a specific city
asn-NUMBER customer-USERNAME-asn-7922 Target a specific ASN
sid-ID customer-USERNAME-sid-abc Sticky session (same IP for the default TTL of 120 seconds)
ttl-SECONDS customer-USERNAME-sid-abc-ttl-600 Custom TTL in seconds (requires sid)
strict-true customer-USERNAME-country-us-strict-true Fail with 502 instead of falling back to a broader pool

No sid means a new IP for each request. Adding a sid means a sticky session.

Rotating: a new IP per request 

import requests

proxy = "customer-USERNAME-country-us:PASSWORD@p.shifter.io:443"
proxies = {"http": proxy, "https": proxy}

for page in range(1, 4):
    r = requests.get("http://ip-info.com/json", proxies=proxies)
    print(page, r.text)
Enter fullscreen mode Exit fullscreen mode

Sticky: one IP for 10 minutes 

import uuid
import requests

sid = uuid.uuid4().hex[:8]  # one id per job or per account
proxy = f"customer-USERNAME-country-us-sid-{sid}-ttl-600:PASSWORD@p.shifter.io:443"
proxies = {"http": proxy, "https": proxy}

session = requests.Session()
session.proxies.update(proxies)

# every request in this session leaves from the same IP until the TTL ends
print(session.get("http://ip-info.com/json").text)
print(session.get("http://ip-info.com/json").text)
Enter fullscreen mode Exit fullscreen mode

To check it yourself, run the sticky example twice with the same sid and compare the output, then change the sid and compare again.

A few habits that save time

  • Use one sid per account or per job, never one shared sid for everything.
  • Set the TTL to cover the whole flow, and start a new sid when a flow ends or fails.
  • Add strict-true when you must not fall back to a wider geography. You get a clear 502 instead of results from the wrong place.
  • Keep concurrency per target site reasonable, and only collect public data in line with the site's terms.

Try it

Shifter's residential proxies cover 195+ countries, and there is a free trial at shifter.io if you want to run the examples above.

Top comments (0)