Proxy Authentication & Session Persistence in Python: Sticky Sessions, Rotating Sessions, and When to Use Each

#python #webscraping #tutorial #requests

Most proxy tutorials cover one thing: how to rotate. Fewer cover the harder question: when not to rotate. If you're logging into a site, filling a multi-step form, or building up a cart before checkout, swapping your IP mid-flow can look exactly like account takeover to the site you're hitting, and you'll get flagged or logged out.

This post covers both authentication mechanics and the rotating-vs-sticky decision, since they usually come up together. Code samples below use <PROXY_HOST>:<PROXY_PORT> placeholders, swap in credentials from whatever provider you're using; the examples assume a standard user:pass@host:port format, which is what most providers expect.

Two ways to authenticate a proxy

Credentials in the URL is the simplest and most common approach:

import requests

proxy = "http://username:password@<PROXY_HOST>:<PROXY_PORT>"
proxies = {"http": proxy, "https": proxy}

resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
print(resp.json())

requests parses the username:password@ portion automatically and handles the handshake for you.

The Proxy-Authorization header is the alternative, and some proxy providers or corporate setups require it explicitly rather than accepting inline credentials:

import base64
import requests

def build_proxy_auth_header(username, password):
    credentials = base64.b64encode(f"{username}:{password}".encode()).decode()
    return {"Proxy-Authorization": f"Basic {credentials}"}

proxy = "http://<PROXY_HOST>:<PROXY_PORT>"
proxies = {"http": proxy, "https": proxy}
headers = build_proxy_auth_header("username", "password")

resp = requests.get("https://httpbin.org/ip", proxies=proxies, headers=headers, timeout=10)

Functionally these do the same thing, both send Basic auth to the proxy, but if inline credentials ever silently fail against a particular proxy setup, the header version is the fallback worth trying.

Rotating vs. sticky: the actual decision

	Rotating (new proxy per request)	Sticky (same proxy for a duration)
Use for	Scraping many independent pages, price checks, rank tracking	Logins, multi-step checkouts, anything with server-side session state
Why	Spreads load, avoids per-IP rate limits	Consistent IP across a flow avoids triggering "session hijack" style detection
Risk if you get it backwards	Sticking to one proxy for bulk scraping gets that IP rate-limited fast	Rotating mid-login or mid-checkout gets the session invalidated or flagged

The rule of thumb: rotate when each request stands alone, stay sticky when requests belong to the same logical session.

Building a sticky session manager

The pattern below maps an arbitrary "session key" — a user ID, task ID, or account handle, whatever makes sense for your workload, to a fixed proxy for a configurable time-to-live. Once the TTL expires, it gets reassigned:

import time
import random

class StickySessionManager:
    def __init__(self, proxies, ttl_seconds=600):
        self.proxies = proxies
        self.ttl_seconds = ttl_seconds
        self._assignments = {}  # session_key -> (proxy, assigned_at)

    def get_proxy(self, session_key):
        now = time.time()
        assignment = self._assignments.get(session_key)

        if assignment:
            proxy, assigned_at = assignment
            if now - assigned_at < self.ttl_seconds:
                return proxy

        proxy = random.choice(self.proxies)
        self._assignments[session_key] = (proxy, now)
        return proxy

    def release(self, session_key):
        self._assignments.pop(session_key, None)

Calling get_proxy("user-42") twice within the TTL window returns the same proxy both times. Call release("user-42") once that flow is done (checkout completed, login task finished) to free the assignment up rather than waiting out the full TTL.

Wiring it into requests.Session

Combine the manager with requests.Session so cookies persist alongside the fixed proxy, this is what actually keeps a login or cart flow coherent end to end:

import requests

PROXIES = [
    "http://user:pass@proxy1.example.com:8000",
    "http://user:pass@proxy2.example.com:8000",
    "http://user:pass@proxy3.example.com:8000",
]

sticky = StickySessionManager(PROXIES, ttl_seconds=600)

def fetch_with_sticky_session(session_key, url, **kwargs):
    proxy = sticky.get_proxy(session_key)
    proxies = {"http": proxy, "https": proxy}
    session = requests.Session()
    session.proxies = proxies
    return session.get(url, **kwargs)

# Login, cart, and checkout for "user-42" all hit the same proxy
resp1 = fetch_with_sticky_session("user-42", "https://example.com/login")
resp2 = fetch_with_sticky_session("user-42", "https://example.com/cart")
resp3 = fetch_with_sticky_session("user-42", "https://example.com/checkout")

sticky.release("user-42")

One thing worth flagging: creating a fresh requests.Session() per call in this example means cookies aren't actually shared across resp1/resp2/resp3 above, only the proxy assignment is. If you need both a persistent proxy and persistent cookies across a flow, keep one Session object alive for the duration of that session key instead of creating a new one per call, and swap its .proxies only when the TTL expires.

When TTL should match your workload, not a default

600 seconds is a reasonable starting point, but the right TTL depends on what you're doing:

Login + immediate action: short TTL (60–120s) is often enough
Multi-step checkout flows: match your longest expected flow duration, with some buffer
Long-running authenticated sessions: consider extending the TTL on each successful request instead of using a fixed expiry, so an active session doesn't get reassigned mid-use

Wrapping up

Authentication is the easy half of this, inline credentials or a header, pick whichever your proxy setup accepts. The part worth actually thinking through is matching your proxy strategy to the shape of your traffic: rotate for independent requests, stay sticky for anything that depends on continuity. Getting this backwards is a common reason "the proxies are fine but I keep getting logged out" bugs show up.

If you're shopping for a provider that supports both patterns cleanly, Squid Proxies' docs and pricing are worth a look, residential and datacenter proxies on one dashboard, no separate accounts to juggle for sticky vs. rotating use cases.

DEV Community