DEV Community

Cover image for What an HTTP request looks like after it passes through an HTTP proxy
IPOasis
IPOasis

Posted on

What an HTTP request looks like after it passes through an HTTP proxy

#programming #python #networking #proxy

An HTTP proxy takes the request your script wrote and sends it out from another address. The script's socket ends at the proxy. The site's socket starts at the proxy exit.

What the script sends to the proxy

requests opens the proxy when the target URL is http:// and proxies has an http entry. The request line is the full URL, which is what the proxy dials.

GET http://checkip.amazonaws.com/ HTTP/1.1
Host: checkip.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

A direct request line is only the path, because the socket is already on the site. Through a proxy the request line is the absolute URL.

import os
import requests

def public_ip(proxy_url: str | None) -> str:
    proxies = None
    if proxy_url:
        proxies = {"http": proxy_url}
    response = requests.get(
        "http://checkip.amazonaws.com",
        proxies=proxies,
        timeout=20,
    )
    response.raise_for_status()
    return response.text.strip()

print("direct ", public_ip(None))
print("proxied", public_ip(os.environ["PROXY_URL"]))
Enter fullscreen mode Exit fullscreen mode

PROXY_URL is that proxy URL, username included.

http://USERNAME:PASSWORD@HOST:PORT
Enter fullscreen mode Exit fullscreen mode

The first line is the machine running the script. The second is the exit.

A proxy that asks for a login gets Proxy-Authorization on this request. The site gets Host and the path.

What the site records as the client

The site's connection log shows the exit. That is the second printed line. The headers on that request are the ones requests built, without the proxy login.

Matching lines mean the call left from an address you already had. Check that PROXY_URL is set, and that its host is the provider gateway, before reading anything into the echo.

Why a second call can leave from another exit

A pool gateway picks the exit when it handles the request. No session id in the username, and the next request can leave from another exit. Host and port in PROXY_URL stay put. The pick happens after the proxy reads the username.

requests.Session keeps a connection pool and a cookie store in the process. The second request still carries the same proxy URL. The exit is still the gateway's pick.

import os
import requests

proxy_url = os.environ["PROXY_URL"]

with requests.Session() as client:
    client.proxies.update({"http": proxy_url})
    for n in range(2):
        response = client.get("http://checkip.amazonaws.com", timeout=20)
        response.raise_for_status()
        print(n, response.text.strip())
Enter fullscreen mode Exit fullscreen mode

Two addresses from one username means the gateway was free to move the exit. One address means it reused an exit, or the username already carries a session id. Providers put that id in the username. A new username starts a new session.

A hold is a number of minutes on that username. IPOasis dynamic residential sets it from 1 to 120 minutes. After that, the next request can move.

What SOCKS5 changes in this picture

SOCKS5 uses a handshake instead of an absolute-URL request line. The script still dials the proxy, the proxy still dials the site, and the site still logs the exit.

socks5h://USERNAME:PASSWORD@HOST:PORT
Enter fullscreen mode Exit fullscreen mode

socks5h sends the hostname. The proxy resolves it. socks5 resolves it locally and sends an IP. Either scheme needs PySocks. The echo URL stays http://checkip.amazonaws.com.

Top comments (0)