Originally published on proxymint.
Crawlee proxy setup for Node and Python: ProxyConfiguration, how sessions pin proxies, the browser trap where proxyInfo lies, tiers, SOCKS5 and 407 loops.
TL;DR
- ProxyConfiguration is the whole setup. Pass proxyUrls (Node) or proxy_urls (Python) with the login inside each URL, hand the object to the crawler, and read proxyInfo in the handler to see which URL Crawlee picked.
- Rotation follows sessions. Each new session takes the next URL in round robin and keeps it. The default pool holds 1,000 sessions, so early in a crawl nearly every request gets a new session and the next proxy.
- Browser crawlers can report the wrong proxy. In crawlee 3.18.2 a default PlaywrightCrawler logged three proxies in proxyInfo while every page exited through the first; useIncognitoPages or browserPerProxy fixed it.
- socks5:// depends on the client. got-scraping, the Node default, accepts only HTTP proxy URLs, and Crawlee for Python rejects SOCKS URLs at construction. Browser crawlers and Node's ImpitHttpClient accept SOCKS5 with a login.
Key takeaways
- Check the exit, not proxyInfo. Run ten requests to an https:// IP echo with a uniqueKey on each, and compare the IP the target saw with the port Crawlee logged.
- Set useIncognitoPages or browserPerProxy on browser crawlers. Without one of them, pages can run in a browser launched with another proxy and rotation results mean nothing.
- Give HTTP crawlers the HTTP endpoint. An HTTP proxy with an https:// target sends CONNECT with the hostname, so DNS resolves on the proxy side without socks5h.
- Lower maxSessionRotations while testing a new proxy. With the default Node client a wrong password is a session error, retried up to 10 times per request, and the final message only says a session was rotated.
- Order tiers by what the target accepts. No proxy, then datacenter, then residential, on Crawlee 3.x; the 4.0 release candidate removes tieredProxyUrls.
Read the full guide on proxymint, with the commands, tables and sources.
Top comments (0)