Originally published on proxymint.
Ruby proxy setup for Net::HTTP, Faraday 2 and HTTParty: proxy arguments, credentials, http_proxy and no_proxy, SOCKS5, 407 errors, reuse and rotation.
TL;DR
- Each client has one place for the proxy. Net::HTTP takes host, port, user and password as arguments 3-6 of new or start; Faraday 2 takes a proxy: URL or hash; HTTParty takes http_proxyaddr, http_proxyport, http_proxyuser and http_proxypass.
- HTTPS needs nothing extra. All three send a CONNECT request with Basic credentials to the proxy and run TLS to the website inside that tunnel.
- The environment rules differ. Net::HTTP and HTTParty read only http_proxy, even for https:// URLs, while Faraday reads https_proxy for them. Setting both variables to the same URL covers all three.
- A 407 changes shape with the scheme. It is a normal response on http:// URLs and an exception on https:// URLs: Net::HTTPClientException in Net::HTTP and HTTParty, Faraday::ConnectionFailed in Faraday.
Key takeaways
- Prefer raw credentials over URLs. Net::HTTP arguments, HTTParty options and Faraday's user: and password: keys need no encoding; in a proxy URL, run both parts through URI.encode_uri_component. open-uri is the exception: it never decodes a proxy URL, so give it raw values in proxy_http_basic_authentication.
- Write socks:// in Faraday. Faraday only treats socks:// as SOCKS, and only with the socksify gem installed; socks5:// is sent to Net::HTTP as an HTTP proxy and fails.
- Pass nil to go direct. HTTParty and Net::HTTP pick up http_proxy from the shell unless http_proxyaddr: nil or a nil third argument says otherwise.
- Match connections to the rotation mode. One Net::HTTP.start block keeps one tunnel and one exit; Faraday's net_http adapter and HTTParty open a new connection per call, which suits every-request rotation.
Read the full guide on proxymint, with the commands, tables and sources.
Top comments (0)