DEV Community

Andrea Mancuso
Andrea Mancuso

Posted on

When HTTP/2 quietly broke our proxy routing

Our headless Chrome scrapers, driven by Puppeteer, needed to fetch pages from specific countries. The corporate network handled this: add a routing header to a request, and the upstream proxy sent it out through the right country. It worked for most sites and silently failed for others, which fell back to a default US exit. After a lot of NetLog traces and packet captures (and a lot of back and forth with an LLM to make sense of them), the pattern was clear: every failing host negotiated HTTP/2. We reproduced it outside the browser with an HTTP/2-enabled curl against an IP geolocation endpoint. HTTP/1.1 with the header inside the tunnel: routed correctly. HTTP/2 with the header inside the tunnel: ignored. HTTP/2 with the header on the CONNECT request itself: routed correctly.
So the network layer could read our header inside the tunnel for HTTP/1.1 but not for HTTP/2, and the only reliable place for it was the CONNECT request that opens the tunnel. Chrome gives you no way to add custom headers there: Puppeteer's extra headers apply to the requests inside the tunnel, not to the proxy handshake. The owners of the routing agreed the behaviour was on their side but couldn't change it soon, so my team wrote a small forwarding proxy in Node.js. Chrome connects to it locally, and it forwards each CONNECT upstream with the routing header attached. Routing no longer depends on which HTTP version a site negotiates. The lesson: when routing depends on headers, check which request carries them, because with HTTPS through a proxy, the one that matters may be the CONNECT.

Top comments (0)