Recently I run into a problem where the Golang offical httputil reverse proxy get stuck when testing my open source reverse proxy server Zoraxy.
Zoraxy
A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!
Features
- Simple to use interface with detail in-system instructions
- Reverse Proxy (HTTP/2)
- Virtual Directory
- WebSocket Proxy (automatic, no set-up needed)
- Basic Auth
- Alias Hostnames
- Custom Headers
- Redirection Rules
- TLS / SSL setup and deploy
- ACME features like auto-renew to serve your sites in https
- SNI support (and SAN certs)
- DNS Challenge for Let's Encrypt and these DNS providers
- Blacklist / Whitelist by country or IP address (single IP, CIDR or wildcard for beginners)
- Global Area Network Controller Web UI (ZeroTier not included)
- Stream Proxy (TCP & UDP)
- Integrated Up-time Monitor
- Web-SSH Terminal
- Utilities
- CIDR IP converters
- mDNS Scanner
- Wake-On-Lan
- Debug Forward Proxy
- IP Scanner
- Others
- Basic single-admin management mode
- External permission management system for easy system integration
- SMTP config for password reset
Downloads
Windows / Linux (amd64) / Linux (arm64)
For other systems or architectures…
Have you ever try to open a website that is under heavy load? The connection just freeze there and do nothing until other objects in your site is loaded and free up some connections, then your content continue to load. This should not be happening as Go is known to be able to handle many requests in the same time using its magical go-routine. That is why I started to question about the implementation of the httputil.reverseproxy library.
If you don't know there is an official http reverse proxy in the Golang source code, this is the one I am talking about.
https://go.dev/src/net/http/httputil/reverseproxy.go
And within the source code, you will notice there is a line in the reverse proxy struct object that make use of something called the http.Transport
// The transport used to perform proxy requests.
// If nil, http.DefaultTransport is used.
Transport http.RoundTripper
After debugging a few days and tracing the Go source, I notice the issue was that the http.DefaultTransport pre-set a relatively low limits on the maximum connections counts. This setting is reasonable as most of them time this reverse proxy is used to serve single user instead of using it as a replacement of nginx or apache reverse proxy.
That is why in order to fix it, I included a modified version of the reverse proxy into Zoraxy project with the following modifications:
//Hack the default transporter to handle more connections
thisTransporter := http.DefaultTransport
thisTransporter.(*http.Transport).MaxIdleConns = 3000
thisTransporter.(*http.Transport).MaxIdleConnsPerHost = 3000
thisTransporter.(*http.Transport).IdleConnTimeout = 10 * time.Second
thisTransporter.(*http.Transport).MaxConnsPerHost = 0
Now, my reverse proxy can serve much more request in a given time compare to the http.DefaultTransport settings used in the original reverse proxy utilities!
---------- Updates -------------
After a few months of further investigation and testing, I found that it would be better for each of the reverse proxy object (aka it will only have 1 host handled by each Transport object) to have the following configurations
//Hack the default transporter to handle more connections
thisTransporter := http.DefaultTransport
optimalConcurrentConnection := 32
thisTransporter.(*http.Transport).MaxIdleConns = optimalConcurrentConnection * 2
thisTransporter.(*http.Transport).MaxIdleConnsPerHost = optimalConcurrentConnection
thisTransporter.(*http.Transport).IdleConnTimeout = 30 * time.Second
thisTransporter.(*http.Transport).MaxConnsPerHost = optimalConcurrentConnection * 2
thisTransporter.(*http.Transport).DisableCompression = true
Compare to the previous settings which uses around 85% of my bandwidth, this settings can utilize up to around 90 - 92% of my outbound bandwidth. I have no idea where these numbers come from, but it works great.
Top comments (1)
would be nice to have some banchmarks