Two features on our site fetch addresses that strangers type in. When a maker submits a product, the server fetches the product's website to find a logo. And the meta tag and social card checker fetches any public page you give it and shows what a link preview would contain.
A server that fetches arbitrary URLs is a classic server side request forgery risk. The server can reach things nobody else can: the API on localhost, a database container on the internal network, or a cloud metadata service on 169.254.169.254. If a user can point the fetcher there, the server becomes their proxy.
This post shows the Go client we use for this, and why the check sits inside the dialer instead of before the request.
The obvious version and its gap
The first idea is to look up the hostname, check that every address it resolves to is public, and then make the request:
ips, err := net.LookupIP(host)
// ... reject if any ip is not public
resp, err := http.Get(rawURL)
The problem is that on a new direct connection, the client resolves the hostname again, independently of the check. If it receives a different answer, it can connect to an address the check never approved. A hostname with a very short TTL can answer the check with a public address and the connection's lookup with 127.0.0.1. This is known as DNS rebinding, and a check before the request cannot see it, because by the time the client connects, the answer that was checked is no longer the one being used.
Redirects make it worse. A perfectly public page can answer with a redirect to http://127.0.0.1/, and a default client follows it without asking anyone.
Check the address the connection uses
Go's net.Dialer has a Control field. Its documentation says that if Control is not nil, it is called after creating the network connection but before actually dialing. By then the hostname has been resolved, and the address argument holds the IP and port of this particular connection attempt. That is exactly the value we want to check.
// internal/audit/ssrf.go, comments omitted
func NewPublicOnlyClient(timeout time.Duration, maxRedirects int) *http.Client {
dialer := &net.Dialer{
Timeout: timeout,
Control: func(network, address string, _ syscall.RawConn) error {
host, _, err := net.SplitHostPort(address)
if err != nil {
return fmt.Errorf("bad address")
}
ip := net.ParseIP(host)
if ip == nil || !isPublic(ip) {
return fmt.Errorf("address is not public")
}
return nil
},
}
return &http.Client{
Timeout: timeout,
Transport: &http.Transport{DialContext: dialer.DialContext, Proxy: nil, DisableKeepAlives: true},
CheckRedirect: func(req *http.Request, via []*http.Request) error {
if len(via) > maxRedirects {
return fmt.Errorf("too many redirects")
}
if req.URL.Scheme != "http" && req.URL.Scheme != "https" {
return fmt.Errorf("redirect to a non-web address")
}
return nil
},
}
}
A few details carry the weight here:
-
Anything that is not an IP is refused. If
addressdoes not parse as an IP,net.ParseIPreturns nil and the dial fails. The check never has to trust a name. -
Every connection attempt is checked. If a hostname has several addresses and the dialer tries more than one, each attempt calls
Control. A redirect opens a new connection, so it goes throughControlas well, whatever DNS says about the new host. -
Proxy: nil. Thehttp.Transportdocumentation says that ifProxyis nil, no proxy is used. Go's default transport reads proxy settings from the environment. With a proxy configured, the dialer would connect to the proxy, andControlwould only ever see the proxy's address, not the target's. -
DisableKeepAlives: true. Connections are not reused between requests, so each request opens its own connection and passes throughControl.
Which addresses count as public
// internal/audit/ssrf.go, comments omitted
func isPublic(ip net.IP) bool {
if ip.IsLoopback() || ip.IsPrivate() || ip.IsUnspecified() ||
ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() ||
ip.IsInterfaceLocalMulticast() || ip.IsMulticast() {
return false
}
if v4 := ip.To4(); v4 != nil {
if v4[0] == 100 && v4[1] >= 64 && v4[1] <= 127 {
return false
}
if v4[0] == 0 {
return false
}
}
return true
}
The net.IP helpers cover loopback, the private ranges 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 and fc00::/7, link-local addresses, which include the metadata address 169.254.169.254, and multicast. The two extra checks block carrier-grade NAT, 100.64.0.0/10, and 0.0.0.0/8, which have no helper.
Be honest with yourself about what this is: a deny list of the ranges that matter for our server. The Go documentation for IsPrivate itself says it "does not describe a security property of addresses, and should not be used for access control". There are special purpose ranges this function does not list. If your network has other internal ranges, or translates addresses between IPv6 and IPv4, add them or, better, also block outgoing connections to internal networks at the firewall, so the code is not the only line of defense.
Redirects that you follow yourself
The logo fetcher lets the client follow redirects, up to its limit. The meta tag checker wants to show every redirect step to the user, so it turns automatic redirects off and follows them by hand:
// internal/inspect/inspect.go, excerpt, comments omitted
c := *client
c.CheckRedirect = func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }
c.Jar = nil
Each hop is a new request through the same guarded client, so each new connection passes Control. Before following a hop, the checker also requires an http or https scheme, a hostname, no username or password in the URL, and either no explicit port or port 80 or 443, and it stops after 5 redirects.
One more thing worth copying from it: errors shown to the user are turned into short fixed sentences. Blocked addresses, timeouts, redirect limits and unsupported redirect targets get their own fixed messages, and other connection failures become "could not connect to the address". The raw error text, which can contain internal addresses, never reaches the page.
Tests
The address list is tested with literal addresses, including the ones that motivate it:
// internal/audit/ssrf_test.go, abridged: fewer entries, reasons shortened
blocked := map[string]string{
"127.0.0.1": "loopback",
"10.0.0.5": "private network",
"172.17.0.2": "the Docker network the database sits on",
"169.254.169.254": "cloud metadata service",
"100.64.0.1": "carrier-grade NAT",
"fe80::1": "link-local",
// ...
}
The client itself is tested by pointing it at a local httptest server. That server listens on loopback, so the request must fail at the dialer:
// internal/audit/ssrf_test.go
func TestPublicOnlyClient_RefusesLoopbackAtDial(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("secret"))
}))
defer srv.Close()
client := NewPublicOnlyClient(2*time.Second, 3)
resp, err := client.Get(srv.URL)
if err == nil {
resp.Body.Close()
t.Fatal("a loopback server was reachable through the public-only client")
}
}
The same fact makes the redirect limit awkward to test: a local test server can never be reached through the guarded client. So that test builds the client and then swaps in http.DefaultTransport, which keeps the redirect policy under test while removing the dial check.
What the tests do not do is simulate a real rebinding DNS server. The argument for the design is the one above: the check runs on the address being dialed, so there is no second lookup to race against.
Takeaways
- Do not check a hostname and then fetch it. Check the IP inside the dialer, in
net.Dialer.Control, where the address is the one the connection will use. - Refuse anything in
Controlthat is not a parseable IP. - Turn off environment proxies for this client, or the check only sees the proxy.
- Treat the address list as a deny list for your own network, and add an egress firewall rule if you can.
- Follow redirects through the same guarded client and validate scheme, port and credentials on every hop.
- Never show raw connection errors to the user.
Top comments (0)