DEV Community

ProxyMaster
ProxyMaster

Posted on Edited on

Configuring Proxies in Go: net/http Transport and Colly

#go #webscraping #proxy #tutorial

WinGate private IPv4 and SOCKS5 proxies, free 2 hour test

Go is a favorite for scrapers and crawlers because its concurrency model lets you run thousands of requests without much ceremony. That strength is also a trap: it is very easy to fire so many requests from one address that the target blocks you in seconds. The fix is to route through proxies and rotate them. This post shows how to wire proxies into raw net/http and into Colly, the most common Go crawling framework.

Proxies in net/http

The standard library handles proxies at the Transport level. You give the Transport a function that returns the proxy URL for each request, and every request made through that client goes out through it.

package main

import (
    "net/http"
    "net/url"
    "time"
)

func client() *http.Client {
    proxyURL, _ := url.Parse("http://user:pass@proxy.host:8080")
    return &http.Client{
        Transport: &http.Transport{
            Proxy: http.ProxyURL(proxyURL),
        },
        Timeout: 20 * time.Second,
    }
}
Enter fullscreen mode Exit fullscreen mode

Because Proxy is a function, you can return a different address per request. With a rotating endpoint you keep pointing at the same URL and the pool changes the exit for you, which is the simplest setup that scales.

Proxies in Colly

Colly has a proxy switcher built in. RoundRobinProxySwitcher rotates across a list, but the cleaner path with a rotating provider is to point every request at one rotating endpoint and let the pool do the cycling.

package main

import (
    "github.com/gocolly/colly/v2"
    "github.com/gocolly/colly/v2/proxy"
)

func main() {
    c := colly.NewCollector(colly.Async(true))
    c.Limit(&colly.LimitRule{Parallelism: 20, RandomDelay: 500})

    rp, err := proxy.RoundRobinProxySwitcher("http://user:pass@proxy.host:8080")
    if err == nil {
        c.SetProxyFunc(rp)
    }

    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        e.Request.Visit(e.Attr("href"))
    })
    c.Visit("https://example.com")
    c.Wait()
}
Enter fullscreen mode Exit fullscreen mode

The LimitRule is important. Async concurrency plus rotation is powerful, but a random delay and a sane parallelism keep you from turning your own crawler into a denial of service against the target.

What the pool behind the endpoint needs

The Go code is short because the hard part is the address pool, not the client. A few properties decide whether a crawl survives.

  • Private addresses. Shared pools arrive pre-flagged from other users. Private IPv4 carries only your reputation.
  • Real rotation. A large pool that cycles the exit is what keeps each address under the rate limit, which is the whole reason you added proxies.
  • Throughput. Go will happily run thousands of goroutines. The pool has to support that concurrency or it becomes the bottleneck.
  • Protocol coverage. HTTP, HTTPS, and SOCKS5 support means net/http and Colly both connect without special cases.

WinGate fits this directly. It provides private IPv4 proxies with SOCKS5 and automatic rotation from a worldmix pool, unlimited traffic, and support for up to 5000 threads, which matches Go's concurrency instead of throttling it. The rotating endpoint cycles exits for you, so both snippets above work by swapping in the URL.

An honest note: proxies and rotation stop the address from being the reason your crawler dies, they do not make aggressive crawling acceptable. Keep the LimitRule sane, respect the target's terms, and pace your goroutines. There is a free 2 hour test, so point a small Colly run at your own targets and watch the success rate before you scale the parallelism up.

Related reading

The takeaway: in Go the proxy wiring is a few lines, at the Transport for net/http or the proxy func for Colly. The real work is a private, rotating pool that can keep up with Go's concurrency. Get that right and your crawler uses Go's strength without tripping the first rate limit it meets.

Top comments (0)