Google Search Results in Go with a SERP API

Go doesn't get enough love in the "scrape search results" space — most tutorials are Python or Node. But Go is a genuinely great fit for SERP work: you often want to fire off a batch of queries concurrently, and goroutines make that trivial. This post is the Go version: one request, concurrent batch crawling, JSON parsing with encoding/json, no browser, no HTML parsing.

The request

The API I'm using is SerpBase (https://api.serpbase.dev). Auth is a plain X-API-Key header, every endpoint takes a POST JSON body. The standard library's net/http is all you need:

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "os"
)

const baseURL = "https://api.serpbase.dev"

type SearchResult struct {
    Rank  int    `json:"rank"`
    Title string `json:"title"`
    Link  string `json:"link"`
}

type SerpResponse struct {
    Status  int            `json:"status"`
    Organic []SearchResult `json:"organic"`
}

func search(apiKey, query string) (SerpResponse, error) {
    body, _ := json.Marshal(map[string]any{
        "q": query, "hl": "en", "gl": "us",
    })

    req, _ := http.NewRequest(http.MethodPost, baseURL+"/google/search", bytes.NewReader(body))
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("X-API-Key", apiKey)

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return SerpResponse{}, err
    }
    defer resp.Body.Close()

    var data SerpResponse
    if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
        return SerpResponse{}, err
    }
    return data, nil
}

func main() {
    data, err := search(os.Getenv("SERPBASE_API_KEY"), "go concurrency")
    if err != nil {
        panic(err)
    }
    for _, r := range data.Organic {
        fmt.Printf("%d. %s\n   %s\n", r.Rank, r.Title, r.Link)
    }
}

The json struct tags map the response fields directly. Struct-based parsing is one of Go's quiet advantages here — no dynamic dict juggling.

Concurrent batch crawling with goroutines

The payoff of Go shows up when you have a keyword list. Each keyword is an independent HTTP call, so they parallelize cleanly:

func crawlBatch(apiKey string, keywords []string) []SerpResponse {
    results := make([]SerpResponse, len(keywords))
    var wg sync.WaitGroup
    var mu sync.Mutex

    for i, kw := range keywords {
        wg.Add(1)
        go func(i int, kw string) {
            defer wg.Done()
            data, err := search(apiKey, kw)
            if err != nil {
                mu.Lock()
                results[i] = SerpResponse{Status: -1} // mark as failed
                mu.Unlock()
                return
            }
            mu.Lock()
            results[i] = data
            mu.Unlock()
        }(i, kw)
    }

    wg.Wait()
    return results
}

Notes on this pattern:

Results are written by index (not appended), so order is preserved.
A mutex guards the slice because multiple goroutines write it.
A failed request is marked with Status: -1 so the caller can retry just those.

If your QPS matters, wrap the loop with a small semaphore to cap concurrency (a buffered channel works great):

sem := make(chan struct{}, 10) // max 10 concurrent
for i, kw := range keywords {
    sem <- struct{}{} // acquire
    go func(i int, kw string) {
        defer func() { <-sem }() // release
        // ...same search logic
    }(i, kw)
}

Retrying safely

SerpBase refunds credits for failed dispatches and upstream timeouts, so retries don't burn your balance. In Go, a small retry wrapper is easy:

func searchWithRetry(apiKey, query string, maxRetries int) (SerpResponse, error) {
    var data SerpResponse
    var err error
    for attempt := 0; attempt < maxRetries; attempt++ {
        data, err = search(apiKey, query)
        if err == nil && data.Status == 0 {
            return data, nil
        }
        time.Sleep(time.Duration(attempt+1) * 500 * time.Millisecond)
    }
    return data, fmt.Errorf("failed after %d retries: %w", maxRetries, err)
}

Handling optional fields

The response schema has optional fields (snippet, display_url, position). In Go, use pointers or omitempty-friendly structs so missing fields decode to zero values instead of errors:

type SearchResult struct {
    Rank      int     `json:"rank"`
    Position  *int    `json:"position,omitempty"`  // optional alias
    Title     string  `json:"title"`
    Link      string  `json:"link"`
    Snippet   *string `json:"snippet,omitempty"`   // may be absent
}

A *string field that's absent decodes to nil, which you can check with if r.Snippet != nil.

Cost before you scale

/google/search costs 1 credit per request. A batch of 500 keywords, checked daily, is ~15k requests/month — on a standard pack (~$0.50/1k) that's around $7.50/month. The free 100 searches on signup cover your first build week.

Wrapping up

Go + SERP API is a small, boring, fast stack: standard library HTTP, struct-based JSON, goroutines for concurrency. No cheerio, no puppeteer, no selector maintenance.

The full response schema (including all optional modules like featured_snippet and people_also_ask) is in the SerpBase documentation. Run the single-query version first, then add the concurrent batch — it's the difference between a script and a tool.

DEV Community