Fetching Instagram profiles concurrently in Go with HikerAPI

HikerAPI has official clients for Python and Node, but not for Go. I wanted to pull public Instagram profile data (followers, post count, bio) from a Go program, so I wrote a small client using only the standard library.

By the end of this post you'll have a CLI that:

takes a list of usernames,
fetches them concurrently with a bounded worker pool,
prints one JSON object per line (ready for jq or a .jsonl file),
and comes with a test that runs against a fake server, so it doesn't spend your API credits.
Setup

Create an account at hikerapi.com and generate an access key. The free tier gives you 100 requests, which is plenty to follow along.

bash
mkdir igprofiles && cd igprofiles
go mod init igprofiles
export HIKERAPI_KEY="your-access-key"

The endpoint we'll use is:

GET https://api.hikerapi.com/v2/user/by/username?username=
Header: x-access-key:

The response wraps the profile in a user object.

Modeling only what we need

Instagram's user object has dozens of fields. I only decode the ones I care about, and encoding/json silently ignores the rest:

go
type Profile struct {
PK FlexID json:"pk"
Username string json:"username"
FullName string json:"full_name"
Biography string json:"biography"
IsPrivate bool json:"is_private"
IsVerified bool json:"is_verified"
FollowerCount int json:"follower_count"
FollowingCount int json:"following_count"
MediaCount int json:"media_count"
ExternalURL string json:"external_url"
}

type userResponse struct {
User Profile json:"user"
}
The pk gotcha

The user ID (pk) can show up as a number or as a string depending on the endpoint. If you declare it as string and get a number, decoding fails. If you declare it as int64 and get a string, same thing.

A tiny custom type handles both and always stores a string:

go
type FlexID string

func (f *FlexID) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err == nil {
*f = FlexID(s)
return nil
}
var n json.Number
if err := json.Unmarshal(b, &n); err != nil {
return err
}
*f = FlexID(n.String())
return nil
}

Using json.Number instead of float64 matters here: Instagram IDs are big enough that a float can lose digits.

The client
go
const baseURL = "https://api.hikerapi.com"

type Client struct {
key string
base string
http *http.Client
}

func NewClient(key string) *Client {
return &Client{key: key, base: baseURL, http: &http.Client{Timeout: 20 * time.Second}}
}

func (c *Client) UserByUsername(ctx context.Context, username string) (*Profile, error) {
u := c.base + "/v2/user/by/username?username=" + url.QueryEscape(username)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return nil, err
}
req.Header.Set("x-access-key", c.key)
req.Header.Set("accept", "application/json")

resp, err := c.http.Do(req)
if err != nil {
    return nil, err
}
defer resp.Body.Close()

body, err := io.ReadAll(resp.Body)
if err != nil {
    return nil, err
}
if resp.StatusCode != http.StatusOK {
    return nil, fmt.Errorf("status %d: %s", resp.StatusCode, truncate(string(body), 200))
}

var ur userResponse
if err := json.Unmarshal(body, &ur); err != nil {
    return nil, fmt.Errorf("decode: %w", err)
}
if ur.User.Username == "" {
    return nil, errors.New("empty profile (user not found?)")
}
return &ur.User, nil

}

Two small decisions worth pointing out:

base is a field, not just a constant. That's what lets the test point the client at a fake server.
The empty-username check. Some error responses still come back as valid JSON, so "it decoded" doesn't mean "we got a profile".
Fetching many profiles concurrently

Firing one goroutine per username is fine for 5 names and a bad idea for 500. A fixed pool of workers reading from a channel keeps concurrency bounded:

go
const workers = 5
jobs := make(chan string)
results := make(chan result)

var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for name := range jobs {
p, err := client.UserByUsername(ctx, name)
results <- result{username: name, profile: p, err: err}
}
}()
}

go func() {
for _, u := range usernames {
jobs <- u
}
close(jobs)
}()

go func() {
wg.Wait()
close(results)
}()

enc := json.NewEncoder(os.Stdout)
for r := range results {
if r.err != nil {
fmt.Fprintf(os.Stderr, "✗ %s: %v\n", r.username, r.err)
continue
}
_ = enc.Encode(r.profile)
}

Profiles go to stdout and errors go to stderr, so you can redirect the data to a file and still see failures in the terminal:

bash
go run . nasa natgeo instagram > profiles.jsonl
jq '{username, follower_count}' profiles.jsonl

Testing without spending requests

net/http/httptest spins up a local server, so the test covers the real parsing and error handling without touching the network:

go
func TestUserByUsername(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("x-access-key") != "k" {
w.WriteHeader(http.StatusUnauthorized)
return
}
switch r.URL.Query().Get("username") {
case "num":
w.Write([]byte({"user":{"pk":528817151,"username":"num","follower_count":10}}))
case "str":
w.Write([]byte({"user":{"pk":"123","username":"str"}}))
default:
w.Write([]byte({"detail":"not found"}))
}
}))
defer srv.Close()

c := NewClient("k")
c.base = srv.URL

p, err := c.UserByUsername(context.Background(), "num")
if err != nil || p.PK != "528817151" || p.FollowerCount != 10 {
    t.Fatalf("numeric pk: %v %+v", err, p)
}
p, err = c.UserByUsername(context.Background(), "str")
if err != nil || p.PK != "123" {
    t.Fatalf("string pk: %v %+v", err, p)
}
if _, err := c.UserByUsername(context.Background(), "nope"); err == nil {
    t.Fatal("expected error for missing user")
}

bad := NewClient("wrong")
bad.base = srv.URL
if _, err := bad.UserByUsername(context.Background(), "num"); err == nil {
    t.Fatal("expected error for bad key")
}

}
bash
go test ./... -v
What I'd add next
Retries with backoff for 429 and 5xx responses.
A -workers flag instead of a constant.
Reading usernames from stdin, so it composes with other tools.