Fetching Instagram Data Concurrently in Python with HikerAPI

When I started collecting at , a simple sequential loop quickly became the bottleneck. Even if each request only takes a fraction of a second, hundreds or thousands of API calls add up.

The solution wasn't making each request faster—it was making many requests at the same time.

In this article I'll show the simple progression I followed: start with a single request, then parallelize it with threads, discuss an async alternative, and finish with some practical rate-limit handling.

For this example I'm using HikerAPI, but the concurrency patterns apply to most REST APIs.

The baseline

Everything starts with a single request:

import requests

headers = {"x-access-key": "YOUR_KEY"}

resp = requests.get(
"https://api.hikerapi.com/v2/hashtag/medias/top",
params={"name": "travel"},
headers=headers,
)

print(resp.json())

This is perfectly fine if you're making one request.

It becomes slow when you need dozens or hundreds.

Parallelizing with ThreadPoolExecutor

Since HTTP requests spend most of their time waiting on the network, Python threads work well here.

import requests
from concurrent.futures import ThreadPoolExecutor, as_completed

headers = {"x-access-key": "YOUR_KEY"}

hashtags = [
"travel",
"nature",
"food",
"fitness",
"photography",
"technology",
]

def fetch_hashtag(name):
response = requests.get(
"https://api.hikerapi.com/v2/hashtag/medias/top",
params={"name": name},
headers=headers,
timeout=30,
)

response.raise_for_status()

return {
    "hashtag": name,
    "data": response.json(),
}

results = []

with ThreadPoolExecutor(max_workers=5) as executor:
futures = [executor.submit(fetch_hashtag, tag) for tag in hashtags]

for future in as_completed(futures):
    results.append(future.result())

print(f"Fetched {len(results)} hashtag responses.")

The nice thing about this approach is that the rest of my code barely changes. I still use the familiar requests library while making multiple API calls concurrently.

Async is another option

If I'm already using an async application (FastAPI, asyncio workers, etc.), I'd probably use httpx.AsyncClient or aiohttp instead of threads.

The overall idea stays the same:

create one HTTP client
schedule many requests
wait for all of them together
process the results

Async usually integrates more naturally into async applications, while threads are often the quickest upgrade for existing synchronous code.

Handle rate limits gracefully

Concurrency doesn't mean "send unlimited requests."

Every API has limits, and increasing the number of workers too aggressively usually hurts reliability instead of improving throughput.

A few practices have worked well for me:

Keep the worker count conservative (for example, 5–10 to start).
Retry temporary failures with exponential backoff.
Respect HTTP 429 responses.
Add request timeouts.
Log failed requests so they can be retried later.

Here's a simple retry example:

import time
import requests

def fetch_with_retry(url, params, headers, retries=3):
for attempt in range(retries):
response = requests.get(
url,
params=params,
headers=headers,
timeout=30,
)

    if response.status_code == 429:
        wait = 2 ** attempt
        time.sleep(wait)
        continue

    response.raise_for_status()
    return response.json()

raise RuntimeError("Maximum retries exceeded.")

The important part isn't just retrying—it's slowing down after receiving a rate-limit response instead of immediately sending more requests.

Choosing the right concurrency level

It's tempting to keep increasing the worker count until requests stop getting faster.

In practice, there's usually a sweet spot.

Too few workers leave bandwidth unused.

Too many workers increase contention, retries, and rate-limit responses.

I usually start small, monitor success rates and latency, then increase concurrency gradually until additional workers stop improving overall throughput.

Final thoughts

Once I started collecting at , sequential requests simply didn't scale well enough.

Using a small thread pool dramatically reduced total execution time without making the code much more complicated. If my project were already async, I'd use an async HTTP client instead, but the underlying principle would be exactly the same: keep multiple network requests in flight while respecting the API's rate limits.

Concurrency isn't about overwhelming an API—it's about making efficient use of the time your application would otherwise spend waiting for network responses.

I'm part of HikerAPI's user-rewards program — they credit my account for posts like this. Sharing my actual experience.