DEV Community: ashk asef

Fetching Instagram Data Concurrently in Python with HikerAPI

ashk asef — Sun, 26 Jul 2026 14:39:01 +0000

When I started collecting at , a simple sequential loop quickly became the bottleneck. Even if each request only takes a fraction of a second, hundreds or thousands of API calls add up.

The solution wasn't making each request faster—it was making many requests at the same time.

In this article I'll show the simple progression I followed: start with a single request, then parallelize it with threads, discuss an async alternative, and finish with some practical rate-limit handling.

For this example I'm using HikerAPI, but the concurrency patterns apply to most REST APIs.

The baseline

Everything starts with a single request:

import requests

headers = {"x-access-key": "YOUR_KEY"}

resp = requests.get(
"https://api.hikerapi.com/v2/hashtag/medias/top",
params={"name": "travel"},
headers=headers,
)

print(resp.json())

This is perfectly fine if you're making one request.

It becomes slow when you need dozens or hundreds.

Parallelizing with ThreadPoolExecutor

Since HTTP requests spend most of their time waiting on the network, Python threads work well here.

import requests
from concurrent.futures import ThreadPoolExecutor, as_completed

headers = {"x-access-key": "YOUR_KEY"}

hashtags = [
"travel",
"nature",
"food",
"fitness",
"photography",
"technology",
]

def fetch_hashtag(name):
response = requests.get(
"https://api.hikerapi.com/v2/hashtag/medias/top",
params={"name": name},
headers=headers,
timeout=30,
)

response.raise_for_status()

return {
    "hashtag": name,
    "data": response.json(),
}

results = []

with ThreadPoolExecutor(max_workers=5) as executor:
futures = [executor.submit(fetch_hashtag, tag) for tag in hashtags]

for future in as_completed(futures):
    results.append(future.result())

print(f"Fetched {len(results)} hashtag responses.")

The nice thing about this approach is that the rest of my code barely changes. I still use the familiar requests library while making multiple API calls concurrently.

Async is another option

If I'm already using an async application (FastAPI, asyncio workers, etc.), I'd probably use httpx.AsyncClient or aiohttp instead of threads.

The overall idea stays the same:

create one HTTP client
schedule many requests
wait for all of them together
process the results

Async usually integrates more naturally into async applications, while threads are often the quickest upgrade for existing synchronous code.

Handle rate limits gracefully

Concurrency doesn't mean "send unlimited requests."

Every API has limits, and increasing the number of workers too aggressively usually hurts reliability instead of improving throughput.

A few practices have worked well for me:

Keep the worker count conservative (for example, 5–10 to start).
Retry temporary failures with exponential backoff.
Respect HTTP 429 responses.
Add request timeouts.
Log failed requests so they can be retried later.

Here's a simple retry example:

import time
import requests

def fetch_with_retry(url, params, headers, retries=3):
for attempt in range(retries):
response = requests.get(
url,
params=params,
headers=headers,
timeout=30,
)

    if response.status_code == 429:
        wait = 2 ** attempt
        time.sleep(wait)
        continue

    response.raise_for_status()
    return response.json()

raise RuntimeError("Maximum retries exceeded.")

The important part isn't just retrying—it's slowing down after receiving a rate-limit response instead of immediately sending more requests.

Choosing the right concurrency level

It's tempting to keep increasing the worker count until requests stop getting faster.

In practice, there's usually a sweet spot.

Too few workers leave bandwidth unused.

Too many workers increase contention, retries, and rate-limit responses.

I usually start small, monitor success rates and latency, then increase concurrency gradually until additional workers stop improving overall throughput.

Final thoughts

Once I started collecting at , sequential requests simply didn't scale well enough.

Using a small thread pool dramatically reduced total execution time without making the code much more complicated. If my project were already async, I'd use an async HTTP client instead, but the underlying principle would be exactly the same: keep multiple network requests in flight while respecting the API's rate limits.

Concurrency isn't about overwhelming an API—it's about making efficient use of the time your application would otherwise spend waiting for network responses.

I'm part of HikerAPI's user-rewards program — they credit my account for posts like this. Sharing my actual experience.

Why I Stopped Self-Hosting My Instagram Scraper

ashk asef — Sun, 26 Jul 2026 14:35:07 +0000

Why I Stopped Self-Hosting My Instagram Scraper

For a long time, I assumed self-hosting was the "right" way to build anything that interacted with Instagram. If I controlled the infrastructure, I could customize everything, avoid vendor lock-in, and potentially save money.

That assumption lasted until my project became more than a weekend experiment.

When the scraper became the project

I originally built using instagrapi because it gave me direct access to Instagram without relying on a hosted service. It worked well enough in development, and getting the first features running was surprisingly straightforward.

The problems started once I wanted the system to run consistently.

Instead of focusing on building product features, I found myself spending more and more time dealing with infrastructure problems. Login sessions expired. Accounts required verification. Some requests suddenly stopped working after platform changes. Every time I thought everything was stable, something else needed attention.

The scraper slowly became a project of its own.

What kept breaking

None of these issues were impossible to solve individually, but together they created a steady maintenance burden.

Some of the recurring problems included:

Login sessions expiring unexpectedly.
Account challenges and verification requests.
Temporary rate limits.
Occasional account restrictions or bans.
Library updates following Instagram changes.
Monitoring multiple accounts and keeping them healthy.

My biggest frustration wasn't that things broke—it was that they broke unpredictably.

I'd fix one issue, only to discover another one a week later. The actual business logic for became the easy part. Keeping the data pipeline alive was the difficult part.

Looking for a different approach

Eventually I started asking myself whether I actually wanted to maintain Instagram scraping infrastructure.

The answer was no.

I wanted to build , not become an expert in session cookies, account recovery, and scraper maintenance.

That's when I decided to try HikerAPI, a hosted REST API for Instagram.

The migration wasn't really about adding features.

It was about removing operational work.

What changed after migrating

The biggest improvement wasn't performance or speed.

It was simplicity.

Instead of maintaining accounts and scraper infrastructure myself, my application could simply make HTTP requests and consume structured JSON responses.

A minimal example looks like this:

import requests

headers = {"x-access-key": "YOUR_KEY"}

user = requests.get(
    "https://api.hikerapi.com/v2/user/by/username?username=google",
    headers=headers
).json()

resp = requests.get(
    "https://api.hikerapi.com/v2/user/followers",
    params={"user_id": user["pk"]},
    headers=headers
)

print(resp.json())

That reduced a surprising amount of complexity in my backend.

The tradeoffs

Moving to a hosted API isn't a perfect solution.

The most obvious downside is cost. Running your own scraper can be cheaper if you're operating at very low volume and don't mind maintaining it yourself. A hosted service also means depending on another company, which reduces some of the control you have over your stack.

If you enjoy optimizing infrastructure or need highly customized behavior, self-hosting may still be the better option.

On the other hand, if your goal is shipping product features, paying for managed infrastructure can be a reasonable tradeoff.

For me, it came down to opportunity cost.

Every hour spent debugging logins or recovering accounts was an hour I wasn't improving .

Would I make the same decision again?

Yes.

If I were starting from scratch today, I would spend less time building scraping infrastructure and more time building the product itself.

That doesn't mean self-hosting is the wrong approach. It's an excellent way to learn how these systems work, and for some projects it's still the right choice.

But once maintenance became the largest consumer of my development time, I realized I had optimized for the wrong thing.

Today I'd rather invest my time in features users actually notice than in fixing behind the scenes.

I'm part of HikerAPI's user-rewards program — they credit my account for posts like this. Sharing my actual experience.