Lessons Learned: Dealing With Instagram Rate Limits and Blocks

When I started building , one of the first approaches I tried was interacting with Instagram directly.

It seemed straightforward: make requests, collect the public data I needed, process it, and move on.

It worked at first.

Then the rate limits and blocks started becoming a problem.

What Went Wrong

My initial approach looked roughly like this:

import requests

response = requests.get(
"https://www.instagram.com//",
headers={"User-Agent": "Mozilla/5.0"}
)

print(response.status_code)

For , this approach became unreliable.

Requests that worked previously could start returning errors, responses could change, and repeated requests made the workflow increasingly difficult to maintain.

The biggest lesson for me was:

Getting a request to work once is very different from building a reliable data pipeline.

I could keep adding delays, retries, different headers, and other workarounds, but that also meant more code to maintain and more things that could break.

The Problem With Handling Everything Myself

I considered several ways to deal with the situation:

• Adding delays between requests
• Implementing retry logic
• Managing sessions
• Handling different response formats
• Trying to reduce request frequency

Some of these techniques helped temporarily, but they didn't solve the underlying problem.

I didn't really want my application to become a project for managing Instagram's request behavior.

I wanted it to focus on .

That was the point where I started looking at hosted APIs instead.

Why I Moved to HikerAPI

I eventually moved the Instagram data requests to HikerAPI.

"HikerAPI" (https://reference-url-citation.invalid/1)

The main attraction was simplicity.

Instead of my application having to handle the Instagram communication directly, I could make a normal HTTP request and work with the JSON response.

HikerAPI provides REST endpoints for things such as user profiles, posts, reels, stories, followers, following, hashtags, and locations. It also uses API-key authentication rather than requiring me to build my own Instagram session-management layer.

The pricing is pay-per-request. The current pricing page lists plans starting at $0.60 per 1,000 requests, while the popular Standard tier is $1 per 1,000 requests, and new accounts receive 100 free requests.

For my use case, predictable per-request costs were easier to reason about than spending time maintaining a custom scraping setup.

A Simple Request

One thing I liked was that I could use a standard Python HTTP client.

For example:

import requests

headers = {"x-access-key": "YOUR_KEY"}

user = requests.get(
"https://api.hikerapi.com/v2/user/by/username?username=nasa",
headers=headers
).json()

r = requests.get(
"https://api.hikerapi.com/v2/user/followers",
params={"user_id": user["pk"]},
headers=headers
)

print(r.json())

There is no special networking layer here.

It's just an HTTP request, an API key, and JSON.

That makes it easy to integrate into an existing Python project.

What Changed in My Architecture

Before:

My application
↓
Instagram
↓
Rate limits / blocks / changing behavior
↓
My application handles failures

After:

My application
↓
HikerAPI REST API
↓
JSON response
↓
My application

This doesn't magically eliminate every possible failure. An external API is still another dependency, and its availability, endpoint behavior, pricing, and limits need to be considered.

But it moved a lot of Instagram-specific complexity away from my application.

The Tradeoffs

Moving to a hosted API isn't automatically the right choice for every project.

There are some clear tradeoffs.

You Pay for Usage

Direct requests might appear cheaper because there isn't an API bill.

With a hosted API, every request can have a cost.

For a small project, this may be negligible. At large scale, you need to calculate the cost carefully.

You Add a Dependency

Your application now depends on another service.

If the API has an outage or an endpoint changes, your application needs to handle it.

You Have Less Control

With your own implementation, you control the entire request and processing pipeline.

With a hosted API, part of that infrastructure is outside your control.

But You Save Development Time

This was the biggest benefit for me.

Instead of spending more time dealing with , I could spend that time working on .

For me, that tradeoff made sense.

What I Would Do Differently

If I were starting again, I would evaluate the API layer much earlier.

I'd first define:

1. What Instagram data I actually need
2. How many requests I expect to make
3. Whether the data needs to be real-time
4. How much failure/retry handling my application needs
5. What the monthly API cost would look like
6. Whether the data source and use case comply with the relevant platform rules

Then I'd test the API with a small number of requests before designing the entire application around it.

The 100 free requests available with HikerAPI are useful for this kind of initial evaluation.

Final Takeaway

The main lesson wasn't simply "use HikerAPI."

It was that reliability has a cost, and developer time is part of that cost.

Trying to handle every Instagram-specific problem myself initially seemed like the cheaper option.

Eventually, I realized that maintaining was taking attention away from the actual application.

Moving that responsibility behind a hosted REST API gave me a much simpler interface:

Send a request → receive JSON → build the feature.

For , that was a much better tradeoff.

python #webscraping #api