When I started working on , I needed a reliable way to retrieve Instagram data programmatically.
My first thought was to interact with Instagram directly using .
That approach can work for experimentation, but as soon as the application starts making repeated requests, things become more complicated.
Rate limits, blocks, authentication/session handling, changing responses, and maintenance can quickly become part of the project itself.
Eventually, I decided to move the Instagram data layer to HikerAPI.
HikerAPI provides a REST API for accessing public Instagram data, including profiles, posts, reels, stories, followers, comments, hashtags, locations, and other data through API endpoints.
The Use Case
The problem I was trying to solve was:
«»
For example, my workflow needed to:
- Start with an Instagram username.
- Retrieve the user's profile information.
- Get the user's media.
- Process the returned data inside my application.
The important part was that I didn't want the Instagram request layer to become the main thing I had to maintain.
I wanted something closer to a normal API integration.
Why Scraping Became Difficult
Direct scraping sounds simple:
import requests
url = "https://www.instagram.com//"
response = requests.get(url)
print(response.status_code)
The problem is that a production workflow isn't just about making one successful request.
You also have to think about things like:
- Rate limits
- Blocks
- Request failures
- Session management
- Authentication
- Changes to Instagram's responses
- Parsing HTML or internal responses
- Retries
- Proxies and infrastructure
- Maintaining the scraper when things change
For , this created more engineering overhead than I wanted.
The question became less about "Can I scrape this?" and more about:
"Do I want to maintain an Instagram scraping system?"
My answer was no.
What About instagrapi?
I also considered "instagrapi".
It's a useful option when you want Python-level control and are comfortable managing the Instagram client yourself.
For example, a typical workflow can look conceptually like:
from instagrapi import Client
cl = Client()
cl.login("USERNAME", "PASSWORD")
user = cl.user_info_by_username("")
medias = cl.user_medias(user.pk, amount=10)
print(medias)
The advantage is control.
Your application communicates through a Python client, and you can build your own logic around it.
The downside is that you also inherit more of the operational complexity.
You may need to deal with sessions, authentication, rate limiting, account health, retries, and changes in Instagram behavior.
For some projects, that tradeoff is completely reasonable.
For mine, I preferred moving that complexity behind an API.
Moving to a REST API
This is where HikerAPI fit my workflow better.
Instead of having my Python application communicate with Instagram directly, I could make normal HTTP requests to an API.
The authentication model is simple: the API key is sent through the "x-access-key" header.
The basic flow becomes:
My Python application
↓
HikerAPI REST endpoint
↓
JSON response
↓
My application processes the data
That is a much simpler boundary for my application.
Making the First Request
The request itself is straightforward.
Here is the complete example:
import requests
headers = {"x-access-key": "YOUR_KEY"}
user = requests.get(
"https://api.hikerapi.com/v2/user/by/username?username=nike",
headers=headers
).json()
resp = requests.get(
"https://api.hikerapi.com/v2/user/medias",
params={"user_id": user["pk"]},
headers=headers
)
print(resp.json())
The first request finds the user by username.
The response contains the user's "pk", which I can then pass to the media endpoint.
The second request retrieves the user's media.
The nice part is that the application doesn't need to understand Instagram's internal page structure. It just needs to understand the API response.
Keeping the API Key Safe
One thing I wouldn't recommend is hardcoding the real API key into production code.
Instead, I would keep it in an environment variable:
import os
import requests
API_KEY = os.environ["HIKERAPI_KEY"]
headers = {
"x-access-key": API_KEY
}
Then set the key outside the source code.
For example:
export HIKERAPI_KEY="your-real-key"
This makes it much easier to keep secrets out of Git repositories and shared code.
Handling API Errors
One mistake I've learned to avoid with API integrations is assuming every request will succeed.
Instead of immediately calling ".json()", I can inspect the response:
import requests
headers = {"x-access-key": "YOUR_KEY"}
response = requests.get(
"https://api.hikerapi.com/v2/user/by/username",
params={"username": "nike"},
headers=headers,
timeout=30
)
if response.ok:
user = response.json()
print(user)
else:
print("Request failed:", response.status_code)
print(response.text)
This gives my application a chance to handle failures gracefully.
For a real application, I would also consider retry logic, logging, timeouts, and validation of the returned JSON.
Why I Preferred This Architecture
The biggest benefit wasn't that the Python code was shorter.
It was that I could separate responsibilities.
My application is responsible for:
- Business logic
- Data processing
- Storage
- Authentication for my own users
- Scheduling
- Analytics
The API handles the Instagram-specific communication layer.
That separation makes the architecture easier for me to reason about.
The Cost
Of course, a hosted API isn't free.
HikerAPI uses pay-per-request pricing. Its current pricing starts at $0.001 per request on the Standard tier, with higher-volume tiers offering lower per-request rates, and new accounts receive 100 free requests for testing.
That means I need to consider request volume.
If my application makes 100 requests, the cost is very different from an application making millions of requests.
So before choosing a hosted API, I would calculate:
Expected requests per day
×
Cost per request
×
Days per month
Estimated monthly API cost
For small projects and prototypes, the cost can be easier to justify because I'm also saving development and maintenance time.
For very large workloads, I'd definitely benchmark the economics against running my own infrastructure.
Hosted API vs Scraping
There isn't one answer that's best for everyone.
Direct scraping
Pros:
- More control
- No third-party API markup
- Potentially lower direct API costs
- Full control over infrastructure
Cons:
- More maintenance
- Rate-limit handling
- Blocks
- Proxy/infrastructure considerations
- Parsing and response changes
- More Instagram-specific code
instagrapi
Pros:
- Python-friendly
- More direct control
- Useful for developers who want to manage the client themselves
- Can be integrated deeply into a Python application
Cons:
- You still own much of the operational complexity
- Sessions and authentication need attention
- Instagram changes can require maintenance
- Scaling requires more infrastructure planning
HikerAPI
Pros:
- Simple REST interface
- API-key authentication
- JSON responses
- Many Instagram-specific endpoints
- Pay-per-request model
- 100 free requests for testing
Cons:
- Adds a third-party dependency
- Costs money once usage exceeds the free requests
- Less control over the underlying Instagram communication
- API availability and endpoint changes become another dependency to monitor
For , the reduction in maintenance was worth the tradeoff.
What I Would Do Before Building Around It
I wouldn't immediately rewrite an entire application around any API.
I'd test the exact endpoints I need first.
For example:
import requests
headers = {"x-access-key": "YOUR_KEY"}
response = requests.get(
"https://api.hikerapi.com/v2/user/by/username",
params={"username": "nike"},
headers=headers,
timeout=30
)
print(response.status_code)
print(response.json())
Then I'd test:
- Expected users
- Invalid usernames
- Large accounts
- Different response cases
- API failures
- Request volume
- Response speed
- Data completeness
The 100 free requests make this initial evaluation relatively easy.
The Biggest Lesson
The biggest lesson I took from this wasn't simply "use an API instead of scraping."
It was that engineering time is also a cost.
I can spend days building and maintaining infrastructure around .
Or I can pay for a service that gives my application a cleaner interface and spend that engineering time on the actual product.
Neither option is universally better.
If you have low volume, need maximum control, or already have a stable scraping infrastructure, managing the stack yourself may make sense.
If you want a simpler integration and don't want Instagram-specific infrastructure to become a major part of your application, a hosted API can be a much more attractive option.
For me, HikerAPI gave me a straightforward REST layer between my application and Instagram data.
And sometimes, simplifying the architecture is more valuable than having complete control over every layer.
Top comments (0)