The first version of my Instagram giveaway picker worked perfectly.
Until it didn’t.
The idea was simple: get the comments under a giveaway post, process them, and pick a winner. I initially handled the Instagram data collection myself because it seemed unnecessary to use another service for something that sounded this straightforward.
Then requests started failing. I retried them. Some worked again. Others didn’t. Then I ran into blocks and rate limits.
At some point, I realized I was spending more time trying to collect the comments than actually working on the giveaway picker.
The difficult part wasn’t picking a winner
Once I have the comments, the rest is fairly ordinary code.
The frustrating part was reliably getting the data in the first place.
Instagram obviously has good reasons to limit automated requests, but from my side that meant dealing with things like failed requests, rate limits, blocks, retries, and changes in behavior.
My first instinct was to keep fixing each problem myself.
That works, but I learned that there’s a difference between getting a scraper to work once and keeping it working.
I moved the data collection to an API
Eventually I decided I didn’t actually want to maintain the Instagram scraping layer myself.
I moved that part to HikerAPI, which provides Instagram data through a hosted REST API.
A basic Python request looks like this:
import requests
headers = {"x-access-key": "YOUR_KEY"}
user = requests.get(
"https://api.hikerapi.com/v2/user/by/username?username=instagram",
headers=headers
).json()
r = requests.get(
f"https://api.hikerapi.com/gql/user/followers/chunk?user_id={user['pk']}",
headers=headers
)
print(r.json())
For my giveaway use case, the important change wasn’t writing less Python. It was removing infrastructure that I didn’t really want to maintain.
Instead of spending time figuring out why Instagram rejected a particular request, I can focus on processing the data my application needs.
An API doesn’t remove every problem
Moving to an API doesn’t mean errors suddenly disappear.
I still need to handle failed requests properly. I still need sensible retry logic. I still need to think about rate limits and what happens if the data I expect isn’t returned.
And there’s obviously a financial tradeoff.
HikerAPI starts at $0.001 per request and offers 100 free requests, but using a hosted service means accepting an ongoing API cost instead of trying to run everything yourself.
For a large enough system, that calculation could look very different.
For my use case, though, developer time was becoming the more annoying cost.
The main lesson
The biggest thing I learned was that building the feature and maintaining the data source are two separate problems.
I wanted to build a giveaway picker.
I accidentally started maintaining Instagram scraping infrastructure.
There’s nothing inherently wrong with doing that yourself, and for some projects it will absolutely make sense. But in my case, outsourcing that part made the project simpler.
Sometimes the code you stop maintaining is more valuable than the code you write.
Top comments (0)