DEV Community

Oliver Monte
Oliver Monte

Posted on

What finally fixed my Instagram profile lookups — a pay-per-request API instead of my own scrapers

For about a year I've been building a small lead-facing tool that shows a snapshot of a public Instagram profile — name, profile photo, follower and post counts, whether the account is private. Nothing exotic. The hard part was never the UI. It was getting the data reliably.

I started the way everyone does: instagrapi + a couple of aged accounts + residential proxies. It worked on my machine for about two weeks at a time. Then a checkpoint would pop, a session would die, or Instagram would change a payload and half my fields came back empty. I was spending more time babysitting login sessions than building the actual product.

What I actually needed was boring: give it a username, get clean JSON back, don't make me think about accounts or proxies. I moved the lookups to hikerapi.com — a hosted endpoint where one GET with a key in the header returns the profile as JSON. The part I care about: no login session to keep alive, and the fields come back consistently enough that I normalize them once and move on. Pay-per-request also means I can reason about cost per lookup instead of amortizing a pile of proxy and account spend.

Tradeoffs / what didn't totally work: it's per-request, so for big bulk pulls the cost adds up and you'll want aggressive caching (I cache each profile for a few hours). It's also a third-party dependency, so you still have to handle empty responses for deleted or odd accounts gracefully. Compared to rolling my own with instagrapi, I traded "free but fragile and high-maintenance" for "costs money but predictable." For a product that needs to not break on a random Tuesday, that was the right trade. If you're doing a one-off research scrape, honestly your own instagrapi script is probably fine.

If I were starting over I'd put the caching layer in from day one instead of bolting it on after the first surprise bill.

I'm part of HikerAPI's user-rewards program — they credit my account for posts like this. Sharing my actual experience.

Top comments (0)