Scraping 150k+ Instagram followers reliably: batching, resume-on-error, and enrichment

Marcos Faccin da Silva — Sun, 06 Sep 2026 00:20:26 +0000

I run a small AI/automation consultancy in Brazil, and a recent lead-research project needed the full follower list of a public Instagram profile — about 153,000 followers — plus enrichment (bio, public email/phone) to find business accounts worth contacting.

The problem

Pulling a list that size is never one API call. Instagram reports ~153,628 followers; you get them page by page, and any long-running extraction WILL hit a failed request eventually. If your pipeline can't resume, you start over from zero — which is expensive and slow.

What I built

The pipeline runs on n8n with Supabase as the datastore:

Batched extraction — followers are downloaded in batches of up to 10,000 per cycle, on a schedule, instead of one giant run.
Resume on error — every page cursor and count is persisted. When a request fails mid-run (in one run it stopped at 4,782 followers after 96 pages read), the job logs the error, emails me a status report, and picks up from the same point on the next cycle instead of restarting.
Enrichment pass — a second workflow walks the stored followers and pulls profile details, flagging commercial accounts and any public email/phone in the bio. Personal/private accounts return no contact data, which the report counts separately.
Email reports — each cycle sends me a summary: profile, followers reported vs. downloaded, pages read, batch name, and the exact error if one occurred.

For the Instagram data layer I used HikerAPI — I tested a few other options first, and it won on pricing and rate limits for this volume. It handled the pagination fine: the run above made 100+ requests without me managing sessions or proxies myself.

Tradeoffs / what didn't go perfectly
Long extractions still fail sometimes (timeouts); resume logic is not optional at this scale, whatever API you use.
Early days for me on this stack: so far it has worked well, but I'm still collecting more data before I'd call the pipeline battle-tested. I'll know more after a few full 150k-follower cycles.
Enrichment yield is modest: most personal/private accounts expose no public contact data, so plan conversion expectations accordingly.
Takeaway

Treat large social extractions as a resumable batch job, not a script: persist cursors, cap batch sizes, and report every cycle. The datastore + scheduler matter more than the scraping API itself.

Disclosure: I'm part of HikerAPI's user-rewards program — they credit my account for posts like this. Sharing my actual experience.

DEV Community: Marcos Faccin da Silva

Scraping 150k+ Instagram followers reliably: batching, resume-on-error, and enrichment