What Broke When I Started Tracking Instagram Follower Growth Across Multiple Accounts?

My follower-growth chart stopped updating on time when I added more Instagram accounts to my side project. The problem was not the chart or the database. It was the data collection loop: each HTTP request waited for the previous one to finish.

For a handful of accounts, sequential requests were acceptable. As the list grew, network latency accumulated, and refreshing the dataset became slower than it needed to be.

I changed the collection layer to fetch independent results concurrently using Python threads. The same approach works for other API-driven monitoring tasks, provided concurrency stays within the provider's rate limits.

Start With a Simple HikerAPI Request

I use HikerAPI as the REST interface for retrieving Instagram-related data. Its requests use an x-access-key header, and the basic Python implementation looks like this:

import requests

headers = {"x-access-key": "YOUR_KEY"}
r = requests.get(
"https://api.hikerapi.com/v2/fbsearch/topsearch",
params={"query": "travel photographer"},
headers=headers,
)
print(r.json())

This example searches for the query travel photographer. It demonstrates the request structure; it does not, by itself, retrieve follower counts or historical follower growth.

For the charting project, I use the same HTTP-request pattern with the appropriate account lookup or profile-data endpoint. I then store timestamped follower-count observations so that I can calculate changes over time.

The key optimization is straightforward: independent requests do not need to run one after another.

Parallelize Requests With a Thread Pool

Because HTTP requests spend much of their time waiting for network responses, threads are a practical starting point. Python's concurrent.futures.ThreadPoolExecutor handles the scheduling without requiring an asynchronous rewrite.

Here is a reusable pattern for concurrent requests:

import os
import requests
from concurrent.futures import ThreadPoolExecutor, as_completed

API_KEY = os.environ["HIKERAPI_KEY"]
URL = "https://api.hikerapi.com/v2/fbsearch/topsearch"

def fetch_search(query):
response = requests.get(
URL,
params={"query": query},
headers={"x-access-key": API_KEY},
timeout=20,
)
response.raise_for_status()
return query, response.json()

queries = [
"travel photographer",
"landscape photographer",
"street photographer",
"wildlife photographer",
]

with ThreadPoolExecutor(max_workers=4) as executor:
futures = [
executor.submit(fetch_search, query)
for query in queries
]

for future in as_completed(futures):
    try:
        query, data = future.result()
        print(query, data)
    except requests.RequestException as exc:
        print(f"Request failed: {exc}")

Each worker performs an independent request, and as_completed() processes results as they arrive rather than waiting for submission order.

The example uses search queries to demonstrate concurrency. For follower tracking, I would substitute the relevant profile endpoint and account identifiers, then validate the returned follower-count field before saving it. The endpoint and response schema should be confirmed against the API documentation.

Why Not Use Asyncio?

An asynchronous client such as httpx.AsyncClient or aiohttp is another reasonable option. Async I/O can handle many concurrent connections without allocating a thread to each request.

For a small side project, I prefer threads when the existing code already uses requests. The implementation is easy to understand, and a bounded thread pool gives me a clear concurrency limit.

I would consider async when the collector needs to manage a much larger number of simultaneous network operations or already runs inside an asynchronous application. Neither approach removes API rate limits.

Handle Rate Limits Before Scaling Up

More workers do not automatically mean more useful data. If concurrency exceeds the API's permitted request rate, requests may be throttled or rejected.

I keep concurrency bounded, use request timeouts, and distinguish rate-limit responses from other HTTP errors. When the provider returns HTTP 429, I respect any Retry-After header and retry with exponential backoff and jitter. Retries should be bounded so a failing service cannot keep the collector running indefinitely.

HikerAPI advertises pay-per-request pricing starting at $0.001 per request and 100 free requests to start. I track request volume alongside collection duration, since polling frequency and the number of monitored accounts determine much of the ongoing cost.

The HikerAPI Instagram API reference is the place to verify the appropriate endpoints and access requirements.

Store Observations, Not Just Current Counts

Follower growth is a time series. I store each observation with an account identifier, collection timestamp, and follower count. The change between two observations is:

growth = current_followers - previous_followers

A database with a unique constraint on the account and observation timestamp helps prevent duplicate samples. Failed requests should not create zero-valued observations, because that would make the chart report a false drop.

Concurrent collection reduces waiting time, but data quality still depends on consistent timestamps, validated responses, and sensible polling intervals.

What Changed

The useful improvement was not simply adding threads. It was separating collection from persistence and charting, then running independent network requests concurrently under a fixed limit.

For this kind of side project, I would start with a small thread pool, measure the actual collection time, and increase concurrency only when the API limits and observed latency justify it. That is easier to maintain than starting with an elaborate asynchronous architecture before the workload requires one.

python #api #async