Ranking Hashtags by Reach with Asyncio
I maintain a ranked table of the hashtags in my niche, sorted by the median play count of their recent posts, with a post-count column next to it so I can tell a real signal from three lucky videos. I regenerate it on demand in about the time of the slowest handful of HTTP requests, not the sum of all of them.
The ranking is the useful part. A hashtag with 4M posts and a median in the hundreds is a landfill. A hashtag with 60k posts and a median in the tens of thousands is where my content should go. You cannot eyeball this from the Instagram app, and you cannot get it from one request.
Here is how the script works.
The one call everything is built on
The whole thing started as this, against HikerAPI's REST endpoints:
import requests
headers = {"x-access-key": "YOUR_KEY"}
r = requests.get("https://api.hikerapi.com/v2/hashtag/medias/recent?name=food", headers=headers)
print(r.json())
One hashtag, one round trip, a few hundred milliseconds of waiting during which my CPU does nothing. With 150 candidate hashtags in a sequential loop, that is 150 sequential round trips. The work is entirely I/O-bound, which is the easy case: I just need many requests in flight at once.
Two things to fix before scaling it. The key goes in an environment variable, and the hashtag goes in params rather than string-concatenated into the URL, so anything with a non-ASCII character survives.
import os
BASE = "https://api.hikerapi.com/v2/hashtag/medias/recent"
HEADERS = {"x-access-key": os.environ["HIKER_KEY"]}
r = requests.get(BASE, params={"name": "food"}, headers=HEADERS, timeout=30)
r.raise_for_status()
Threads first, because it is four lines
If you have a few dozen hashtags, ThreadPoolExecutor wrapped around the call you already have is the entire upgrade. One caveat: requests.Session is not documented as thread-safe, so I give each worker thread its own via threading.local() rather than sharing one across the pool.
import os
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
BASE = "https://api.hikerapi.com/v2/hashtag/medias/recent"
HEADERS = {"x-access-key": os.environ["HIKER_KEY"]}
_local = threading.local()
def session():
if not hasattr(_local, "s"):
_local.s = requests.Session()
return _local.s
def fetch_hashtag(name):
r = session().get(BASE, params={"name": name}, headers=HEADERS, timeout=30)
r.raise_for_status()
return name, r.json()
def fetch_all(names, workers=8):
results = {}
with ThreadPoolExecutor(max_workers=workers) as pool:
futures = {pool.submit(fetch_hashtag, n): n for n in names}
for future in as_completed(futures):
name = futures[future]
try:
name, payload = future.result()
except Exception as exc:
print(f"{name}: {exc}")
continue
results[name] = payload
return results
max_workers is your concurrency limit. It is not a dial to turn up until things go fast; it is the thing that decides whether you get rate limited. More on that below.
This is fine and I used it for weeks. It stops being fine when the list grows, because every in-flight request costs you an OS thread.
Asyncio when the list gets long
Once I was fanning out across hundreds of hashtags, plus a second pass over the top performers, I moved to httpx.AsyncClient. Same mental model, no thread per request, and a Semaphore gives me an explicit knob for concurrency that is independent of the connection pool.
import asyncio
import os
import random
import httpx
BASE = "https://api.hikerapi.com/v2/hashtag/medias/recent"
HEADERS = {"x-access-key": os.environ["HIKER_KEY"]}
MAX_CONCURRENCY = 8
MAX_RETRIES = 4
def backoff(attempt):
return min(2 ** attempt, 30) + random.random()
def retry_delay(response, attempt):
header = response.headers.get("Retry-After")
if header:
try:
return float(header)
except ValueError:
pass
return backoff(attempt)
async def fetch_hashtag(client, sem, name):
async with sem:
for attempt in range(MAX_RETRIES):
try:
r = await client.get(BASE, params={"name": name}, headers=HEADERS)
except httpx.TransportError:
await asyncio.sleep(backoff(attempt))
continue
if r.status_code == 429:
await asyncio.sleep(retry_delay(r, attempt))
continue
if 500 <= r.status_code < 600:
await asyncio.sleep(backoff(attempt))
continue
r.raise_for_status()
return name, r.json()
raise RuntimeError(f"gave up on {name}")
async def fetch_all(names, concurrency=MAX_CONCURRENCY):
sem = asyncio.Semaphore(concurrency)
limits = httpx.Limits(max_connections=concurrency, max_keepalive_connections=concurrency)
async with httpx.AsyncClient(timeout=30, limits=limits) as client:
tasks = [fetch_hashtag(client, sem, n) for n in names]
settled = await asyncio.gather(*tasks, return_exceptions=True)
results = {}
for item in settled:
if isinstance(item, Exception):
print(item)
continue
name, payload = item
results[name] = payload
return results
Three details that matter more than they look:
return_exceptions=True means one dead hashtag does not cancel the other 149. Without it, gather propagates the first exception and you lose the whole batch.
The Semaphore and httpx.Limits are set to the same number on purpose. The semaphore caps requests in flight; the limits cap TCP connections. If the limits are lower, requests queue inside httpx where your retry logic cannot see them, and your timeouts start firing for the wrong reason.
Retries are inside the semaphore. A task that is sleeping off a 429 is still holding its slot, so the backoff actually reduces pressure on the API instead of letting another task take the slot immediately.
Rate limits are a concurrency problem, not a retry problem
Retries are the bandage. The fix is picking the right MAX_CONCURRENCY.
Your allowed rate depends on your plan, so read the limits for yours rather than guessing from mine. The method I use is boring and works: start at 4, run the full batch, count how many 429s showed up. Zero 429s means step up. Any 429s at all means step back down. You want a concurrency level where the retry path almost never runs, because retries are requests, and requests cost money.
Jitter in backoff() is not decoration. Without it, every task that hit a 429 in the same window wakes up at the same instant and hits the API simultaneously, which produces another wave of 429s. The random.random() smears them out.
I also respect Retry-After when it is present rather than overriding it with my own exponential curve. If the server tells you how long to wait, arguing with it just burns quota.
The last piece is caching. At per-request pricing, the expensive mistake is not concurrency, it is re-fetching data you already have because your script crashed on row 140. I write every raw payload to a JSON file keyed by hashtag and timestamp, and the fetch layer checks for a fresh-enough file before making a call. Parsing bugs then cost nothing to fix, because iterating on the parser never touches the network.
HikerAPI starts at $0.001 per request with 100 free to begin with, which is enough to tune your concurrency and confirm your parser against real payloads before you point it at a long list: HikerAPI Instagram scraper API.
Getting numbers out of the payloads
Response shapes nest differently across endpoints, and I did not want my ranking logic to care. So I walk each payload once and pull out anything that looks like a media object, identified by the engagement keys rather than by position in the tree.
MEDIA_KEYS = {"play_count", "view_count", "like_count"}
def walk_medias(node):
if isinstance(node, dict):
if MEDIA_KEYS & node.keys():
yield node
return
for value in node.values():
yield from walk_medias(value)
elif isinstance(node, list):
for value in node:
yield from walk_medias(value)
This is deliberately forgiving. When a payload comes back wrapped one level deeper than last month, nothing breaks.
Then the ranking. Median, not mean, because one viral post in a sample of twenty will drag a mean somewhere useless and tell me a dead hashtag is alive.
from statistics import median
def rank(results):
rows = []
for name, payload in results.items():
plays = []
for m in walk_medias(payload):
value = m.get("play_count") or m.get("view_count")
if value:
plays.append(value)
if len(plays) < 5:
continue
rows.append({
"hashtag": name,
"sampled": len(plays),
"median_plays": int(median(plays)),
})
return sorted(rows, key=lambda r: r["median_plays"], reverse=True)
def main(names):
results = asyncio.run(fetch_all(names))
for row in rank(results):
print(f"{row['hashtag']:<28} {row['median_plays']:>10,} (n={row['sampled']})")
The len(plays) < 5 guard drops hashtags where I got too thin a sample to say anything. Ranking on two posts is noise with a confident-looking number attached.
If an endpoint hands back a pagination cursor, I do not chase it on the first pass. I fan out wide and shallow across every candidate hashtag, rank them, then paginate only the ones near the top. Depth is expensive and most of the list does not deserve it.
Which one to use
Threads if your fan-out is in the dozens and you already have working requests code. The migration is a ThreadPoolExecutor and a thread-local session, and you keep every library you know.
Asyncio if you are in the hundreds or thousands, if you want one explicit place to express "no more than N requests in flight", or if this fetch is one stage in a pipeline that already has an event loop in it.
Both get you the same win, which is that total time collapses toward the latency of your slowest request instead of the sum of all of them. The part that actually took me iterations was not the concurrency. It was learning that max_workers is a budget decision, not a performance one.
Tags: #python #api #async
Top comments (0)