DEV Community

Jinyoung Byun
Jinyoung Byun

Posted on

Ranking Hashtags by Reach with Asyncio

Ranking Hashtags by Reach with Asyncio

I maintain a ranked table of the hashtags in my niche, sorted by the median play count of their recent posts, with a post-count column next to it so I can tell a real signal from three lucky videos. I regenerate it on demand in about the time of the slowest handful of HTTP requests, not the sum of all of them.

The ranking is the useful part. A hashtag with 4M posts and a median in the hundreds is a landfill. A hashtag with 60k posts and a median in the tens of thousands is where my content should go. You cannot eyeball this from the Instagram app, and you cannot get it from one request.

Here is how the script works.

The one call everything is built on

The whole thing started as this, against HikerAPI's REST endpoints:

import requests

headers = {"x-access-key": "YOUR_KEY"}
r = requests.get("https://api.hikerapi.com/v2/hashtag/medias/recent?name=food", headers=headers)
print(r.json())
Enter fullscreen mode Exit fullscreen mode

One hashtag, one round trip, a few hundred milliseconds of waiting during which my CPU does nothing. With 150 candidate hashtags in a sequential loop, that is 150 sequential round trips. The work is entirely I/O-bound, which is the easy case: I just need many requests in flight at once.

Two things to fix before scaling it. The key goes in an environment variable, and the hashtag goes in params rather than string-concatenated into the URL, so anything with a non-ASCII character survives.

import os

BASE = "https://api.hikerapi.com/v2/hashtag/medias/recent"
HEADERS = {"x-access-key": os.environ["HIKER_KEY"]}

r = requests.get(BASE, params={"name": "food"}, headers=HEADERS, timeout=30)
r.raise_for_status()
Enter fullscreen mode Exit fullscreen mode

Threads first, because it is four lines

If you have a few dozen hashtags, ThreadPoolExecutor wrapped around the call you already have is the entire upgrade. One caveat: requests.Session is not documented as thread-safe, so I give each worker thread its own via threading.local() rather than sharing one across the pool.

import os
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed

import requests

BASE = "https://api.hikerapi.com/v2/hashtag/medias/recent"
HEADERS = {"x-access-key": os.environ["HIKER_KEY"]}

_local = threading.local()


def session():
    if not hasattr(_local, "s"):
        _local.s = requests.Session()
    return _local.s


def fetch_hashtag(name):
    r = session().get(BASE, params={"name": name}, headers=HEADERS, timeout=30)
    r.raise_for_status()
    return name, r.json()


def fetch_all(names, workers=8):
    results = {}
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(fetch_hashtag, n): n for n in names}
        for future in as_completed(futures):
            name = futures[future]
            try:
                name, payload = future.result()
            except Exception as exc:
                print(f"{name}: {exc}")
                continue
            results[name] = payload
    return results
Enter fullscreen mode Exit fullscreen mode

max_workers is your concurrency limit. It is not a dial to turn up until things go fast; it is the thing that decides whether you get rate limited. More on that below.

This is fine and I used it for weeks. It stops being fine when the list grows, because every in-flight request costs you an OS thread.

Asyncio when the list gets long

Once I was fanning out across hundreds of hashtags, plus a second pass over the top performers, I moved to httpx.AsyncClient. Same mental model, no thread per request, and a Semaphore gives me an explicit knob for concurrency that is independent of the connection pool.

import asyncio
import os
import random

import httpx

BASE = "https://api.hikerapi.com/v2/hashtag/medias/recent"
HEADERS = {"x-access-key": os.environ["HIKER_KEY"]}

MAX_CONCURRENCY = 8
MAX_RETRIES = 4


def backoff(attempt):
    return min(2 ** attempt, 30) + random.random()


def retry_delay(response, attempt):
    header = response.headers.get("Retry-After")
    if header:
        try:
            return float(header)
        except ValueError:
            pass
    return backoff(attempt)


async def fetch_hashtag(client, sem, name):
    async with sem:
        for attempt in range(MAX_RETRIES):
            try:
                r = await client.get(BASE, params={"name": name}, headers=HEADERS)
            except httpx.TransportError:
                await asyncio.sleep(backoff(attempt))
                continue

            if r.status_code == 429:
                await asyncio.sleep(retry_delay(r, attempt))
                continue

            if 500 <= r.status_code < 600:
                await asyncio.sleep(backoff(attempt))
                continue

            r.raise_for_status()
            return name, r.json()

    raise RuntimeError(f"gave up on {name}")


async def fetch_all(names, concurrency=MAX_CONCURRENCY):
    sem = asyncio.Semaphore(concurrency)
    limits = httpx.Limits(max_connections=concurrency, max_keepalive_connections=concurrency)

    async with httpx.AsyncClient(timeout=30, limits=limits) as client:
        tasks = [fetch_hashtag(client, sem, n) for n in names]
        settled = await asyncio.gather(*tasks, return_exceptions=True)

    results = {}
    for item in settled:
        if isinstance(item, Exception):
            print(item)
            continue
        name, payload = item
        results[name] = payload
    return results
Enter fullscreen mode Exit fullscreen mode

Three details that matter more than they look:

return_exceptions=True means one dead hashtag does not cancel the other 149. Without it, gather propagates the first exception and you lose the whole batch.

The Semaphore and httpx.Limits are set to the same number on purpose. The semaphore caps requests in flight; the limits cap TCP connections. If the limits are lower, requests queue inside httpx where your retry logic cannot see them, and your timeouts start firing for the wrong reason.

Retries are inside the semaphore. A task that is sleeping off a 429 is still holding its slot, so the backoff actually reduces pressure on the API instead of letting another task take the slot immediately.

Rate limits are a concurrency problem, not a retry problem

Retries are the bandage. The fix is picking the right MAX_CONCURRENCY.

Your allowed rate depends on your plan, so read the limits for yours rather than guessing from mine. The method I use is boring and works: start at 4, run the full batch, count how many 429s showed up. Zero 429s means step up. Any 429s at all means step back down. You want a concurrency level where the retry path almost never runs, because retries are requests, and requests cost money.

Jitter in backoff() is not decoration. Without it, every task that hit a 429 in the same window wakes up at the same instant and hits the API simultaneously, which produces another wave of 429s. The random.random() smears them out.

I also respect Retry-After when it is present rather than overriding it with my own exponential curve. If the server tells you how long to wait, arguing with it just burns quota.

The last piece is caching. At per-request pricing, the expensive mistake is not concurrency, it is re-fetching data you already have because your script crashed on row 140. I write every raw payload to a JSON file keyed by hashtag and timestamp, and the fetch layer checks for a fresh-enough file before making a call. Parsing bugs then cost nothing to fix, because iterating on the parser never touches the network.

HikerAPI starts at $0.001 per request with 100 free to begin with, which is enough to tune your concurrency and confirm your parser against real payloads before you point it at a long list: HikerAPI Instagram scraper API.

Getting numbers out of the payloads

Response shapes nest differently across endpoints, and I did not want my ranking logic to care. So I walk each payload once and pull out anything that looks like a media object, identified by the engagement keys rather than by position in the tree.

MEDIA_KEYS = {"play_count", "view_count", "like_count"}


def walk_medias(node):
    if isinstance(node, dict):
        if MEDIA_KEYS & node.keys():
            yield node
            return
        for value in node.values():
            yield from walk_medias(value)
    elif isinstance(node, list):
        for value in node:
            yield from walk_medias(value)
Enter fullscreen mode Exit fullscreen mode

This is deliberately forgiving. When a payload comes back wrapped one level deeper than last month, nothing breaks.

Then the ranking. Median, not mean, because one viral post in a sample of twenty will drag a mean somewhere useless and tell me a dead hashtag is alive.

from statistics import median


def rank(results):
    rows = []
    for name, payload in results.items():
        plays = []
        for m in walk_medias(payload):
            value = m.get("play_count") or m.get("view_count")
            if value:
                plays.append(value)

        if len(plays) < 5:
            continue

        rows.append({
            "hashtag": name,
            "sampled": len(plays),
            "median_plays": int(median(plays)),
        })

    return sorted(rows, key=lambda r: r["median_plays"], reverse=True)


def main(names):
    results = asyncio.run(fetch_all(names))
    for row in rank(results):
        print(f"{row['hashtag']:<28} {row['median_plays']:>10,}  (n={row['sampled']})")
Enter fullscreen mode Exit fullscreen mode

The len(plays) < 5 guard drops hashtags where I got too thin a sample to say anything. Ranking on two posts is noise with a confident-looking number attached.

If an endpoint hands back a pagination cursor, I do not chase it on the first pass. I fan out wide and shallow across every candidate hashtag, rank them, then paginate only the ones near the top. Depth is expensive and most of the list does not deserve it.

Which one to use

Threads if your fan-out is in the dozens and you already have working requests code. The migration is a ThreadPoolExecutor and a thread-local session, and you keep every library you know.

Asyncio if you are in the hundreds or thousands, if you want one explicit place to express "no more than N requests in flight", or if this fetch is one stage in a pipeline that already has an event loop in it.

Both get you the same win, which is that total time collapses toward the latency of your slowest request instead of the sum of all of them. The part that actually took me iterations was not the concurrency. It was learning that max_workers is a budget decision, not a performance one.


Tags: #python #api #async

Top comments (0)