DEV Community

Prospy
Prospy

Posted on

Find buying-intent posts on Hacker News with ~100 lines of Python

#python #tutorial #opensource #marketing

People on Hacker News regularly say what they need in plain words: "is there a tool that…", "looking for an alternative to…", "any recommendations for…". If you build a product, those posts are useful. Sometimes you can help directly, and even when you can't, they show you how people describe the problem.

This post walks through a small, dependency-free Python watcher that finds those posts using only HN's public, read-only APIs, and then scores them for intent. The code comes from two MIT-licensed repos:

  • hn-keyword-watch: fetches and matches (the core module is about 100 lines of code once you drop docstrings and blank lines)
  • intent-score: scores text 0–100 with editable regex rules

Everything below uses the standard library only (Python 3.9+).

Two official data sources

HN has two read-only APIs, and they suit different jobs.

1. The official HN Firebase API (hacker-news.firebaseio.com/v0). Every story and comment is an item with an increasing integer id. maxitem.json gives you the newest id, and you walk backwards from there. It's the most direct source, but it's one HTTP request per item, and there's no search.

2. The Algolia HN Search API (hn.algolia.com/api/v1). This is full-text search over HN. The search_by_date endpoint returns the newest matches first, and numericFilters=created_at_i>… restricts the results to a time window. It's far fewer requests for keyword work, so it's the default in the tool.

The HTTP helper is just urllib with an honest User-Agent:

ALGOLIA_URL = "https://hn.algolia.com/api/v1/search_by_date"
FIREBASE_URL = "https://hacker-news.firebaseio.com/v0"

def http_get_json(url: str, timeout: float = 15.0) -> object:
    req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.loads(resp.read().decode("utf-8"))
Enter fullscreen mode Exit fullscreen mode

Searching with Algolia 

def search_algolia(query, since=None, hits_per_page=50,
                   tags="(story,comment)", fetch=http_get_json):
    params = {"query": query, "tags": tags, "hitsPerPage": str(hits_per_page)}
    if since:
        params["numericFilters"] = f"created_at_i>{int(since)}"
    data = fetch(f"{ALGOLIA_URL}?{urllib.parse.urlencode(params)}")
    return [_from_algolia(h) for h in data.get("hits", [])]
Enter fullscreen mode Exit fullscreen mode

tags="(story,comment)" is an OR filter, so you get both stories and comments. Comments are where most "can anyone recommend…" posts live. The fetch parameter is there so the tests can pass in a fake fetcher and never touch the network.

Walking the newest items with Firebase 

def scan_firebase(count=200, fetch=http_get_json, delay=0.0):
    max_id = int(fetch(f"{FIREBASE_URL}/maxitem.json"))
    for item_id in range(max_id, max(max_id - count, 0), -1):
        item = fetch(f"{FIREBASE_URL}/item/{item_id}.json")
        if delay:
            time.sleep(delay)
        if not isinstance(item, dict) or item.get("deleted") or item.get("dead"):
            continue
        if item.get("type") not in ("story", "comment"):
            continue
        yield _from_firebase(item)
Enter fullscreen mode Exit fullscreen mode

Skip deleted and dead items, and ignore polls and job posts. HN text fields are HTML, so the tool strips the tags and unescapes the entities before matching:

def clean_html(text):
    if not text:
        return ""
    text = text.replace("<p>", "\n\n")
    return html.unescape(_TAG_RE.sub("", text)).strip()
Enter fullscreen mode Exit fullscreen mode

Whole-word phrase matching

Plain substring matching gets noisy fast: crm matches "crmify", and "looking for a tool" misses "looking for a\ntool". Each phrase is compiled into a regex that only matches whole words and accepts any whitespace between them:

def _compile(phrases):
    out = []
    for p in phrases:
        p = p.strip()
        if not p:
            continue
        body = r"\s+".join(re.escape(w) for w in p.split())
        out.append((p, re.compile(rf"(?<!\w){body}(?!\w)", re.IGNORECASE)))
    return out
Enter fullscreen mode Exit fullscreen mode

(?<!\w) and (?!\w) act like \b, but they still work when a phrase starts or ends with punctuation, as in "c++". A quick check:

>>> match_text("Need a CRM.", ["crm"])
['crm']
>>> match_text("crmify", ["crm"])
[]
>>> match_text("looking   for a\ntool", ["looking for a tool"])
['looking for a tool']
Enter fullscreen mode Exit fullscreen mode

With Algolia, each phrase is sent as a quoted query, and every hit is then checked again locally with the same regex. Algolia's matching is fuzzier than you'd want for this, and the local check is what keeps the results honest.

Rate-limit etiquette

Both APIs are free and public, so use them gently:

  • Firebase: the HN API README currently says "There is currently no rate limit." Treat that as a courtesy, not an invitation. Scanning 150 items took roughly 20 seconds in my run, since it's one request per item. Keep --scan modest, and use the delay parameter if you run it often.
  • Algolia: prefer it for keyword work. One request per phrase replaces hundreds of item fetches. The CLI sleeps 200 ms between phrases. Algolia publishes its usage limits on the API page, so stay well under them.
  • Polling: --watch enforces a minimum interval of 60 seconds and remembers ids it has already printed. Every 10–15 minutes is plenty for HN.
  • Send a descriptive User-Agent so the operators can tell what's hitting them.
  • Read only: the tool never posts or votes. If you find a post you can genuinely help with, reply by hand.

Scoring with intent-score

A phrase match only tells you a phrase appeared. "Alternative to" shows up just as often in "X is a credible alternative to Y" as in "I need an alternative to Y". intent-score adds a second, transparent filter. Each rule in rules.json is {id, pattern, weight, reason}, every matching rule adds its weight once, and the total is clamped to 0–100:

def score(self, text):
    raw, reasons, ids = 0, [], []
    for rule in self.rules:
        if rule.regex.search(text or ""):
            raw += rule.weight
            ids.append(rule.id)
            reasons.append(f"{'+' if rule.weight >= 0 else ''}{rule.weight} {rule.reason}")
    value = max(0, min(100, raw))
    return Score(score=value, label=self.label_for(value), reasons=reasons, rules=ids)
Enter fullscreen mode Exit fullscreen mode

Negative weights cover self-promotion, job posts and "not looking for…". Here's a live example:

$ intent-score "Show HN: Balsio – a free European alternative to WhatsApp for work chats"
  0 low    Show HN: Balsio – a free European alternative to WhatsApp for work chats
           +25 evaluating alternatives; -30 self-promotion, not buying
Enter fullscreen mode Exit fullscreen mode

That's a launch, not a buyer, so it drops to zero, and the reasons tell you exactly why.

Putting it together: a real run

Both tools speak JSON lines, so they compose in a shell pipe. This is a fresh, read-only run from 23 October 2026 (usernames removed, output trimmed):

$ hn-keyword-watch "looking for a tool" "alternative to" "any recommendations" \
    "is there a tool" --hours 48 --json > hn.jsonl
$ wc -l < hn.jsonl
23
$ intent-score --jsonl hn.jsonl --field title,text --min 25
 25 medium Docker is weird I have recently discovered INCUS, very nice alternative to docker when a simple c...
           +25 evaluating alternatives
 30 medium What Is Gridifcation I love jazz, but my problem with newer stuff is a lot of it feels like the c...
           +25 asks for recommendations; +5 asks a question
 55 medium On-Device Dictation I just open-sourced a project for on-device dictation, seeing if anyone is lo...
           +30 asks for a solution; +25 evaluating alternatives
 ...
Enter fullscreen mode Exit fullscreen mode

The honest result: of 23 phrase matches over 48 hours, 18 scored "medium", 5 scored "low", and none scored "high". Most of the "alternative to" hits were people comparing technologies, not shopping. A separate Firebase scan of the newest 150 items found no matches at all. That's normal. Real buying-intent posts on HN are fairly rare, which is exactly why it's worth filtering automatically instead of reading everything.

What to do with that:

  1. Tighten your phrases. "alternative to hubspot" or "looking for a crm" beats a bare "alternative to".
  2. Tune the rules. Copy rules.json, add patterns for your niche, and pass --rules my_rules.json.
  3. Raise --min once you trust the rules, for example --min 60 for "high" only.
  4. Read before you reply. A score is a first filter, not a verdict.

Try it 

git clone https://codeberg.org/prospy/hn-keyword-watch
git clone https://codeberg.org/prospy/intent-score
pip install ./hn-keyword-watch ./intent-score

hn-keyword-watch "is there a tool" "looking for a" --hours 24 --json \
  | intent-score --jsonl - --field title,text --min 25
Enter fullscreen mode Exit fullscreen mode

Both repos have unit tests that run offline: PYTHONPATH=src python -m unittest discover -s tests -v. If you watch Bluesky too, bsky-keyword-watch does the same thing, read-only, with the public AT Protocol searchPosts endpoint.

If you'd rather not run this yourself, Prospy does social listening and AI lead generation across X/Twitter, Reddit, Bluesky and Hacker News, surfacing buying-intent posts.

Issues and pull requests for better rules are welcome on Codeberg, especially false positives you've run into.

Top comments (0)