DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Two Thirds Of Stocktwits Messages Have No Sentiment — And The API Says So Explicitly

#webscraping #python #api #apify

Quick answer

Stocktwits' public symbol stream returns a sentiment field on every message — and on most of them its value is literally null. In a live pull of the AAPL stream just now: 30 messages, 11 tagged (Bullish/Bearish), 19 explicitly null, and zero with the key missing. So the tag is optional for the poster, not optional in the payload. If you build a bullish/bearish ratio by counting messages, you are dividing by the wrong denominator — roughly 63% of the stream has no opinion attached at all.

There's a second, nastier one in the same response: likes is absent, not zero, on messages nobody engaged with. Same pull: 10 of 30 messages had no likes key whatsoever.

Watching both fire against the live endpoint 📈 

from curl_cffi import requests

r = requests.get(
    "https://api.stocktwits.com/api/2/streams/symbol/AAPL.json",
    impersonate="chrome131", timeout=30,
)
messages = r.json()["messages"]

tagged  = sum(1 for m in messages if m["entities"].get("sentiment"))
explicit_null = sum(1 for m in messages if "sentiment" in m["entities"]
                    and m["entities"]["sentiment"] is None)
key_absent = sum(1 for m in messages if "sentiment" not in m["entities"])

print(len(messages), tagged, explicit_null, key_absent)
# 30 11 19 0

print(sum(1 for m in messages if "likes" not in m))
# 10
Enter fullscreen mode Exit fullscreen mode

A tagged message nests one level deeper than you'd guess, too — it's {"basic": "Bullish"}, not the bare string:

m["entities"]["sentiment"]           # {'basic': 'Bullish'}
m["entities"]["sentiment"]["basic"]  # 'Bullish'
Enter fullscreen mode Exit fullscreen mode

Why the two failure modes are different, and one is worse

The sentiment case is honest: the key is always there, so m["entities"]["sentiment"] never raises. You get None, you handle it, you move on. The danger is purely analytical — treating "untagged" as "neutral" and folding it into a ratio.

The likes case is the one that bites at runtime. m["likes"]["total"] raises KeyError on a third of a typical stream. The instinctive patch is worse than the crash:

likes = m.get("likes", {}).get("total", 0)   # ← silently wrong
Enter fullscreen mode Exit fullscreen mode

That collapses "this message has no engagement data" into "this message has zero likes." Those are not the same claim, and if you are ranking messages by engagement, the difference decides your ordering. We keep the distinction: absent stays None, a real zero stays 0.

What we do about it

Our Stocktwits Sentiment Scraper treats sentiment as a first-class column with three honest states — Bullish, Bearish, null — rather than inventing a "Neutral" bucket that the API never claimed existed. Engagement fields preserve absent-vs-zero. Every row is a validated Pydantic model, so a shape change upstream fails loudly instead of quietly writing None into your dataset.

We also handle the boring, unglamorous part: this is an undocumented internal API that rate-limits. We rotate a fresh proxy exit on every retry rather than hammering one IP into a 429 — because a scraper that gets itself banned halfway through your symbol list is worth less than no scraper at all. Blocks, retries, and backoff are our problem, not yours.

Try it

The Actor is on the Apify Store — give it a list of symbols and it returns one row per message with sentiment, author, timestamp and engagement intact.

Counts in this post are from a live pull on 2026-09-10 and will drift with the stream — the shapes won't.

Top comments (0)