I've been building a small tool that watches Bluesky and Mastodon for posts that mention a keyword: your product, your project, your name. Matching a keyword in a post sounds like the trivial part. It wasn't.
The first surprise: who is actually posting
Before writing any matching code I wanted to know which languages matter. So I pulled the latest ~100 Bluesky search results per language for two keywords and measured how fast they came in.
| Keyword | en | ja | es | pt | fr | de | ko |
|---|---|---|---|---|---|---|---|
| chatgpt (posts/hour) | 47.5 | 11.9 | 2.2 | 2.2 | 2.2 | 2.1 | 0.3 |
| netflix (posts/hour) | 51.1 | 3.9 | 3.5 | 2.6 | 3.2 | 1.5 | 0.1 |
English dominates, no surprise. But for "chatgpt", Japanese was a clear second, more than five times Spanish or Portuguese. That's a tiny sample on one day, so don't quote it as a market study. It was enough to make me test Japanese posts properly, and that's where the matcher broke.
Trap 1: \b and \w think Japanese is one long word
My matcher used word boundaries so that "apify" doesn't match "apifying", something like (?<!\w)keyword(?!\w). In Python 3, \w is Unicode-aware, which sounds good, until you remember that hiragana, katakana and kanji are all word characters.
Japanese doesn't put spaces between words. In ChatGPTで使う the character after "ChatGPT" is で, which is \w, so there is no boundary and no match. Every Japanese mention of an English product name was silently dropped. Unit tests with English text passed. Only a real run with languages: ["ja"] showed the misses: the search API returned the posts, my filter threw them away.
The fix: a boundary is any non-word character or a character from a script that is written without spaces (CJK, kana, Hangul, Thai, Lao, Myanmar, Khmer).
_NO_SPACE_SCRIPTS = re.compile(r'[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff'
r'\uac00-\ud7af\u0e00-\u0e7f\u0e80-\u0eff\u1000-\u109f'
r'\u1780-\u17ff\uff66-\uff9f]')
def _boundary(ch: str) -> bool:
return not ch or not re.match(r'\w', ch) or bool(_NO_SPACE_SCRIPTS.match(ch))
Keywords that are themselves written in those scripts are matched as plain substrings, because there's no word boundary to look for.
Trap 2: "ignore accents" can turn か into が
I wanted an option so that eleicao matches eleição and Muller matches Müller. The textbook way is NFD normalization, then strip all combining marks.
The problem: Japanese dakuten (the two little strokes that turn か into が) are combining marks too (U+3099/U+309A). Strip all combining marks and か and が become the same character, which changes the word. So I only strip U+0300–U+036F, the combining diacritics used by Latin, Greek and Cyrillic:
_DIACRITICS = re.compile(r'[\u0300-\u036f]')
def fold(text, ignore_accents=False):
t = unicodedata.normalize('NFKC', text or '').casefold()
if ignore_accents:
t = unicodedata.normalize('NFC', _DIACRITICS.sub('', unicodedata.normalize('NFD', t)))
return t
Trap 3: lower() is not enough
NFKC + casefold() instead of lower() handles full-width letters (ＣｈａｔＧＰＴ, common in Japanese posts), ligatures, and German ß → ss. Small thing, but full-width brand names were another source of silent misses.
Mastodon is a different problem
On Mastodon, full-text search generally needs an account, so the tool reads public hashtag timelines on a list of servers. In my test 9 of 10 big servers answered normally; one (chaos.social) returned HTTP 422 for the tag timeline, so the tool skips servers like that and lists them in the run summary instead of failing.
What I ended up with
The monitor runs on a schedule and only returns posts it hasn't delivered before, plus an Excel/CSV file and a plain-text digest you can send by email or Slack. It only reads public posts and needs no login. If you want to try it: Bluesky & Mastodon Keyword Monitor on Apify. It's pay-per-use ($0.003 per run + $0.001 per matching post).
If you match keywords in user-generated text, test with Japanese (and Thai) early. English-only test data hides exactly this kind of bug.
Top comments (0)