DEV Community: Sami

Your Chinese training data has a provenance problem — and August 2026 makes it urgent

Sami — Thu, 11 Jun 2026 12:41:16 +0000

If you train or fine-tune models on Chinese-language web text, there's a date you should have circled: August 2, 2026. That's when the EU AI Act's obligations for general-purpose AI (GPAI) models start applying in earnest — including the requirement to publish a sufficiently detailed summary of training data and to put in place a policy to respect TDM (text-and-data-mining) opt-outs under the EU Copyright Directive.

In practice, that means someone on your team will eventually be asked: "For this corpus — where did each document come from, when was it retrieved, and did the source signal an opt-out at the time?"

If your Chinese corpus is a pile of JSONL files scraped (or bought) at some point in 2023–2025 with no per-document metadata, the honest answer is: we don't know. And "we don't know" is becoming an expensive answer — for EU-facing labs directly, and for everyone else indirectly, because data vendors, enterprise customers, and academic review boards are all starting to ask the same questions.

Why Chinese-language corpora are the hardest case

Every web corpus has documentation gaps. Chinese-language corpora have them worse, for structural reasons:

1. Scarcity drives sloppiness. High-quality open Chinese text is scarce relative to English. Common Crawl's Chinese share is small and skews toward SEO spam and mirror farms. Because supply is tight, teams hoard whatever they can get — old dumps, resold datasets, "a folder someone left behind" — and documentation is the first thing sacrificed.

2. Quality variance is extreme. The interesting Chinese text lives on platforms — social discussion, video comments, finance commentary, long-form reviews, lifestyle posts. Mixed in with it: boilerplate, ads, bot chatter, template spam. Without per-document quality scoring you either keep the noise or hand-filter at a cost that kills the project.

3. Near-duplicates are endemic. Chinese platforms are reposting cultures. The same viral post appears dozens of times with minor edits — added emoji, swapped hashtags, platform watermarks. Exact-hash dedup misses almost all of it. Train on it anyway and you get memorization hot spots and inflated dataset counts.

4. PII density is high. User-generated Chinese text is full of phone numbers, national ID numbers, WeChat/QQ handles, addresses, and real names — often embedded mid-sentence. GDPR doesn't care that the data subject is in Shanghai if you're processing in the EU.

5. Source documentation is usually zero. Most Chinese web datasets in circulation — including academic ones — ship as bare text. No URLs, no timestamps, no record of what the source's robots/opt-out posture was at retrieval time. You cannot retrofit provenance. If it wasn't captured at collection time, it's gone.

What per-document provenance actually requires

"Provenance" gets used loosely. For training-data documentation purposes, here's the concrete per-document record you want:

Source URL — the canonical URL of the original document, not just "Weibo".
Retrieval timestamp — when this exact text was collected. Opt-out states change; the timestamp anchors your good-faith record.
Robots / opt-out state at retrieval — what the source's machine-readable signals said at the moment of collection. This is the field everyone is missing and the one TDM-policy questions hinge on.
License hint — the best available signal about the source's terms (platform ToS class, page-level license markers). A hint, not a clearance — but a documented hint beats silence.
Content hash — a stable hash of the normalized text, so you can prove what was in the corpus, detect drift between corpus versions, and answer takedown requests precisely.
Pipeline version — which version of the collection/cleaning pipeline produced the record, so your documentation is reproducible.

A practical checklist (works with any tooling)

Stop ingesting undocumented data now. Every undocumented document you add today is a liability you can't repair later.
Capture provenance at collection time — URL, timestamp, robots/opt-out state, license signal, hash, pipeline version, on every document.
Dedup with MinHash/SimHash, not exact hashes. Near-duplicate detection is the only thing that works on repost-heavy Chinese platforms.
Score quality per document and record the score and threshold, so your filtering is defensible, not vibes.
Scrub PII before storage, and log that scrubbing happened (pipeline version again).
Keep a manifest per corpus version — document counts, source distribution, date ranges — so the "sufficiently detailed summary" is a query, not an archaeology project.
Re-check opt-out signals on refresh. Provenance is a snapshot; periodic refresh keeps your record current.

You can build all this in-house. Budget a few engineer-months for the collection layer, the dedup index, the PII pass, and the metadata plumbing — per platform.

Or use the turnkey version

I built the Chinese AI Training Corpus Engine to do exactly the pipeline above, as a self-serve Apify actor:

Five platforms: Weibo, Bilibili, Xueqiu, Douban, RedNote (Xiaohongshu) — social, video, finance, reviews, lifestyle, so your corpus has register diversity, not just one genre.
MinHash near-duplicate detection across the batch.
Per-document quality scoring with configurable thresholds.
PII scrubbing (phones, national IDs, emails) before output.
Full provenance on every document: source URL, retrieval timestamp, robots state at collection, license hint, content hash, pipeline version — the exact fields your EU AI Act documentation needs.

Pricing is per validated document: $0.025/doc (HTTP tier) or $0.055/doc (browser tier). Documents that fail quality checks or turn out to be duplicates are never charged — you pay for corpus, not for noise. A 10,000-document pilot is $250; you'll know within an hour whether the output fits your pipeline.

Honest limitations

This is documentation tooling, not legal clearance. The engine records what sources signaled at collection time and structures it so you can document your corpus — it does not grant training rights, license the underlying content, or determine that any given use is lawful in your jurisdiction. License hints are hints. Whether and how you may train on any document remains your decision, ideally made with counsel who knows the EU AI Act, the Copyright Directive, and your specific exposure. What the tool guarantees is that when counsel asks "what do we know about this corpus?", you have a real answer per document instead of a shrug.

August 2026 is closer than it looks. The teams that win the documentation question will be the ones who captured the metadata while collecting — not the ones reconstructing it afterward.

Questions or pilot requests: samimassis2002@gmail.com — or just run the actor directly.

How to track Weibo hot-search velocity with Python in 2026 — the trending-delta problem and how to handle it

Sami — Tue, 09 Jun 2026 15:52:00 +0000

If you scrape Weibo's hot-search board you get a snapshot: ~50 trending topics, ranked, right now. That's table stakes — and on its own it's almost useless as a signal. The value isn't what is trending; it's what's moving: which topic just jumped 30 places in 20 minutes, which is decaying, which is brand-new this hour. That's velocity, and velocity is where the signal lives — for brand-crisis teams, consumer-trend desks, and anyone modelling attention in China.

The catch: a single scrape can't tell you velocity. You have to diff the board against its own past, reliably, run after run. That's a stateful pipeline, and it has a few non-obvious gotchas. Here's the shape of the problem and how to handle it.

Why a snapshot isn't enough

Rank-right-now tells you nothing about trajectory. "#7" could be a topic on its way to #1 or one fading out of the top 50 — same row, opposite meaning. To act on a trend you need the derivative: direction, speed, and how long it's been climbing. None of that is in a single pull.

The trending-delta problem

Three things make "just diff the board" harder than it looks:

Key by identity, not position. You can't track a topic by its rank — rank is the thing that changes. Key by the topic itself (its text/keyword) or your deltas are nonsense.
State has to survive between runs. A scheduled scrape is stateless by default — each run starts cold. To compute "this rose 12 places since 30 minutes ago," you must persist the previous board and reload it next run, keyed so independent schedules don't overwrite each other.
The board churns. Topics appear, peak, and fall off. You want each tagged new / rising / falling / steady / dropped, plus how long it's been on the board and its running peak — none of which exist in the raw snapshot.

How to handle it (the pattern)

current  = pull_board()                  # [{topic, rank, heat}, ...]
previous = load_state(key)               # durable store that persists across runs

for t in current:
    prev = previous.get(t.topic)         # match on identity, not rank
    t.rank_delta = (prev.rank - t.rank) if prev else None
    t.heat_delta = (t.heat - prev.heat)  if prev else None
    t.status     = classify(prev, t)     # new / rising / falling / steady
    t.first_seen = prev.first_seen if prev else now()
    t.peak_rank  = min(prev.peak_rank, t.rank) if prev else t.rank

emit(current + dropped(previous, current))   # include topics that fell off
save_state(key, current)                      # for next run

Schedule that (hourly or daily) and every run becomes a velocity reading instead of a snapshot. The hard parts in practice are the durable, per-stream state and stable identity matching — get those wrong and the deltas lie.

How this turns into money

Velocity is a leading indicator, and leading indicators are what people pay for:

Brand-crisis alerting — catch a topic about your brand spiking before it peaks: hours of lead time vs. a once-a-day report. That lead time is the product.
Consumer-trend alt-data — rising-topic velocity is an early read on attention and demand shifts. Trend desks and funds buy exactly this kind of signal; a clean, timestamped delta feed is a sellable input.
Marketing / launch timing — ride a topic while it's ascending, not after it's saturated and CPMs have spiked.

If you're building a product on top, this delta stream is your signal layer — everything downstream (alerts, scoring, dashboards) hangs off it.

The practical path (skip the plumbing)

You can build the stateful diff and session handling yourself, or point a maintained extractor at it. I maintain a Weibo Scraper on Apify with a hot_search_delta mode that does exactly this — pulls the board, persists state across scheduled runs, and returns the new / rising / falling / dropped deltas with rank velocity, time-on-board, and peaks. Pay-per-result, runs on a schedule.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "hot_search_delta",
    "deltaStateKey": "hourly",      # name independent streams (hourly / daily / ...)
})

for topic in client.dataset(run["defaultDatasetId"]).iterate_items():
    if topic["status"] == "rising" and (topic["rankDelta"] or 0) >= 10:
        print(f'{topic["title"]}  +{topic["rankDelta"]} ranks  (heat {topic["hotValue"]})')

Wire it to an Apify Schedule and you have a rolling Weibo trend-velocity feed without owning the pipeline.

What it is — and isn't

Is: a stateful, scheduled velocity feed over China's largest real-time attention signal.
Isn't: a one-off snapshot (that's the standard hot-search mode) — or a sentiment model. You get structured movement; the modelling on top is yours.

Need a field that isn't there yet, or a different cadence? Open an issue on the Actor page — I usually ship small additions within a couple of days. For high-volume or managed feeds, the README has the enterprise contact.

Sourcing clean, multi-platform Chinese-language training data at scale in 2026 — a legal + practical guide for AI teams

Sami — Wed, 03 Jun 2026 00:11:34 +0000

If you're training or fine-tuning a model that needs to understand modern Chinese — consumer slang, product opinions, finance chatter, Gen-Z internet register — you've probably hit the same wall: the open Chinese corpora are stale, web-heavy, and thin on authentic first-person signal. Common Crawl's Chinese slice is noisy and dated; the polished open datasets skew formal/encyclopedic. The living Chinese-language signal — how real people actually write in 2026 — sits on a handful of social platforms, and getting it cleanly, at scale, and on solid legal footing is its own project.

This is a practical guide to doing that without standing up (and babysitting) a five-platform scraping operation.

Why Chinese is the hardest major language to source well

Three things make it uniquely painful:

The register you want is platform-locked. Formal Chinese is everywhere; colloquial, current, opinion-rich Chinese lives inside Weibo, RedNote, Bilibili, Douban and Xueqiu — and each gates and structures its public data differently.
It's fragmented. A model that only sees microblog text misses lifestyle reviews, video-comment register, long-form opinion, and finance vernacular. You need several platforms to cover the distribution.
It moves. Last year's dump is already drifting from how people write today. Good Chinese data is a rolling requirement, not a one-time pull.

What a good Chinese-language corpus actually needs

Scale — hundreds of thousands to millions of records, not a sample.
Recency — a scheduled, rolling pull, not a one-off snapshot.
Register diversity — microblog (Weibo), lifestyle/product reviews (RedNote), video comments + danmaku (Bilibili), long-form reviews/discussion (Douban), retail-finance vernacular (Xueqiu).
Clean structure — normalized fields, consistent encoding, deduplicated across platforms (the same KOL post reposted three places should collapse to one record, or you bias the model).
Provenance you can defend — public surface, no authentication, clear about what it is.

The build-it-yourself trap

You can wire up five scrapers. The honest cost is what comes after:

Five different access surfaces that change on their own schedule, each breaking independently — that's five maintenance burdens, not one.
A normalization + cross-platform dedup layer you now own forever.
A legal/compliance posture you have to reason about per platform.

By the time it's robust, you've built a data-engineering team's worth of plumbing before training a single epoch. For most AI teams, that's not the project they want to be in.

The legal layer (high-level — not legal advice)

This is the part people skip and regret. The landscape in 2026, briefly:

Public, logged-off data sits on firmer ground. In Meta v. Bright Data (N.D. Cal., Jan 2024) a US court held that scraping publicly available, logged-off data — and selling it — did not breach Meta's terms. It's narrow to that case's facts, but the direction is clear: authenticated scraping is the risky lane; public, no-login collection is the defensible one.
Personal data has cross-border obligations. If your corpus carries personal information, China's cross-border data-transfer rules (tightened for 2026) attach compliance steps above volume thresholds. The pragmatic read: favor public-post text and aggregate/derived signal over bulk personal profiles.
Marketplaces increasingly demand clean provenance. AI-data marketplaces now ask for "legally sourced, non-scraped" guarantees — which is exactly why sourcing your own public-surface corpus (where you control and document the use) is often cleaner than buying a mystery dataset.

(None of this is legal advice — run your specific use case past counsel. The point is simply: stay on the public, logged-off, non-PII-heavy lane and document it.)

The practical path: maintained public-surface extractors

Instead of owning the five-platform treadmill, you point a maintained, public-surface, no-login extractor at each platform and get back clean, structured records — on a schedule, at scale, pay-per-result. I maintain exactly this set on Apify:

Weibo Scraper — microblog posts, hot search, comments (broad public-opinion register)
RedNote / Xiaohongshu Scraper — first-person product reviews + lifestyle text
Bilibili Scraper — video metadata, comments, danmaku (Gen-Z register)
Xueqiu Scraper — retail-investor / cashtag finance vernacular
Douban Scraper — long-form reviews and discussion

Each returns clean JSON you can stream straight into your pipeline:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "新能源车",   # "new energy vehicles"
    "maxResults": 5000,
})

for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    text = post.get("text") or post.get("content")
    # → straight into your tokenizer / dedup / corpus store

If you want all five platforms normalized into one schema and deduplicated across platforms (so cross-posts don't inflate your corpus), the Chinese Brand Monitor aggregator does that merge in a single call.

Cost at scale

Pay-per-result, cents per record — so a corpus pull is a line item, not a procurement cycle:

Pull	Order of magnitude	Cost
50K Weibo posts, one-off	small fine-tune slice	~$250
500K records across 3 platforms	a real corpus	low four figures
Scheduled monthly refresh	rolling recency	repeats at the same per-record rate

Compare that to an engineer-month building and maintaining five pipelines.

What this is — and isn't

Is: public-surface text, structured, scheduled, at scale — you run it, you own how you use the output.
Isn't: authenticated/private content, or a "mystery" dataset of unknown provenance.
Isn't: a labeling service — you get raw, structured text + metadata; the curation/filtering is yours.

Getting a bulk corpus

For a one-off corpus or a rolling scheduled feed, the actors above run self-serve on Apify's free tier so you can eyeball the output shape before committing. For high-volume / enterprise — millions of records, a custom schema matched to your warehouse, or a managed recurring feed — open an issue titled "Enterprise inquiry" on any actor, or email samimassis2002@gmail.com.

If a platform or field you need for your corpus isn't covered yet, say so — I usually turn additions around in a couple of days.

How to monitor a brand across 5 Chinese social platforms with Python in 2026 — the cross-platform dedup problem and how to handle it

Sami — Sun, 31 May 2026 16:31:02 +0000

You want to know how a brand is being talked about in China. The catch: the conversation isn't on one platform. It's split across Weibo (microblog), RedNote / Xiaohongshu (product & lifestyle), Bilibili (video), Douban (long-form reviews) and Xueqiu (retail-investor chatter). So you wire up five scrapers — and that's where the real work starts.

The part nobody warns you about

Pulling each platform is the easy 20%. The other 80% is turning five raw feeds into one trustworthy dataset:

Five completely different shapes. A "post" on Weibo, a "note" on RedNote, a "video" on Bilibili, a "review" on Douban, a "cashtag comment" on Xueqiu — different fields, different engagement metrics, different date formats. Normalizing them into one table is a chore you redo every time a platform tweaks its response.
Duplicates everywhere. A KOL announces a collab and it's reposted across three platforms; creators cross-post the same clip. Count naively and your "mention volume" is inflated 2–3×, which quietly ruins every trend line and alert you build on top of it.
Five moving targets. Each platform changes how it serves public data on its own schedule. Keeping five pipelines alive is five maintenance burdens, not one — and they break on their calendar, not yours.
Cross-platform consistency. Sentiment and author-reach have to mean the same thing on every platform, or your dashboard lies to you.

By the time you've built normalization + cross-platform dedup + sentiment + reach scoring for five platforms — and signed up to maintain it forever — you've built a data-engineering project before you've answered a single business question.

The shortcut: one call that returns the merged feed

I maintain Chinese Brand Monitor on Apify. You give it a brand keyword; it returns brand mentions across all five platforms already normalized into one schema, deduplicated to one canonical record per real mention, sentiment-tagged, and reach-scored — so the messy 80% is just… done. Pay-as-you-go at $0.045 per canonical mention: no subscription, no seat fee, no annual contract.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("zhorex/chinese-brand-monitor").call(run_input={
    "brandKeyword": "完美日记",   # Chinese or English
    "platforms": ["weibo", "rednote", "bilibili", "douban", "xueqiu"],
    "lookbackDays": 7,
    "sentimentAnalysis": True,
    "deduplication": True,
})

for m in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(m["platform"], m["sentiment"]["polarity"], "—", m["contentSnippet"])

Clean rows — platform, author, follower count, engagement, sentiment, URL — straight into pandas / BigQuery / Snowflake / whatever you already run. No five-pipeline zoo to babysit.

What you can build on top of it (i.e. how this makes you money)

This is the point. Cheap, clean, cross-platform China data is a raw material — and there's real margin in turning it into a product:

Run a China social-listening service. Agencies bill brands monthly for "monitor my brand + 3 competitors in China." Your data cost is cents per mention; you sell the insight, the dashboard, and the recurring retainer. The data layer that used to require a $36K–$50K+/yr enterprise tool (Synthesio, Brandwatch, Meltwater) is now a line item — the spread is yours.
Sell an alt-data sentiment feed. Funds pay for consumer/retail sentiment on Chinese names ahead of the tape. Pull daily across a basket, build a 7-day sentiment + mention-volume delta per brand/ticker, and sell the series. Costs cents per name per day; replaces a five-figure alt-data subscription.
Productize competitor sweeps. One-off "how is brand X perceived vs Y in China, across 5 platforms" reports are high-margin consulting deliverables built on a few dollars of data.
Supply AI / LLM teams labeled, multi-platform, sentiment-tagged Chinese-language text for training corpora and current-events grounding.

In every one of these, the data is the cheap input and the insight is what you charge for — gross margin on the data side sits near the 96% the Actor itself runs at.

Honest comparison (where the big tools still win)

	Enterprise (Synthesio / Brandwatch / Meltwater)	Chinese Brand Monitor
Managed dashboard + alerting	✅ Built in	❌ You bring your own BI
Global TV / podcast / news	✅ Yes	❌ Chinese social only
Account manager / SLA	✅ Yes	❌ Self-serve (issues answered, no SLA)
Price	$36K–$50K+/yr, annual contract	$0.045/mention, pay-as-you-go
Raw data ownership	Walled-garden export	✅ Your dataset, full export
China platform depth	Often shallow / add-on	✅ Five platforms, native
Time to first data	Sales cycle + onboarding	Minutes

If you want a turnkey managed platform with global coverage and a team behind it, buy the enterprise tool. If you want the Chinese social data — cheaply, in your own pipeline, with no contract — this is the layer to build on.

Realistic cost

Workflow	Volume	Monthly cost
One brand, daily, 7-day lookback	~3K mentions	~$135
5-brand agency, daily, sentiment + dedup	~15K mentions	~$675
20-ticker fund, daily (Xueqiu + Weibo + RedNote)	~22K mentions	~$990
One-off competitor sweep	2,500 mentions	~$112

Each is a fraction of a single enterprise seat — and against what you can bill clients on top, the data cost rounds to noise.

What it's NOT

Not a managed dashboard. It's the data layer; you bring the visualization (that's also where your margin is).
Not global coverage. Chinese social platforms only — by design.
Not real-time streaming. Cron-based polling; great for daily/hourly monitoring.
Not authenticated/private content. Public surface only.

If you only need one platform

The aggregator is for cross-platform monitoring. If you only need depth on a single platform, the standalone Actors go deeper:

Weibo Scraper — microblog, hot search, KOL posts
RedNote / Xiaohongshu Scraper — lifestyle / product sentiment
Bilibili Scraper — video + creator analytics
Xueqiu Scraper — retail-investor / cashtag sentiment
Douban Scraper — long-form reviews

Try it

Apify's free tier covers a first run, so you can see the output shape before committing a cent. Start here: zhorex/chinese-brand-monitor. If a field or platform you need isn't there, open an issue on the Actor page — I usually turn fixes around in a couple of days.

How China-focused funds turn Weibo into alt-data (Python, 2026)

Sami — Fri, 29 May 2026 22:17:26 +0000

If you run a China book — equities, FX, commodities, or just a macro tilt — you already know the problem: the official numbers are slow and the English-language coverage is downstream of what already moved on Chinese social platforms. By the time a theme reaches Bloomberg, retail Weibo has been talking about it for days.

Weibo (微博) is where Chinese consumer and retail-investor sentiment shows up first. 580M+ monthly actives, a public hot-search board that turns over hourly, and cashtag-style chatter on every listed name. The catch: there's no official API for international developers, and the data is in Chinese.

This post walks through how to pull Weibo into a usable alt-data feed with a few lines of Python — hot-search trend tracking, keyword/cashtag sentiment, and KOL post monitoring — using an Apify Actor I maintain, so you don't have to babysit visitor cookies or rate limits.

The three signals worth pulling

1. Hot search board (the leading indicator). Weibo's trending board is the single fastest read on what 1.4B people are paying attention to. A brand, a policy rumor, a product recall, a CEO quote — it surfaces here first. For a fund, the delta matters more than the snapshot: what entered the board in the last hour, and how fast it's climbing.

2. Keyword / cashtag sentiment. Search a ticker's Chinese name, a brand, or a product line and you get the raw retail read — positive, negative, the volume of chatter, and which posts have reach. This is the consumer-demand nowcast that quarterly filings give you 90 days late.

3. KOL post monitoring. A single finance or consumer KOL with 5M followers moves retail flows in hours. Tracking specific accounts' posts (and their engagement velocity) is a cleaner signal than aggregate noise.

Pull the hot-search board

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "hot_search",
    "maxResults": 100,
})

for topic in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(topic["rank"], topic["title"], topic.get("heat"))

Run this on a cron every 30-60 minutes and diff consecutive snapshots. A topic that jumps 40 ranks in one hour is the alpha — not its absolute position.

Keyword sentiment as a consumer nowcast

Say you're long a Chinese EV name and want the retail read before the delivery numbers print:

from apify_client import ApifyClient
import pandas as pd

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "比亚迪",          # BYD in Chinese — Chinese keywords yield far better recall
    "maxResults": 300,
})

df = pd.DataFrame(client.dataset(run["defaultDatasetId"]).iterate_items())

# Reach-weight the chatter: a 2M-follower account counts more than a burner.
df["reach"] = df["repostsCount"].fillna(0) + df["commentsCount"].fillna(0) + df["likesCount"].fillna(0)
print(df.sort_values("reach", ascending=False)[["text", "reach", "createdAt"]].head(10))

Pipe the text field through whatever Chinese sentiment model you already run (or a multilingual LLM) and you have a daily polarity series per name. Track the 7-day delta in mention volume + polarity and you've built a sentiment-velocity factor for the cost of a few cents per run.

Build a daily China alt-data job

The two actors that matter together: Weibo for broad consumer + retail sentiment, and the Xueqiu Scraper for finance-specific cashtag chatter (Xueqiu is China's retail-investor forum — closer to a StockTwits read). Run both on the same cron, join on ticker, and you get consumer sentiment and investor sentiment side by side.

tickers = {"BYD": "比亚迪", "Pop Mart": "泡泡玛特", "Luckin": "瑞幸咖啡"}

rows = []
for name, zh in tickers.items():
    run = client.actor("zhorex/weibo-scraper").call(run_input={
        "mode": "search", "searchQuery": zh, "maxResults": 200,
    })
    items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
    rows.append({"name": name, "mentions": len(items)})

print(pd.DataFrame(rows).sort_values("mentions", ascending=False))

Diff today's mention counts against a trailing 7-day mean and you have a chatter-velocity screen across your whole China book.

Pricing

The Weibo Scraper is pay-per-event — you pay per item returned, no subscription, no seat fee. A 300-post sentiment pull is a few cents. A daily 20-ticker monitoring job across the month lands in the low tens of dollars. Compare that to a Bloomberg China module or a packaged alt-data feed and the math is not close.

Job	Volume	Rough monthly cost
Hourly hot-search tracker	~70K topics/mo	low tens of $
20-ticker daily sentiment	~120K posts/mo	tens of $
One-off theme research	a few K posts	a few $

See the Actor's Pricing tab for the exact per-result rate.

What this is NOT

Honest scoping, because sophisticated buyers care more about this than the pitch:

Not real-time tick data. Cron-based polling; 30-60 min cadence is realistic and plenty for sentiment.
Not a sentiment model. It returns the raw posts + engagement + metadata. You bring (or plug in) the NLP.
Not authenticated content. Public surface only — hot search, public search results, public profiles. Some modes (user timelines) work better with your own session cookie, which is optional.
Not financial advice or a signal in a box. It's a data feed. The factor construction is yours.

The broader China stack

If Weibo is the consumer + retail-sentiment layer, the rest of the stack fills in the gaps:

Xueqiu Scraper — retail-investor forum, cashtag-tagged, the finance-specific sentiment read
RedNote / Xiaohongshu Scraper — consumer-brand and product sentiment, the highest-trust purchase-decision channel in China
Bilibili Scraper — Gen-Z video sentiment and creator analytics
Chinese Brand Monitor — if you'd rather not wire up four scrapers, this aggregates Weibo + RedNote + Bilibili + Douban + Xueqiu into one normalized, deduplicated, sentiment-tagged feed at a per-mention price

Try it

A small Weibo pull costs cents, and Apify's free tier covers a first run. Start here: zhorex/weibo-scraper. If you build a China sentiment factor on top of it, I'd genuinely like to hear how — drop a comment or open an issue on the Actor page.

Synthesio charges $36K+/year for Chinese platform coverage. I built one for $0.045/mention.

Sami — Wed, 20 May 2026 01:46:55 +0000

Synthesio sells Chinese platform coverage for $36K+/year. Brandwatch and Meltwater sit in roughly the same $24K-80K/year band. I built an Apify Actor that does the equivalent core job — Weibo, RedNote, Bilibili, Douban, Xueqiu — for $0.045 per deduplicated mention, billed pay-as-you-go.

If you've ever tried to DIY this, you know the math. Five Chinese platforms means five different parsers, five different rate-limit dances, five different schema-drift surprises every couple of weeks, and zero deduplication when a KOL reposts the same content across all of them. By the time you've normalized author identity, follower counts, and timestamps into a usable cross-platform record, you've built a small distributed system that breaks every other Tuesday.

The pitch for zhorex/chinese-brand-monitor is simple: one API call, one normalized schema, one PPE event per canonical mention. You pass a brand keyword (Chinese or English), get back deduplicated records with sentiment scores and reach signals across all five platforms. You don't write per-platform code. You don't run five cron jobs. You don't pay an enterprise floor.

This post walks through six concrete workflows with runnable Python — brand health, crisis monitoring, KOL discovery, hedge fund alt-data, AI training corpora, and a cross-tool finance signal — so you can decide if this fits your stack.

What it does

The Actor takes a single brand keyword (or a list of keywords) and returns deduplicated, sentiment-scored mentions from five Chinese platforms:

Weibo — China's largest microblog; broad consumer chatter
RedNote / Xiaohongshu (小红书) — lifestyle and product discovery; heavy DTC signal
Bilibili — long-form video community; strong Gen-Z signal
Douban — long-form reviews, especially media and lifestyle
Xueqiu (雪球) — retail investor chatter, cashtag-tracked stock sentiment

The Actor handles:

Single keyword input — Chinese 护肤 or English Estée Lauder both work
Normalized cross-platform schema — same fields on every record, no per-platform parsing in your downstream code
Lexicon-based Chinese sentiment scoring per mention (polarity + score)
Cross-platform deduplication — when the same KOL reposts identical content on Weibo and RedNote, you get one canonical record with crossPlatformReposts listing the other appearances
Author identity normalization with follower count for reach-weighted analysis

Engineering choices worth knowing: a browser-grade HTTP client, polite rate limiting, session warming, and a public-data scope that respects each platform's accessible surface. The point is that you don't have to think about any of that — you call the Actor, you get records.

Six concrete workflows

a) Brand health dashboard (~$135/mo)

Daily 8am cron, single brand, 7-day rolling lookback. Push to Looker, Metabase, or a Notion database. Compare this to a $4K/mo Synthesio seat for the same functional coverage.

from apify_client import ApifyClient
import pandas as pd

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "brandKeyword": "Estée Lauder",
    "platforms": ["weibo", "rednote", "bilibili", "douban", "xueqiu"],
    "lookbackDays": 7,
    "maxMentionsPerPlatform": 600,
    "sentimentAnalysis": True,
    "deduplication": True,
}

run = client.actor("zhorex/chinese-brand-monitor").call(run_input=run_input)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())

df = pd.DataFrame(items)
df["polarity"] = df["sentiment"].apply(lambda s: s["polarity"])
summary = (
    df.groupby(["platform", "polarity"])
      .agg(mentions=("mentionId", "count"),
           reach=("authorFollowerCount", "sum"))
      .reset_index()
)
print(summary)

The grouped DataFrame is what you push to your BI tool. ~3,000 deduplicated mentions/month at this cadence lands around $135 in PPE charges.

b) Crisis monitoring (~$270/mo)

Hourly cron, 1-day lookback, filter for negative polarity from accounts above 10K followers. Slack webhook fires on match. This is the workflow that justifies the spend during a product recall, a CEO quote going viral, or a competitor smear campaign.

import requests
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
SLACK_WEBHOOK = "https://hooks.slack.com/services/XXX/YYY/ZZZ"

run = client.actor("zhorex/chinese-brand-monitor").call(run_input={
    "brandKeyword": "Estée Lauder",
    "platforms": ["weibo", "rednote", "bilibili", "douban", "xueqiu"],
    "lookbackDays": 1,
    "maxMentionsPerPlatform": 200,
    "sentimentAnalysis": True,
})

for m in client.dataset(run["defaultDatasetId"]).iterate_items():
    if m["sentiment"]["polarity"] == "negative" and m["authorFollowerCount"] >= 10000:
        requests.post(SLACK_WEBHOOK, json={
            "text": (
                f"[{m['platform']}] {m['authorName']} "
                f"({m['authorFollowerCount']:,} followers) — "
                f"sentiment {m['sentiment']['score']:.2f}\n"
                f"{m['contentSnippet']}\n{m['url']}"
            )
        })

Hourly × 24 × 30 ≈ ~6,000 deduplicated mentions/month if the brand has steady chatter — roughly $270/mo. Cheap insurance for a comms team.

c) KOL identification (~$90/mo)

Weekly category-keyword run. Skincare = 护肤, sneakers = 球鞋, supplements = 保健品. Filter verified authors above 50K followers, sort by engagement.

from apify_client import ApifyClient
import pandas as pd

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("zhorex/chinese-brand-monitor").call(run_input={
    "brandKeyword": "护肤",
    "platforms": ["weibo", "rednote", "bilibili", "douban"],
    "lookbackDays": 14,
    "maxMentionsPerPlatform": 500,
})

df = pd.DataFrame(list(client.dataset(run["defaultDatasetId"]).iterate_items()))
df["engagement"] = df["engagementMetrics"].apply(
    lambda e: e.get("likes", 0) + e.get("comments", 0) * 3 + e.get("shares", 0) * 5
)

candidates = (
    df[(df["authorVerified"]) & (df["authorFollowerCount"] >= 50000)]
      .sort_values("engagement", ascending=False)
      .drop_duplicates("authorId")
      .head(50)
)
candidates[
    ["authorName", "platform", "authorFollowerCount", "engagement", "url"]
].to_csv("kol_candidates.csv", index=False)

Weekly cadence on 1-2 category keywords ≈ ~2,000 mentions/month — roughly $90/mo. The output is a ranked candidate list your social team can outreach directly.

d) Hedge fund alt-data (~$990/mo)

Daily run across 20 portfolio tickers on Xueqiu + Weibo + RedNote. Build a sentiment-velocity feature: 7-day mention-count delta paired with polarity shift. Join two consecutive runs to compute the velocity.

from apify_client import ApifyClient
import pandas as pd

client = ApifyClient("YOUR_APIFY_TOKEN")

tickers = [
    "BABA", "PDD", "JD", "BIDU", "NIO", "XPEV", "LI", "MEITUAN",
    "TENCENT", "BYD", "LKNCY", "TME", "BILI", "VIPS", "TAL", "YMM",
    "DIDI", "ZH", "NTES", "FUTU",
]

def pull(lookback_days):
    rows = []
    for ticker in tickers:
        run = client.actor("zhorex/chinese-brand-monitor").call(run_input={
            "brandKeyword": ticker,
            "platforms": ["xueqiu", "weibo", "rednote"],
            "lookbackDays": lookback_days,
            "sentimentAnalysis": True,
        })
        for item in client.dataset(run["defaultDatasetId"]).iterate_items():
            rows.append(item)
    return pd.DataFrame(rows)

today = pull(1)
week = pull(7)

def agg(df):
    df = df.copy()
    df["score"] = df["sentiment"].apply(lambda s: s["score"])
    return df.groupby("brandKeyword").agg(
        count=("mentionId", "count"),
        avg_polarity=("score", "mean"),
    )

today_agg = agg(today)
week_agg = agg(week)
features = today_agg.join(week_agg, lsuffix="_1d", rsuffix="_7d")
features["velocity"] = features["count_1d"] - (features["count_7d"] / 7)
features["polarity_shift"] = features["avg_polarity_1d"] - features["avg_polarity_7d"]
print(features.sort_values("velocity", ascending=False))

20 tickers × daily × 3 platforms ≈ ~22K mentions/month — roughly $990/mo. Compare to a single Bloomberg terminal at ~$28K/year for one analyst.

e) AI training corpus (~$2,250 one-shot)

50 brand keywords × 1,000 mentions each = 50K Chinese-language labeled records for SFT or RLHF corpora. Every record has an explicit sentiment polarity, author follower bracket, and platform. Compare to $15-50K academic licensing fees for comparable annotated Chinese sentiment corpora.

from apify_client import ApifyClient
import json

client = ApifyClient("YOUR_APIFY_TOKEN")

brands = [
    "华为", "小米", "比亚迪", "蔚来", "理想汽车", "拼多多", "美团",
    "完美日记", "花西子", "钟薛高", "元气森林", "瑞幸咖啡", "海底捞",
    # ... 50 total
]

with open("china_sft_corpus.jsonl", "w", encoding="utf-8") as f:
    for brand in brands:
        run = client.actor("zhorex/chinese-brand-monitor").call(run_input={
            "brandKeyword": brand,
            "platforms": ["weibo", "rednote", "bilibili", "douban", "xueqiu"],
            "lookbackDays": 30,
            "maxMentionsPerPlatform": 200,
            "sentimentAnalysis": True,
        })
        for m in client.dataset(run["defaultDatasetId"]).iterate_items():
            f.write(json.dumps({
                "text": m["content"],
                "label": m["sentiment"]["polarity"],
                "score": m["sentiment"]["score"],
                "platform": m["platform"],
                "brand": m["brandKeyword"],
            }, ensure_ascii=False) + "\n")

50K records × $0.045 = $2,250. One-shot. No annotator contracts, no FTE-month spent labeling.

f) Cross-tool finance signal: Xueqiu sentiment × TradingView price

Pair the Chinese Brand Monitor with the TradingView Scraper for a sentiment-vs-price divergence signal. When Xueqiu retail sentiment turns sharply positive while the price stays flat or drifts down, you have a setup worth a closer look.

from apify_client import ApifyClient
import pandas as pd

client = ApifyClient("YOUR_APIFY_TOKEN")

sent_run = client.actor("zhorex/chinese-brand-monitor").call(run_input={
    "brandKeyword": "BABA",
    "platforms": ["xueqiu"],
    "lookbackDays": 7,
    "sentimentAnalysis": True,
})
sent = pd.DataFrame(list(client.dataset(sent_run["defaultDatasetId"]).iterate_items()))
sent_score_7d = sent["sentiment"].apply(lambda s: s["score"]).mean()

price_run = client.actor("zhorex/tradingview-scraper").call(run_input={
    "mode": "technical_analysis",
    "symbols": ["NYSE:BABA"],
    "includeIndicators": True,
})
price = next(iter(client.dataset(price_run["defaultDatasetId"]).iterate_items()))
perf_week_pct = (price.get("perfWeek") or 0) / 100

# Positive Xueqiu sentiment minus weekly price return: large positive = retail
# is loud-bullish but the tape hasn't caught up yet.
divergence = sent_score_7d - perf_week_pct
print({
    "ticker": "BABA",
    "xueqiu_sentiment_7d": round(sent_score_7d, 3),
    "tradingview_perfWeek_pct": price.get("perfWeek"),
    "divergence": round(divergence, 3),
})

A positive divergence row is "sentiment positive, price not yet moved." That's the setup quants pay alt-data brokers tens of thousands a year to surface.

Normalized output schema

Every record across every platform has this shape:

{
  "mentionId": "rednote_8b3c2f91a4",
  "platform": "rednote",
  "brandKeyword": "Estée Lauder",
  "brandMatchType": "exact",
  "content": "雅诗兰黛小棕瓶用了三个月，肌肤紧致很多...",
  "contentSnippet": "雅诗兰黛小棕瓶用了三个月...",
  "language": "zh-CN",
  "authorId": "rednote_user_4429871",
  "authorName": "小琳护肤日记",
  "authorFollowerCount": 184230,
  "authorVerified": true,
  "publishedAt": "2026-05-18T14:23:11Z",
  "engagementMetrics": {
    "likes": 2104,
    "comments": 187,
    "shares": 56,
    "views": 18430
  },
  "url": "https://www.xiaohongshu.com/explore/...",
  "mediaUrls": ["https://sns-img-...jpg"],
  "sentiment": { "polarity": "positive", "score": 0.78, "method": "lexicon" },
  "crossPlatformReposts": [
    {
      "platform": "weibo",
      "url": "https://weibo.com/...",
      "publishedAt": "2026-05-18T15:02:00Z"
    }
  ],
  "scrapedAt": "2026-05-20T08:00:01Z"
}

Your downstream code stays platform-agnostic. Pandas, BigQuery, Snowflake, ClickHouse — pick your warehouse and the records load directly.

Pricing

$0.045 per canonical mention, billed only after deduplication. If a KOL reposts the same content across Weibo + RedNote + Bilibili, that's one billable mention with the reposts attached, not three.

Use case	Volume	Monthly cost	Enterprise alternative
Single brand, daily, 7-day lookback	~3K/mo	~$135	$4K/mo Synthesio seat
5-brand agency, daily, sentiment + dedup	~15K/mo	~$675	$24K-80K/year
20-ticker hedge fund	~22K/mo	~$990	$28K/year Bloomberg seat
AI training corpus one-shot	50K	~$2,250	$15K-50K academic license

What this Actor does NOT do

Honest scoping matters more than pitch volume:

Not real-time push streaming. Cron-based polling, 5-minute minimum interval. If you need sub-second push, this isn't it.
Not a historical archive. Maximum 30-day lookback. For multi-year backfill, you need a different tool.
Not authentication-walled content. No Zhihu authenticated answers, no private WeChat groups, no closed Weibo Super Topic posts.
Not a CRM or BI tool. This is the data layer. You bring the dashboard.

If those constraints are dealbreakers for your use case, save the credit and don't run it.

The broader China stack

The main Actor here is zhorex/chinese-brand-monitor, but the rest of the stack exists for cases when you need single-platform depth or a different angle:

For deeper single-platform RedNote dives — full creator profiles, comment threads, hashtag networks — reach for the standalone RedNote/Xiaohongshu Scraper.
For Weibo-only bulk pulls — historical hashtag sweeps, single-account timelines, Super Topic posts — the Weibo Scraper is the dedicated tool.
For Bilibili-only deep pulls — video metadata, danmaku, UP主 channel coverage — use the Bilibili Scraper.
For finance-only sentiment with cashtag granularity and reply trees, the Xueqiu Scraper goes deeper than the brand-monitor surface.
For long-form review extraction, especially books, films, and lifestyle, the Douban Scraper handles the review-thread structure.
For the cross-tool finance workflow above, the TradingView Scraper provides the price half of the sentiment-vs-price divergence signal.
If you're tracking brand mentions, you usually also want competitor pricing — the JD Scraper covers the e-commerce price side of the China stack.

Try it

$5 of Apify free credits cover roughly 100 mentions — enough to run a single brand for a week and see whether the output shape fits your downstream code. Start here: zhorex/chinese-brand-monitor.

If you build something on top of this — a Looker dashboard, a Slack bot, a Streamlit explorer, a sentiment ETF screen — drop a comment, or open an Issue on the Actor page. Schema customization, missing platforms, follower-bracket additions, new sentiment lexicons — those are the kinds of changes that get prioritized when users ask for them.

Pinnacle odds for $0.01 a snapshot: the OddsJam / Odds API replacement sharp bettors are using in 2026

Sami — Tue, 19 May 2026 13:54:32 +0000

If you bet sharp lines, the only book that genuinely matters for fair-value is Pinnacle. Every EV model, every CLV report, every "did I beat the close?" check eventually compresses down to one question: what was Pinnacle showing on this market at T-1?

For years the standard way to get that feed was The Odds API ($249/mo for 15M credits) or OddsJam Gold ($249/mo, $499+ for Pro). For a tipster shop polling 100 fixtures a day that math is tolerable. For a solo bettor running CLV on 20 fixtures it's overspend. For a specials trader it's worse — OddsJam gates futures and yes/no markets behind their highest tier and The Odds API doesn't surface most of them at all.

There's now an Apify Actor that does the same job pay-per-snapshot:

zhorex/sports-odds-aggregator — Pinnacle h2h + spreads + totals + 5,000+ specials per sport, from $0.01 a snapshot. Datacenter-proxy friendly. No login, no monthly minimum.

This post is the playbook: four recipes that show exactly how to run it, what each costs, and where the savings show up vs. the SaaS-incumbent pricing.

Quick note on naming: the Actor's title still references Bet365 because Bet365 is the second-book slot, but Bet365's public mobile-web path is under repair as of May 2026. Pinnacle is shipping today, and the moment Bet365 returns the cross-book best-price flag (isBestPriceAcrossBooks) and fuzzy event-matching activate automatically — no input change on your side.

Pricing in plain English

Four event types, billed pay-per-event (PPE):

Event	Price	When it fires
`odds-snapshot-pre-match`	$0.01 / snapshot	One market-outcome from a scheduled (not in-play) fixture
`odds-snapshot-live`	$0.02 / snapshot	One market-outcome from a live (in-play) match
`odds-snapshot-player-prop`	$0.04 / snapshot	One special / future / yes-no / team prop / exact-totals row
`scheduled-run`	$0.05 / run	Once per cron tick — often fully offset by the dedup window

A typical pre-match fixture with ["h2h", "spreads", "totals"] emits ~7 snapshots (3 h2h outcomes + 2 spreads + 2 totals). Add "specials" and you get an extra 30–80 rows per fixture — yes/no markets, exact totals, first-team-to-score, winning margin per scoreline, team props.

The bit that turns this from "interesting" to "actually cheap": the deduplicationWindowSeconds setting suppresses snapshots when the line hasn't moved. On stable mid-week Premier League pre-match polls you typically charge for 5–15% of "naïve" volume. A 60-second cron on a stable line is essentially free.

Recipe 1 — Pinnacle closing-line value (CLV) tracker

The recipe that pays for the Actor in its first weekend.

Setup:

mode: "pre_match_only"
Trigger at T-30 minutes and T-1 minute per fixture
Bet your soft book at T-30, log Pinnacle's T-1 close, compute CLV per ticket

Pinnacle's closing line is the canonical sharp benchmark. If you're consistently beating Pinnacle's close, your edge is real. If you aren't, you can stop pretending — CLV is the ground truth of whether you're a winning bettor or a noise-trader.

Cost for 200 fixtures/week (h2h+spreads+totals, ~7 snapshots × 2 polls each): ~$65/month.

The Odds API doesn't expose "Pinnacle close at T-1" as a first-class field, so you're paying $249/mo for the feed and still rolling your own snapshot scheduler. Here the snapshot scheduler (Apify cron) and the snapshot itself together come in at ~25% of the price.

Recipe 2 — EV-model live edge harvester

The model-on-top use case. If you have a fair-value model and you harvest the moments where book_price × your_fair_value > 1.03, you want a polling firehose during in-play, not an hourly dump.

Setup:

{
  "books": ["pinnacle"],
  "sports": ["basketball", "tennis", "soccer"],
  "marketTypes": ["h2h", "spreads", "totals"],
  "mode": "live_only",
  "deduplicationWindowSeconds": 0
}

Schedule: 60-second cron during target match windows.

Volume: ~50K live snapshots/month × $0.02 + orchestration ≈ ~$1,080 / month.

That looks pricey until you put it next to OddsJam Pro at $499+/mo for a SaaS API you don't control and that throttles by tier. The trade is: you pay more per request, but you pay only for what you actually consume, you set the cadence, and a stable line costs you nothing.

The other thing the SaaS won't sell you: every snapshot includes isLive, matchClock, and matchScore. Your model doesn't have to join against a separate scoreboard feed during a live NBA fourth quarter.

Recipe 3 — Specials sniper (the OddsJam gating trick)

For value bettors and exact-totals modellers. This is the recipe where the pricing gap gets embarrassing.

Pinnacle's withSpecials=true matchups call returns ~5,000 markets per major sport:

First Team To Score (3-way)
Win to Nil 1st Half (yes/no)
Exact Total Goals 1st Half (multi-way)
Winning Margin per scoreline
A long tail of team props and player-related markets

These are the markets soft books are slowest to sharpen up on — which is where the actual edge lives. OddsJam gates futures and props behind their highest tier. The Odds API doesn't surface most of them.

Setup:

{
  "books": ["pinnacle"],
  "sports": ["soccer"],
  "marketTypes": ["specials"],
  "mode": "pre_match_only",
  "deduplicationWindowSeconds": 60
}

Schedule: 4-hour cron during the season.

Volume: ~6K specials snapshots + 180 runs ≈ ~$250 / month for the segment that powers the largest EV pockets in retail sports betting.

A pattern that works: filter the dataset to marketType == "specials" && impliedProbability < 0.10. Pinnacle longshots above 10× implied with sharp money backing are where the soft-book mispricings concentrate.

Recipe 4 — Tipster Discord auto-poster

The cheapest one and the easiest to sell to a small operation.

Setup:

sports: ["soccer"]
leagueFilter: ["UEFA", "EPL", "La Liga"]
mode: "pre_match_only"
Every 6 hours, webhook → Discord

Cost: ~$30/month for a daily top-10 spreads + totals digest piped straight into the channel. If you currently screenshot OddsJam into Discord by hand, this is the upgrade.

The pay-per-event math in one table

Workflow	Volume	Monthly cost	Replaces
Casual bettor — daily 9am pre-match dump, 30 fixtures	~900 snapshots	~$11	$59/mo Odds API tier
CLV tracker — T-30 + T-1, 80 fixtures/wk	~3.2K snapshots	~$65	$249/mo OddsJam Gold
Tipster shop — 100 fixtures × 7 outcomes, hourly	~21K snapshots	~$245	$249/mo OddsJam Gold (parity)
Specials trader — daily soccer sweep	~6K snapshots	~$250	Highest-tier gate (not available below)
EV model — live NBA + tennis + soccer, 90s cron	~50K live snapshots	~$1,200	OddsJam Pro $499+ + you control cadence

Flat-rate SaaS wins only once you cross ~150K snapshots/month of stable workload. Below that — which is most solo sharps, most tipster operations, and every specials trader — PPE is just cheaper, and the cost curve is linear in actual usage rather than tier-jumpy.

The other PPE advantage that quietly compounds: there's no annual contract. Off-season for a sport? Cron stops, billing stops. You don't pay for unused capacity in August when soccer is dead.

Three steps to a running cron

Step 1 — Pick your sports and markets.
Defaults are ["soccer", "tennis"] — the two highest-liquidity sharp markets year-round. For CLV add "spreads", "totals". For specials sniping add "specials". The full sport list is 11 deep (soccer, tennis, basketball, MMA, baseball, hockey, esports, AFL, NFL/college, golf, rugby).

Step 2 — Run once with default input and verify Pinnacle returns data for your sport+league pick. Output lands in your Apify dataset.

Step 3 — Save as Task → Schedules → New Schedule with the cron string you want:

*/5 * * * *    pre-match every 5 minutes
* * * * *      live every minute during target windows
0 9 * * *      daily 9am pre-match dump
0 */6 * * *    every 6 hours
0 10 * * 6     Saturday morning weekly audit

Attach a webhook to the schedule and ship the dataset into your EV pipeline, Discord/Slack bot, Sheets workbook, or wherever your model lives.

Python in 12 lines

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("zhorex/sports-odds-aggregator").call(run_input={
    "books": ["pinnacle"],
    "sports": ["soccer", "tennis"],
    "marketTypes": ["h2h", "spreads", "totals", "specials"],
    "mode": "pre_match_only",
    "maxEventsPerSport": 50,
    "deduplicationWindowSeconds": 30,
})

for snapshot in client.dataset(run["defaultDatasetId"]).iterate_items():
    if snapshot["marketType"] == "specials" and snapshot["impliedProbability"] < 0.10:
        evaluate_for_bet(snapshot)

That's the whole integration. Every snapshot arrives in a flat per-market-outcome shape with priceAmerican, priceFractional, price (decimal), impliedProbability, and isBestPriceAcrossBooks on every row — your model doesn't have to do format gymnastics or join against a separate American-odds conversion table.

What a snapshot looks like

{
  "snapshotId": "a1b2c3d4e5f6789012345678",
  "book": "pinnacle",
  "sport": "soccer",
  "league": "Premier League",
  "homeTeam": "Manchester City",
  "awayTeam": "Liverpool",
  "commenceTime": "2026-05-22T19:00:00Z",
  "isLive": false,
  "marketType": "h2h",
  "outcomeKey": "home",
  "outcomeLabel": "Home",
  "price": 1.91,
  "priceAmerican": -110,
  "priceFractional": "10/11",
  "impliedProbability": 0.52356,
  "isBestPriceAcrossBooks": true,
  "scrapedAt": "2026-05-18T14:32:00Z"
}

snapshotId is a stable sha1 derived from book+event+market+outcome+timestamp, so it makes a clean primary key if you're persisting into Postgres / DuckDB.

For high-volume operations

If your monthly burn is past 50K snapshots and you need a dedicated polling cadence, custom market types (Asian handicap quarter-lines, derivative props, fancy bets), or a schema SLA for a downstream production pipeline, the Actor page has an "Enterprise inquiry" pointer. Webhook integrations, dedicated proxy pools, and custom dataset views ship in roughly a week. Sustained seven-figure-action operations can talk dedicated-instance posture.

For everyone else the default Apify Proxy works on Pinnacle's guest API — Pinnacle's public surface tolerates datacenter IPs by design (which is why it's on the supported-books list to begin with). If your plan includes datacenter, override apifyProxyGroups: ["DATACENTER"] and your proxy cost drops to roughly 5% of a residential-default scraper.

Things worth knowing before you run it

Not US bookmakers. DraftKings / FanDuel / BetMGM / Caesars / ESPN BET are geo-gated behind Akamai and need US residential proxy, which kills the per-snapshot economics. Other Apify Actors target those — this one stays out.
Personal-analysis use only. Pinnacle's TOS forbids commercial redistribution of raw odds. The architecture is per-buyer-execution — you run it in your own Apify account against your own polling cadence. Don't resell the feed.
Not a streaming WebSocket feed. Poll-based, fastest meaningful cadence ~60s.
Bet365 returns when Bet365 returns. Cross-book best-price flag and fuzzy event-matching are already in the codebase; the day a second book ships, arb infra activates without an input change.

Where to start

If you currently pay The Odds API or OddsJam Gold $249/mo for the Pinnacle column, the cheapest experiment is:

Spin up the Actor with the default input.
Run it on five of your usual fixtures.
Compare the snapshots against whatever your incumbent feed gave you for the same fixtures.

The break-even comes faster than you'd expect — most workflows under 150K snapshots/month earn back the SaaS subscription inside the first month, and the dedup window keeps marginal cost near zero on stable lines.

Actor link: apify.com/zhorex/sports-odds-aggregator

If the Actor saves you a month of OddsJam Gold, the single highest-leverage thing you can do back is a 30-second review on the Actor page — it directly funds the next defensive patch when the books shift their schemas.

Roadmap is public: Smarkets adapter (v0.4) reactivates cross-book arb infra, Pinnacle alternate-lines / period markets (v0.5) opens half/quarter handicap decomposition, Betfair Exchange BYO-credentials (v0.6), WebSocket mode (v0.7), automatic arb finder (v0.8).

$0.005 per Weibo post — the Chinese social data layer Western teams keep skipping

Sami — Sun, 17 May 2026 16:54:46 +0000

I shipped a Weibo scraper on Apify eight months ago. Fifteen customers pay me on it now, another thirty-four use the free tier, and in the last sixteen days they pulled 136,400 posts through it. I built it because every Western social-listening tool I evaluated — Synthesio, Brandwatch, Meltwater — quoted four to five figures a year for China coverage that was thinner than what you get from one tuned Apify run.

The whole pitch is one number: $0.005 per post. Pay only for items you actually take. The Apify free plan covers your first ~1,000 mentions before you spend a cent.

Weibo Scraper on Apify Store →

What's actually in the box

Four modes. All return normalized JSON. No Weibo login. No API key from Weibo. No VPN.

hot_search — the live hot-topics list, i.e. what 580M+ monthly active users are looking at right now. The single most-watched signal in Chinese social.
search — keyword search across public posts. Brand names, ticker symbols, product launches, Chinese or English.
post_comments — every public comment on a given post. Sentiment grenades and viral crises live here.
user_posts — full posting history of any public account. KOL vetting, executive watch, competitor monitoring.

Output is flat JSON — post text, author handle, timestamp, repost / comment / like counts, media URLs. Push it straight into a warehouse, a Pandas DataFrame, or a Slack alert with a 30-line script.

What people actually pay for

I see what runs every day on this actor. The patterns paying customers settle into:

1. Equity / sentiment signal on China-listed names — ~$25/day, ~$750/month

A small fund or research desk covering BABA, NIO, PDD, BILI, JD, BEKE, LI, XPEV, KWEB constituents, or any China-exposed Western name. Scheduled search over 30-50 tickers and brand names, ~5,000 posts a day, fed into a sentiment model. Sentiment shifts on Weibo lead the Hong Kong open by hours. Dedicated enterprise social-listening contracts that even attempt China coverage start near $30K/year, and most don't index Weibo deeply.

2. Brand monitoring for Western brands in China — ~$15/day, ~$450/month

A consumer brand with China exposure — Apple, Tesla, Nike, Starbucks, LVMH, Lululemon, any DTC brand on Tmall — needs ~3,000 mentions/day on brand and product-line keywords. Comments mode catches crisis posts before they trend. Synthesio / Brandwatch / Talkwalker contracts that include China typically run $30K-$100K/year. The same daily mention stream costs you less than a streaming subscription.

3. KOL / influencer due diligence — ~$1 per KOL

Before you wire 50,000-200,000 RMB to a Weibo influencer for a sponsorship, run user_posts against the handle. Look at posting cadence, real engagement (not vanity follower counts), brand affinity history, controversy flags. One avoided bad deal pays for years of usage.

4. AI / LLM training data — ~1M posts = $5,000

Real-world, conversational, dialect-rich Mandarin from public posts. Filtered Weibo subsets sell on data marketplaces for $20K-$50K and ship stale by months. Pull fresh data on the topics and time windows you care about, own the pipeline, and the per-post cost is a small fraction of either marketplace data or annotator-collected datasets.

5. M&A and pre-deal diligence on Chinese targets — $200-$500 one-off

A pre-LOI sentiment pull on a Chinese target — employee chatter, customer complaints, founder reputation, glass-door-equivalent venting. Boutique diligence firms bill $25K-$75K for the equivalent exercise. As a banker or consultant, even a "$500 in cost, $30K invoice" framing is a 60x markup the client is happy to pay for.

6. Crisis monitoring / hourly brand watch — ~$50/month

Schedule a six-times-a-day run on brand keywords. hot_search catches a viral crisis the moment it crosses into the public consciousness — typically a 4-12 hour head start on Western media coverage. For a brand worth eight figures, that gap is the difference between "managed" and "case study."

7. Macro / consumer-trend reading from the hot list — ~$5/day

The hot search list is the cheapest macro signal in Chinese markets. Tariff reactions, regulatory rumblings, viral consumer products, celebrity scandals that wreck brand deals — all surface here first. Hedge fund quants, geopolitical analysts, and morning-brief writers all bake this in.

The number that matters: $0.005 per item

You pay per item returned. No subscription, no surprise overage.

1,000 mentions: $5
10,000 mentions: $50
100,000 mentions: $500

The Apify free plan gives you ~$5/month in platform credit, which covers your first ~1,000 mentions on this actor. You validate the data fits your use case before you spend a cent.

Start on the free plan →

Quick start

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

# Pull 1,000 posts mentioning Tesla. $5 flat.
run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "特斯拉",
    "maxResults": 1000,
})

# Stream the results
for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(post["createdAt"], post["repostsCount"], post["text"][:120])

Same shape works for hot_search, post_comments, user_posts. Swap the mode and the input keys to whatever the run takes. The exact input schema lives on the actor page.

Where the real money lives: recurring runs

One-shot pulls are fine for a diligence assignment. The customers who actually extract serious value from this are the ones running it on a schedule. Apify Schedules takes a cron expression and a saved input — the actor runs forever, the dataset accumulates, and you download it as JSON, CSV, or Excel.

The math gets compelling fast. Below is what my heaviest recurring customers actually run:

Use case	Cron expression	Approx. monthly cost	What it replaces
Morning hot-search dump for the daily brief	`0 9 * * 1-5` (Asia/Shanghai)	~$15	A junior analyst's 30-min daily task
Brand mentions, every two hours	`0 /2 * *`	~$450	$30K/yr Brandwatch contract for China only
Equity tickers, hourly (the highest-ROI cron on this list)	`0 * * * *`	~$750	A $120K/yr China sentiment analyst, half-replicated
Crisis watch, every 30 minutes	`/30 * * *`	~$1,500	A 24/7 PR monitoring agency contract
Overnight KOL sweep on 200 handles	`0 2 * * *`	~$60	$5K/mo influencer-vetting subscription

Set the cron once, walk away, get paid in compounded insight. The customers running hourly equity cron jobs have been doing it for months without touching the config — the actor runs, the data lands, the alpha shows up in their dashboards. That's the only mode of use that actually justifies the time you invested learning the schema.

If you take one thing from this post: don't run it manually twice — wire the second run into a cron. The actor was built for that, the pricing was designed for that, and that's where every customer who renewed went.

Reliability and what happens when things break

You pay per item. If the actor returns nothing on a run, you pay nothing. If it returns 327 items, you pay for 327. That alignment is the whole reason I picked per-event pricing instead of a monthly subscription — my incentive to keep the thing working is exactly your incentive that it works.

I monitor the actor daily. When something upstream changes, I ship a fix within hours, not weeks. The Apify Store rating and issue history on the actor page are public.

Why I priced it like this

I shipped this eight months ago. By month two it was profitable. The most recent sixteen-day window: fifteen paying customers, 136,400 items returned, $697 revenue, $675 profit, 96.79% margin. The margin isn't there because the work is trivial — it's there because per-event pricing means I only earn when the data is actually delivered.

If you're evaluating Chinese social tools and the lowest quote you can get is $20K+, run a 1,000-mention probe through this actor first. You'll know inside ten minutes whether the data covers your use case. Worst case, you spend $5. Then wire it into a cron and forget about it.

Try it on the free plan →

Other Chinese-platform actors I run

Weibo is the macro signal layer for China. These cover the rest of the surface area:

Xiaohongshu / RED scraper — lifestyle, beauty, female-skewing audience. The #1 platform for DTC brand launches in China.
Bilibili scraper — long-form video, Gen Z, gaming / anime / tech vertical signal.
Douban scraper — books, films, music, niche communities. The most "honest" review platform in China.
Xueqiu scraper — retail-trader-heavy financial discussion. Equity-desk supplement to Weibo.
Chinese Brand Monitor — composite brand signal across the platforms above.

All same pricing model. Pay per item. Schedule freely.

Compliance posture

Only public Weibo posts. No private accounts, no DMs, no content behind a login wall.
No login bypass. The actor does not log into Weibo on your behalf, and does not need an account to function.
Optional cookies are user-supplied and only raise your personal rate limit. They are never required for the actor to work.

If your use case requires private data, this actor isn't it — and frankly nothing on the Apify Store will be.

If you actually run something interesting with it, leave a comment or open an issue on the actor page — I read all of them.

apify.com/zhorex/weibo-scraper

Track brand mentions across China's top 5 social platforms in one API call — $0.045 per mention

Sami — Sat, 16 May 2026 10:36:57 +0000

If your brand competes for Chinese consumers and you're not actively monitoring conversations on Weibo, RedNote, Bilibili, Douban, and Xueqiu, you're flying blind in the world's second-largest consumer market.

The problem is that the "enterprise" way to do this — Synthesio, Brandwatch, Talkwalker — starts at $50,000 per year for Chinese platform coverage, with annual contracts, locked-in seats, and a sales cycle measured in weeks. So most mid-market teams just… don't. They monitor English-language Twitter for their global brand, see a sentiment dip in APAC revenue a quarter later, and have no leading signal explaining why.

Chinese Brand Monitor launches today on the Apify Store to fix that. One API call, five platforms, normalized output, $0.045 per mention. No subscription. No annual contract. No minimum spend. Run it once, run it daily, run it hourly — you only pay for the mentions you actually pull.

The 5 platforms in one call

Platform	What it captures	Why it matters for brand monitoring
Weibo (微博)	Public microblog mentions, KOL posts, hot search trending	China's Twitter. 580M+ users. Where consumer crises break first and where KOL endorsements reach hundreds of millions in hours.
RedNote / Xiaohongshu (小红书)	Lifestyle and consumer brand notes, first-person product reviews	300M+ users. The single highest-trust channel for Chinese consumer purchase decisions in beauty, skincare, fashion, food, travel.
Bilibili (B站)	Video titles, descriptions, and creator mentions	China's YouTube. 300M+ users. Where Gen Z consumer brand affinity is built and where unboxing / review culture lives.
Douban (豆瓣)	Movie / book / music subject mentions, brand tie-ins, soundtracks, branded titles	200M+ users. Long-form opinion-rich content — the densest source of detailed consumer attitude data outside Zhihu.
Xueqiu (雪球)	Stock cashtag and corporate mentions for listed brands	20M+ users, financial-grade signal. Critical if your brand is publicly listed (NYSE:BABA, NASDAQ:JD, HK:00700, A-share tickers) — finance KOLs move retail sentiment fast.

These five platforms cover the full spectrum: broad public opinion (Weibo), high-trust consumer reviews (RedNote), Gen-Z video sentiment (Bilibili), long-form opinion (Douban), and investor sentiment (Xueqiu). For most consumer brands, monitoring any 3 of these in real time is a leading indicator that beats your CRM dashboards by 2-6 weeks.

Why each mention is worth $0.045

It's tempting to look at $0.045 and compare it to "free" Twitter mentions. That's the wrong comparison. The right comparison is: what would it cost you to get one Chinese consumer mention, normalized, sentiment-tagged, and cross-platform-deduplicated, any other way?

Synthesio / Brandwatch / Talkwalker enterprise seat: $50K+/year minimum. Cost per mention at typical 100K mentions/year volume: $0.50. 11× the cost of this Actor.
Hire a Chinese-speaking VA to manually check 5 platforms: ~$15/hour, ~30 mentions/hour effective. Cost per mention: $0.50. Same 11× cost, plus 24-48 hour latency.
Build it yourself: 5 separate scrapers, 5 different output schemas to parse, dedup logic, sentiment classifier, ongoing maintenance every time a platform changes their frontend. Conservatively 60-100 engineering hours upfront and 10-20 hours/month ongoing. At $150/hr loaded engineer cost, that's $9K-$15K to build + $1.5K-$3K/month to maintain. Break-even vs this Actor: never, unless you're pulling >100K mentions/month.

Every mention you pull at $0.045 buys you: the raw text, the author identity (handle, follower count, verified flag), the engagement metrics (likes, comments, shares), the timestamp, the URL, media URLs, language detection, lexicon-based sentiment scoring, and — if dedup is enabled — a crossPlatformReposts array showing exactly which other platforms amplified the same content. That's a record your competitive intelligence analyst would gladly take and immediately put into a deck.

The more you run it, the more leverage you get from each run. A single weekly snapshot tells you nothing about velocity. A daily run shows you trends. An hourly run during a crisis or product launch shows you the inflection point in real time — which is when one mention is worth $4.50 to your PR team, not $0.045.

8 concrete use cases (run it like this)

1. Daily brand health dashboard

Workflow: Cron daily at 8am local time, single brand keyword, 7-day lookback, sentiment + dedup enabled. Push the canonical mentions to your BI tool (Looker / Metabase / Hex / Sigma) for a stacked-by-platform sentiment chart, follower-weighted reach total, and top-10 highest-engagement mention list. Run for 30 days, you have a baseline. Run for 90 days, you have a leading-indicator dashboard your CMO will check daily.
Volume: 100 mentions/day × 30 days = 3,000/mo. Cost: ~$135/mo per brand.
Compare to: $4,000/mo for a Synthesio seat covering the same platforms.

2. Crisis monitoring (hourly polling with Slack alerts)

Workflow: Cron every hour, 1-day lookback, sentiment-enabled, filter for sentiment.polarity == "negative" AND authorFollowerCount > 10000. Pipe matching records to a Slack webhook that pings #pr-alerts. The moment a verified KOL posts a negative mention, your PR team knows within 60 minutes — versus 24-72 hours via Google Alerts or "someone forwarded it to me."
Volume: Most hours return 0-5 mentions. ~200 mentions/day amortized = 6,000/mo. Cost: ~$270/mo per brand. Single prevented PR crisis pays for the entire year.

3. Pre-launch competitor intelligence (one-off pull)

Workflow: Before launching a new SKU in China, pull 30 days of mentions on each competitor brand keyword across all 5 platforms. Look at: which platforms each competitor over-indexes on, which KOLs are talking about them, what sentiment dominates, what product attributes get the most positive vs negative mentions. Run this once a quarter on 5 competitors and you have the best competitive intel deck in the room.
Volume: 5 competitors × 500 mentions each = 2,500 mentions one-time. Cost: ~$112 one-time per quarter.

4. KOL identification and vetting

Workflow: Weekly run on your category keyword (e.g. "护肤" for skincare, "球鞋" for sneakers). Pull 500 mentions/week. Filter the output for authorVerified == true AND authorFollowerCount > 50000. Sort by engagementMetrics.likes descending. Top 20 results = your candidate KOL list for the week, scored by actual cultural reach not by paid impressions. Compare against your influencer agency's recommendations.
Volume: 500 mentions/week × 4 weeks = 2,000 mentions/mo. Cost: ~$90/mo per category.

5. China-watcher hedge fund alt-data signal

Workflow: Daily run on each portfolio ticker that has Chinese consumer exposure (BABA, JD, PDD, BIDU, NIO, BYD, ANTA, Yum China, POP MART, etc.). Pull mentions from Xueqiu (financial sentiment) + Weibo (consumer sentiment) + RedNote (brand affinity for consumer brands). Build a sentiment-velocity feature: 7-day mention count delta + sentiment polarity shift. Backtest against earnings surprises and brand event days — Chinese consumer sentiment leads Western analyst consensus by 2-6 weeks for most consumer-facing names.
Volume: 20 tickers × 50 mentions/day = 1,000/day × 22 trading days = 22,000/mo. Cost: ~$990/mo. Compare to: a single Bloomberg China consumer alt-data feed subscription, $80K-$200K/year minimum.

6. AI / LLM training data corpus

Workflow: One-time bulk pull on a diverse set of brand keywords across consumer categories. 50 brands × 1,000 mentions each = 50K labeled Chinese-language consumer text records with explicit sentiment labels. Drop into your SFT or RLHF pipeline for Chinese-language consumer-domain fine-tuning. This is the densest source of brand-grounded labeled Chinese text outside of paid academic corpora.
Volume: 50,000 mentions one-time. Cost: ~$2,250 one-time. Compare to: licensing a comparable academic corpus from Trinity College Dublin or Tsinghua, $15K-$50K per corpus, single-use license, 6-month delivery.

7. Cross-platform virality discovery (run it weekly, look at the dedup array)

Workflow: Weekly run, dedup ENABLED, scan the output for canonical records where crossPlatformReposts.length >= 2. Those are mentions that spread across multiple platforms within 24 hours — the closest thing to a "viral" signal you can extract from raw mention data. Use it to identify breakout moments before they hit mainstream Chinese media.
Volume: 500 mentions/week × 4 weeks = 2,000/mo. Cost: ~$90/mo per brand. Most viral moments cost $50K-$200K in PR services to capitalize on; this is how you find them 48-72 hours earlier than the agency.

8. Multi-brand portfolio monitoring (agency workflow)

Workflow: One Actor run per client brand, scheduled daily via Apify Schedules. 10 clients × 500 mentions/brand/day = 5,000 mentions/day. Each client gets their own dataset and dashboard. The agency bills $2K-$5K/client/month for "China monitoring," delivers a custom dashboard, and the underlying data cost is ~$675/client/month — leaving healthy 70%+ gross margin per client.
Volume: 5,000 mentions/day × 30 days = 150,000/mo. Cost: ~$6,750/mo for 10 brands. Revenue at $3K/client × 10 = $30K/mo. Gross margin: 77.5%.

What the output looks like

Every mention is normalized to the same schema regardless of platform. Here's a real Weibo mention from a test run on the Chinese sportswear brand 李宁 (Li-Ning):

{
    "mentionId": "weibo_4923475823745",
    "platform": "weibo",
    "brandKeyword": "李宁",
    "brandMatchType": "exact",
    "content": "李宁的新款跑鞋质量真不错，比之前的耐克舒服多了！",
    "language": "zh-CN",
    "authorName": "运动达人",
    "authorFollowerCount": 12500,
    "authorVerified": true,
    "publishedAt": "2026-05-15T14:32:11+00:00",
    "engagementMetrics": { "likes": 234, "comments": 18, "shares": 42 },
    "url": "https://weibo.com/1234567890/4923475823745",
    "sentiment": { "polarity": "positive", "score": 0.72 },
    "crossPlatformReposts": [
        { "platform": "rednote", "url": "https://www.xiaohongshu.com/explore/abc123" }
    ]
}

Three fields buyers tell me they care about most:

sentiment — lexicon-based Chinese sentiment scoring on every mention. Polarity (positive / neutral / negative) plus a numeric score. Disable it if you have your own pipeline; enabled by default.
crossPlatformReposts — the same viral post often appears across Weibo and RedNote within hours. The aggregator detects this with SimHash similarity and merges duplicates into the canonical record, with the repost paths preserved. You don't pay twice for the same mention, and you get a free virality signal.
authorFollowerCount / authorVerified — the difference between a 200-follower throwaway account and a 1.2M-follower verified KOL is the difference between "ignore this" and "alert the C-suite." Follower-weighting your dashboard is the first thing every serious buyer does with the data.

Quick start

The input is brutally simple. This is a complete config:

{
    "brandKeyword": "李宁",
    "platforms": ["weibo", "bilibili", "rednote", "douban", "xueqiu"],
    "maxMentionsPerPlatform": 200,
    "lookbackDays": 7,
    "sentimentAnalysis": true,
    "deduplication": true
}

One brand keyword, one call, get back a normalized stream of mentions across all five platforms. The lookbackDays filter applies per platform so you only get fresh content; deduplication collapses cross-platform reposts; sentimentAnalysis tags every record.

To run it on a schedule (which is where the real value compounds): use Apify Schedules and set a cron expression. 0 8 * * * for daily 8am runs, 0 * * * * for hourly, */15 * * * * for every 15 minutes during a launch or crisis. Each run hits the same Actor with your saved input config, pushes to the same dataset, and bills only on the new canonical mentions. The Actor is built to be run thousands of times — that's how you go from "snapshot" to "monitoring system."

If you have cookies for any of the platforms (logged-in browser session) you can pass them under cookieStrings to unlock higher recall and rate limits. Cookies are optional — the actor degrades gracefully without them.

For deeper single-platform scraping (full comment trees, infinite scroll, profile enrichment), use the dedicated single-platform actors directly:

Weibo Scraper — posts, hot search, comments, profiles
RedNote (Xiaohongshu) Scraper — notes, comments, profiles, video
Douban Scraper — long-form reviews and group discussions
Xueqiu Scraper — ticker-tagged posts, KOL tracking
JD.com Scraper — product detail extraction

The aggregator is for recurring cross-platform brand monitoring with normalized output. The single-platform scrapers are for one-off deep extraction inside one platform.

What this is NOT

Being honest about scope is more useful than vague promises:

Not real-time streaming. Poll-based — 5-15 minute effective refresh is realistic. If you need millisecond latency, this isn't it.
Not WeChat coverage. WeChat has no public scraping interface; trying is a fast way to get accounts banned.
Not Douyin (TikTok China). Out of scope for v0.1 — under evaluation for the roadmap.
Not a Synthesio replacement at the largest enterprise scale. Synthesio also covers TV, podcasts, news, and provides a managed-service layer. This Actor is the data layer; bring your own BI / dashboard / alerting stack. Most teams who pick this over Synthesio are choosing it because they already own their BI stack and just need the raw normalized feed at a sane price.

Compliance posture

Only public mentions — no private accounts, no DMs, no follower lists.
No login bypass; cookies are user-supplied for higher rate limits only, and they're stored as a secret in the Apify input schema (encrypted at rest).
Reviewer / commenter nicknames are partially redacted by the source platforms; this Actor passes through what the platforms display. No additional PII enrichment.

Try it free, then scale it

The Apify free plan includes monthly platform credit that covers a meaningful first batch of mentions — enough to validate the data quality on your own brand keyword before any commitment.

Try Chinese Brand Monitor on Apify →

The flow most teams follow:

Week 1: Run it once manually on your main brand keyword. Verify the output quality on a brand you know well — every mention should be one you'd recognize. (~$2-5 spend.)
Week 2: Wire it into a daily Apify Schedule. Stream the dataset to your BI tool. Watch one week of trend data. (~$25-50 spend.)
Week 3: Add a second brand keyword (competitor, partner, or category term). Add a Slack webhook for negative-sentiment alerts above a follower threshold. (~$50-100 spend.)
Month 2 onward: Production. Daily monitoring on your core brand portfolio, hourly during launches and crises, monthly competitive intel pulls. (Typical mid-market team: $200-1,500/mo.)

Pricing is transparent at every step: $0.045 per canonical mention, billed only on the deduplicated records. No subscription, no minimum spend, no annual contract. Pause it for a month, scale it up 10x next week, switch brand keywords mid-run — it all just works.

The teams getting the most out of this are running it on a schedule, daily or hourly, across multiple brand keywords, piping the normalized output into their existing BI / Slack / dashboard stack. Each run pays for itself in the first time it surfaces a mention you would have missed.

Open an issue on the Actor page if you hit any edge case. Typical turnaround on fixes is 48 hours.

JD.com's isJdSelfRun Flag Is the Best Gray-Market Detection Signal in Chinese E-Commerce (Python Scraper Inside)

Sami — Fri, 15 May 2026 22:08:41 +0000

If your brand sells on JD.com (China's #2 e-commerce platform, ~600M annual active users) — or competes against one that does — there's a gray-market problem you can't see without one specific field in JD's data.

That field is isJdSelfRun. It tells you whether a given product listing is fulfilled by JD itself (their warehouses, their warranty, their return logistics) or by a third-party merchant on JD's marketplace. Combined with the seller's sellerType (flagship / franchise / specialty / self-run), it's the single cleanest signal for detecting unauthorized resellers on Chinese e-commerce — and almost no generic scraper surfaces it.

This post walks through:

Why JD's hybrid retail model creates the gray-market detection opportunity
The exact field signatures (isJdSelfRun, sellerType) and what they mean
Three concrete workflows: brand authorization audit, competitive pricing, gray-market detection
A 50-line Python integration with the Apify Actor I built around this
Honest cost math at indie scale and at hedge-fund scale

If you don't want to read the whole thing: the Actor is at zhorex/jd-scraper, and pricing is $0.008 per product detail + $0.02 per seller store record (pay-per-event, no subscription).

The hybrid retail model that creates the signal

JD.com is structurally different from Tmall and Pinduoduo. Tmall is a marketplace — every SKU is sold by a third-party merchant; Alibaba just runs the platform. JD operates a hybrid: a meaningful chunk of its catalog is sold and shipped by JD itself (JD Logistics, JD Plus warranty, JD's own returns), with the rest fulfilled by marketplace merchants.

That hybrid creates an information asymmetry buyers can exploit. When a consumer searches a brand's SKU on JD, they see all listings — but the trust signal comes from whether it's JD-self-run or a third-party. For a brand monitoring team, the question becomes: of the third-party listings of my SKU, which are authorized resellers and which are gray-market?

The data answers it in two fields:

{
    "productId": "100009082476",
    "sellerName": "Apple产品京东自营旗舰店",
    "isJdSelfRun": true,
    "sellerId": "1000003566",
    ...
}

isJdSelfRun: true means JD is the seller. The other listings — those with isJdSelfRun: false — are where the gray-market questions live, and where you need the seller's type to decide.

The seller type enum

A separate scrape against the seller store endpoint resolves to one of four values:

{
    "sellerId": "1000003566",
    "sellerType": "flagship_store",
    "serviceScore": 4.9,
    "logisticsScore": 4.8,
    "descriptionAccuracyScore": 4.9,
    ...
}

flagship_store (官方旗舰店) — the brand's own JD store. There should be exactly one per brand. If you see multiple, you have a counterfeit-or-impersonator problem.
franchise_store (品牌专营店) — authorized franchise of the brand. Brands typically maintain a list of these.
specialty_store (专卖店) — third-party that specializes in selling the brand. Often authorized via distribution agreement; sometimes gray-market.
jd_self_run (京东自营) — JD's direct retail. Always legitimate.

The canonical gray-market signature: a flagship_store listing alongside three specialty_store listings priced 20-40% lower on the same SKU. Those specialty stores are usually moving inventory acquired outside the authorized channel (parallel imports, diverted product, refurbished-as-new). They're flagged the moment your monitoring sees the price gap.

Three workflows the data unlocks

Workflow 1 — Brand authorization audit

Submit your SKU IDs. Get back a record per listing with sellerType resolved. Filter to entries where isJdSelfRun: false AND sellerType is not in your authorized list. That's your unauthorized reseller list, refreshed on whatever cadence you want.

A small brand watching 50 SKUs across 200 listings (4 average sellers per SKU) costs about $2 per refresh: 200 seller records × $0.02 + 50 product details × $0.008 = $4.40 ($2.20 if you skip product detail and only check sellers).

Workflow 2 — Competitive pricing intelligence

The product detail mode returns a realtimePrice field that is fetched fresh at scrape time, not parsed from cached HTML. JD runs flash discounts that move prices within hours; cached scrapers miss them entirely.

Tracking 200 competitor SKUs hourly = 200 × 24 × 30 = 144,000 detail records per month, $1,152 in raw event cost. At hedge-fund-grade refresh rates this is real money, but it's the right order of magnitude for the buyer cohort that already pays $3K-15K/month for alt-data feeds.

Tracking 200 SKUs daily (more realistic for a brand team) = 6,000 records × $0.008 = $48/month. Cheap enough to run as a cron.

Workflow 3 — Gray-market detection at scale

The canonical pattern in code:

for product in products:
    listings = [l for l in product["allListings"] if not l["isJdSelfRun"]]
    cheap_specialty = [
        l for l in listings
        if l["sellerType"] == "specialty_store"
        and l["price"] < product["msrp"] * 0.80
    ]
    if len(cheap_specialty) >= 3 and any(l["sellerType"] == "flagship_store" for l in listings):
        alert(product, cheap_specialty)

That's the brand-monitoring signal: a real flagship store coexisting with three or more sub-MSRP specialty stores on the same SKU. Brand teams pay agencies five-figure annual contracts to surface exactly this kind of alert; running it yourself on this data feed costs cents per check.

The 50-line Python integration

Here's the working integration end-to-end. Replace YOUR_TOKEN with your Apify API token:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

# Scrape product details for your SKU list
run = client.actor("zhorex/jd-scraper").call(run_input={
    "mode": "product_detail",
    "productUrls": [
        "https://item.jd.com/100009082476.html",
        "https://item.jd.com/100012345678.html",
    ],
})

unauthorized_sellers = set()
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if not item["isJdSelfRun"]:
        unauthorized_sellers.add(item["sellerId"])
        print(
            f"Non-self-run listing: {item['productTitle']}\n"
            f"  Seller: {item['sellerName']} (id {item['sellerId']})\n"
            f"  Price: {item['realtimePrice']}\n"
        )

# Now drill into those sellers to classify them
if unauthorized_sellers:
    seller_run = client.actor("zhorex/jd-scraper").call(run_input={
        "mode": "seller_store",
        "sellerUrls": [f"https://mall.jd.com/index-{sid}.html" for sid in unauthorized_sellers],
    })
    for seller in client.dataset(seller_run["defaultDatasetId"]).iterate_items():
        flag = "⚠️ AUDIT" if seller["sellerType"] == "specialty_store" else "✅"
        print(f"{flag} {seller['sellerName']} → {seller['sellerType']} (service: {seller['serviceScore']})")

That's the whole audit. Two API calls, classified output, ready to feed into Slack alerts / spreadsheet exports / BI dashboards.

Honest pricing — what does this cost in production?

Workflow	Volume / month	Cost / month
Brand watchlist — 50 SKUs daily	1,500 product details	~$12
Brand authorization audit	500 sellers, monthly	~$10
Competitive pricing — 200 SKUs daily	6,000 product details	~$48
Competitive pricing — 200 SKUs hourly	144,000 product details	~$1,152
Gray-market sweep — 200 SKUs + 50 sellers	200 details + 50 sellers	~$2.60

Indie brand teams typically run the daily/monthly workflows ($10-60/month). Hedge-fund alt-data and agency-scale customers run hourly or 15-minute refreshes (low four figures monthly). Both work on the same Actor with the same event-priced billing.

What this Actor doesn't do

Two honesty disclosures:

No search discovery. You bring the SKU list. Discovery requires a different scraping pattern that doesn't survive shared residential proxy pools the way product detail and seller store do.
No review scraping. Same reason — JD's WAF gates the review API at the IP-reputation level on shared pools. If you need review sentiment, the Apify Store has other scrapers, or contact me for a premium-proxy integration.

The README on the Actor page documents this in a "Known limitations" section. If your workflow needs either, this Actor isn't the right tool.

Why I built this

I run a portfolio of six Chinese-platform scrapers on Apify Store (zhorex). Five of them cover sentiment and content: Weibo for trending, RedNote (Xiaohongshu) for lifestyle, Bilibili for video, Douban for long-form reviews, Xueqiu for stock-cashtag discussion. The JD scraper extends the suite into commerce — the missing layer for buyers who already use the social ones for brand monitoring.

The six together are a stack. A consumer-electronics brand can track sentiment on Weibo, video reviews on Bilibili, lifestyle unboxings on RedNote, and gray-market resellers on JD — all on the same vendor, same billing, same API surface.

Try it

The Actor is live: zhorex/jd-scraper. Pay-per-event billing — no subscription, no setup fee. Run a small evaluation batch (the Apify Free plan includes monthly platform credit you can apply to the run) to confirm output quality on your SKU list before scaling up.

The rest of the Chinese Digital Intelligence Suite:

Weibo Scraper — pair with JD to catch when a SKU trends socially before stock-outs hit
RedNote Scraper — Chinese lifestyle unboxings; useful for fashion, beauty, baby, home brands
Bilibili Scraper — video reviews; especially valuable for tech and consumer electronics SKUs
Xueqiu Scraper — Chinese retail-investor sentiment; pair if you trade JD stock (NASDAQ:JD) alongside operational metrics
Douban Scraper — long-form film / book / music reviews; less relevant for commerce but useful for IP / entertainment teams

If you ship a brand-monitoring workflow on top of any of these, drop a comment with what you're tracking. If this saved you the time of building an integration from scratch, a heart on the post or a follow keeps these writeups coming.

Scraping Chinese Social Platforms for LLM Training Data: A Practical Multi-Source Pipeline (Python, 2026)

Sami — Tue, 12 May 2026 20:02:05 +0000

If you're training Chinese-language models — or multilingual models that need real Chinese coverage, not just translated English — the data problem is the bottleneck. Common Crawl gives you the open web. HuggingFace gives you the curated stuff. But the linguistic patterns that matter most for cultural alignment — slang, memes, code-mixed English-Chinese, regional variations, real-time discourse — those live in places Common Crawl barely touches.

Three platforms that matter most for Chinese training corpora in 2026:

Weibo (微博) — 580M+ MAU, microblogging, real-time discourse, similar role to X/Twitter
Bilibili (哔哩哔哩) — 300M+ MAU, video platform, comments + danmaku give you code-mixed natural language at volume
Xiaohongshu / RedNote (小红书) — 300M+ MAU, lifestyle posts with longer-form content, female-skewed register

This post walks through how to build a multi-source pipeline that pulls clean structured data from all three, normalize across platforms, and ship it into your training datasets. With code, schema, and economics.

A note on legal posture: this entire pipeline accesses only publicly visible data — no auth bypass, no captcha solving, no scraping behind login. That matches the standard most AI training teams operate under in 2026, post-NYT-vs-OpenAI. Always consult your legal team for your specific use case and jurisdiction.

Why these three (and not, say, Douyin or Zhihu)

Each platform contributes a different linguistic register:

Weibo posts are short, high-frequency, conversational. Best for:

Everyday Mandarin patterns
Trending slang and memes (热搜 reflects what's actually viral right now)
Public sentiment on news and policy
Brand-mention contexts

Bilibili comments and danmaku are unique:

Heavy code-mixing English ↔ Chinese (gaming, tech, anime communities)
Real-time chat-style language
Subculture vocabulary (gaming, fandom, two-dimensional culture / 二次元)
Longer thread discussions on long-form videos

RedNote posts lean longer and more curated:

Beauty / lifestyle / travel / food vocabulary
Product-attribute language (skincare ingredients, fashion descriptors)
Female-skewed register and topics
Aspirational / descriptive framing

Douyin (Chinese TikTok) and Kuaishou are dominantly video — text data is sparse. Zhihu (Q&A) is great for long-form but dominated by single-author voice. The triad above gives you the best balance of volume, diversity, and accessibility.

Pipeline architecture

The cleanest architecture for an AI training data pipeline:

[Weibo Scraper]    →
[Bilibili Scraper] →  [Normalize]  →  [Dedup + Filter]  →  [JSONL]
[RedNote Scraper]  →

Each scraper outputs platform-native JSON. A normalization layer flattens to a common schema. Deduplication on text hash + filtering by min-length / language detection ships clean data into your training format.

Below: I use Apify-hosted scrapers for the extraction layer (they handle anti-bot, rate limiting, and schema stability so you don't have to). The normalization + dedup is your code — straight Python.

Step 1 — Pulling from Weibo

For training data, the high-value combination is:

Hot search topics (real-time trending — what people are talking about right now)
Posts under those topics (organic conversation about real issues)

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

def collect_weibo_corpus(target_topics: int = 50, posts_per_topic: int = 100):
    # 1a. Pull current trending topics
    topics_run = client.actor("zhorex/weibo-scraper").call(run_input={
        "mode": "hot_search",
        "maxResults": target_topics,
    })
    topics = list(client.dataset(topics_run["defaultDatasetId"]).iterate_items())

    # 1b. For each topic, pull underlying posts
    corpus = []
    for topic in topics:
        posts_run = client.actor("zhorex/weibo-scraper").call(run_input={
            "mode": "search",
            "searchQuery": topic["title"],
            "maxResults": posts_per_topic,
        })
        for post in client.dataset(posts_run["defaultDatasetId"]).iterate_items():
            corpus.append({
                "platform": "weibo",
                "topic": topic["title"],
                "category": topic.get("category"),
                "text": post.get("text", ""),
                "author": post.get("authorName"),
                "engagement": (post.get("attitudesCount", 0) +
                               post.get("commentsCount", 0) +
                               post.get("repostsCount", 0)),
                "post_url": post.get("postUrl"),
                "scraped_at": post.get("scrapedAt"),
            })
    return corpus

Volume math: 50 topics × 100 posts = 5,000 items per snapshot. At $0.005/item that's $25 per pull. Run daily for a year ≈ $9,125.

Step 2 — Pulling from Bilibili

Bilibili gives you something the others don't: comments on long-form videos. That's where heavy code-mixing happens (tech tutorials, gaming streams, study-with-me content, drama analysis). For training data, comments are higher-value than video metadata.

def collect_bilibili_comments(category: str = "knowledge",
                               videos: int = 50,
                               comments_per: int = 100):
    # Get popular videos in the category
    popular_run = client.actor("zhorex/bilibili-scraper").call(run_input={
        "mode": "popular",
        "category": category,
        "maxResults": videos,
    })
    items = list(client.dataset(popular_run["defaultDatasetId"]).iterate_items())
    bvids = [v["bvid"] for v in items if v.get("bvid")]

    # Pull comments on each
    corpus = []
    for bvid in bvids:
        comments_run = client.actor("zhorex/bilibili-scraper").call(run_input={
            "mode": "video_comments",
            "videoUrls": [f"https://www.bilibili.com/video/{bvid}"],
            "maxComments": comments_per,
            "sortComments": "hot",
        })
        for c in client.dataset(comments_run["defaultDatasetId"]).iterate_items():
            if c.get("type") != "comment":
                continue
            corpus.append({
                "platform": "bilibili",
                "category": category,
                "text": c.get("text", ""),
                "author": c.get("authorName"),
                "engagement": c.get("likeCount", 0),
                "video_bvid": bvid,
                "scraped_at": c.get("scrapedAt"),
            })
    return corpus

Note: Bilibili throttles comment depth on cloud IPs — top ~3 per video without residential proxies. For training-data scale you don't need every comment, just enough diversity, so the top-N approach is fine and cheaper.

Categories worth pulling for diverse coverage: knowledge, tech, game, life, food, fashion, cars, entertainment.

Step 3 — Pulling from RedNote

RedNote gives you longer, more curated content — good for training models on aspirational and descriptive Chinese. The seed-query approach lets you control topical distribution, important for avoiding bias toward whatever's trending the day you scrape.

def collect_rednote_corpus(seed_queries: list, posts_per_query: int = 50):
    corpus = []
    for query in seed_queries:
        run = client.actor("zhorex/rednote-xiaohongshu-scraper").call(run_input={
            "mode": "search",
            "searchQuery": query,
            "maxResults": posts_per_query,
        })
        for post in client.dataset(run["defaultDatasetId"]).iterate_items():
            corpus.append({
                "platform": "rednote",
                "topic": query,
                "text": post.get("title", ""),
                "author": (post.get("author") or {}).get("nickname"),
                "engagement": post.get("likes", 0),
                "post_url": post.get("postUrl"),
                "scraped_at": post.get("scrapedAt"),
            })
    return corpus

# Diverse seed queries spread coverage across topics
seeds = [
    "护肤心得",      # skincare experience
    "穿搭",          # outfits
    "美食推荐",      # food recommendations
    "旅行攻略",      # travel guides
    "健身打卡",      # fitness check-in
    "读书笔记",      # reading notes
    "育儿日记",      # parenting diary
    "职场感悟",      # work reflections
]
rednote_data = collect_rednote_corpus(seeds, posts_per_query=100)

For richer body content per post (beyond title), pivot to mode: post_details with the post URLs you want to deep-dive on.

Step 4 — Normalization and dedup

All three scrapers produce platform-specific schemas; the per-step code above already brings them to a common shape:

{
    "platform": "weibo" | "bilibili" | "rednote",
    "topic": str,
    "text": str,
    "author": str,
    "engagement": int,
    "scraped_at": ISO8601,
}

Enough to ship into a JSONL training format. For higher quality, layer in filtering:

import hashlib

def filter_corpus(corpus, min_chars: int = 10, max_chars: int = 5000):
    seen = set()
    out = []
    for item in corpus:
        text = (item.get("text") or "").strip()
        if not (min_chars <= len(text) <= max_chars):
            continue
        h = hashlib.md5(text.encode("utf-8")).hexdigest()
        if h in seen:
            continue
        seen.add(h)
        out.append(item)
    return out

For pretraining-grade quality, also add fastText / langdetect to filter non-Chinese content, and a profanity / PII pass appropriate to your training context.

Economics at training-corpus scale

A reasonable Chinese-language pretraining contribution might be 10M items across platforms:

Platform	Items	Cost @ $0.005
Weibo	5M	$25,000
Bilibili	3M	$15,000
RedNote	2M	$10,000
Total	10M items	$50,000

Apify free tier ($5/month credit) covers ~1,000 items per actor for prototyping.

For comparison, hiring 2 senior engineers to build and maintain DIY Chinese-platform extraction for 6 months: $150K-300K — and you don't even get the data, just the tooling.

For 100M+ items (real pretraining scale), volume pricing or a custom enterprise contract makes sense. See enterprise section below.

When to build vs buy

Build it yourself if:

You're scraping 100M+ items per month and have a dedicated team
You need real-time streaming below 1-second latency (this pipeline is batch)
Your legal team requires you to own the entire data path

Use the hosted scrapers if:

You're under 50M items per month per platform
You want time-to-data measured in hours, not months
You don't want to maintain three platform-specific scrapers as APIs evolve

The actors

Weibo Scraper — hot_search, search, post_comments, user_posts
Bilibili Scraper — search, popular, video_detail, video_comments, user_videos
RedNote (Xiaohongshu) Scraper — six modes covering posts, profiles, comments, video

All three at $0.005/result. Pure HTTP — no browser, no proxy required for moderate volumes.

Enterprise / training-scale

If you're building actual training corpora (not prototyping), DM me on any actor page or open an Issue with subject "Training data inquiry":

Custom output schemas matched to your training pipeline (Parquet / Arrow / your dialect of JSONL)
Volume pricing above 1M items/month per platform
Dedicated proxy infrastructure for sustained throughput
Schema stability SLA so your training runs don't break mid-epoch

Issues typically get a response within 48 hours.

FAQ

Is this legal? Each Actor accesses only publicly visible data — no auth, no captcha bypass, no login walls. The same data any anonymous browser user can see. Standard ToS-compliant scraping posture as of 2026. Consult your legal team for jurisdiction-specific guidance.

What about rate limits? The hosted Actors handle rate-limit responses with exponential backoff. For 1M+ items/day per platform, talk to me about dedicated infrastructure.

Can I get historical data? The Actors return what's currently public. For longitudinal datasets, schedule them via Apify Schedules at the cadence you need (hourly / daily / weekly) and version-control your dataset snapshots.

Do you offer streaming / real-time? Not currently. The Actors are pull-based. If you need streaming, that's a custom integration.

Other platforms? I also maintain a RedNote Shop Scraper for Xiaohongshu e-commerce listings — useful if your model needs to reason about products, pricing, or commerce vocabulary.

Other relevant work

If you're building Chinese intelligence at scale, the full suite:

RedNote Scraper — lifestyle social
RedNote Shop Scraper — Xiaohongshu e-commerce (product metadata, pricing, vendor info)
Weibo Scraper — microblogging, hot search, sentiment
Bilibili Scraper — video creator analytics

If this saved you a quarter of dev time, a 30-second review on any of the Actor pages helps a lot. ⭐

Found a bug or have a feature request? Open an Issue — I usually ship fixes within 48 hours.

Influencer Vetting at Scale on Xiaohongshu (RedNote): A Practical Python Guide for Brand Teams 2026

Sami — Mon, 11 May 2026 16:14:10 +0000

RedNote — known internationally as Xiaohongshu (小红书) or Little Red Book — has become the single most consequential platform for influencer marketing in China. With 300M+ monthly active users skewed female and Gen Z, it's where beauty, fashion, lifestyle, and travel brands first place a campaign before going wider. After the TikTok-uncertainty migrations of 2024–2025, RedNote also became the de facto Western fallback for many creators.

If you're a brand team, agency, or media buyer working in or with China, you need a way to vet RedNote influencers at scale. Manual scrolling doesn't cut it past five creators. Here's the structured approach.

What "vetting at scale" actually means

For a single influencer partnership, you typically want answers to:

Reach: how many followers, but more importantly how many people actually see their content (median impressions / followers)?
Engagement quality: average likes/comments/saves per post — and the distribution. A creator with 100K followers and 50 posts averaging 500 likes is very different from one with 100K and a few viral 10K posts.
Niche fit: do their tags and topics align with your category?
Audience hints: from bio, location, profile signals — who follows them?
Authenticity: posting cadence, sponsored-content ratio, content reuse from other platforms.

The data lives in two places on RedNote: profile metadata (followers, bio, location, verified status, total likes received) and recent posts (titles, like counts, content type, publish dates).

What you can extract per creator

{
    "userId": "5d7439b40000000001009f54",
    "nickname": "BeautyBlogger123",
    "avatar": "https://sns-avatar-qc.xhscdn.com/...",
    "redId": "100123456",
    "description": "Skincare reviews, K-beauty translations. Seoul-based.",
    "followers": 184500,
    "following": 320,
    "totalLikes": 1240000,
    "gender": 1,
    "location": "Seoul, South Korea",
    "isVerified": true,
    "tags": ["Beauty Blogger", "K-Beauty", "Skincare"]
}

Per recent post:

{
    "postId": "64be395b0000000010030b56",
    "type": "video",
    "title": "Morning skincare routine for dry skin",
    "likes": 15234,
    "scrapedAt": "2026-05-06T12:00:00Z"
}

Combined, you can compute:

Median likes / post (use median, not mean — viral outliers skew means)
Engagement rate = median likes / followers (RedNote benchmark for healthy: 2–5%, viral creators 8%+)
Post frequency (posts per week — burnout warning if dropping)
Content-type ratio (video vs image — videos get higher reach in 2026)
Niche concentration (% of recent posts matching your category keywords)

Practical Python: building a vetting batch

from apify_client import ApifyClient
import statistics

client = ApifyClient("YOUR_APIFY_TOKEN")

def vet_creator(profile_url: str) -> dict:
    # Profile data
    profile_run = client.actor("zhorex/rednote-xiaohongshu-scraper").call(run_input={
        "mode": "profile",
        "userUrl": profile_url,
    })
    profile = next(client.dataset(profile_run["defaultDatasetId"]).iterate_items())

    # Recent posts
    posts_run = client.actor("zhorex/rednote-xiaohongshu-scraper").call(run_input={
        "mode": "user_posts",
        "userUrl": profile_url,
        "maxResults": 50,
    })
    posts = list(client.dataset(posts_run["defaultDatasetId"]).iterate_items())

    likes = [p.get("likes", 0) for p in posts if p.get("likes")]
    median_likes = statistics.median(likes) if likes else 0
    er = (median_likes / profile["followers"]) * 100 if profile.get("followers") else 0

    return {
        "nickname": profile.get("nickname"),
        "followers": profile.get("followers"),
        "verified": profile.get("isVerified"),
        "tags": profile.get("tags"),
        "post_count_recent": len(posts),
        "median_likes": median_likes,
        "engagement_rate_pct": round(er, 2),
        "video_ratio": sum(1 for p in posts if p.get("type") == "video") / max(len(posts), 1),
    }

# Vet a list
candidates = [
    "https://www.xiaohongshu.com/user/profile/USER_ID_1",
    "https://www.xiaohongshu.com/user/profile/USER_ID_2",
]

results = [vet_creator(u) for u in candidates]
results.sort(key=lambda r: r["engagement_rate_pct"], reverse=True)

for r in results:
    print(f"{r['nickname']:25s}  followers={r['followers']:>7,}  ER={r['engagement_rate_pct']}%")

That gives you a sortable spreadsheet of candidates ranked by genuine engagement, not vanity follower counts.

Red flags to look for

Engagement rate < 0.5% with > 100K followers → likely bought or stale
Only image posts when category is video-heavy → low reach in 2026's algorithm
Posts in the last 30 days < 4 → low retainer reliability
No tags or generic tags → low discoverability inside RedNote search
Median likes within 10x of mean → relatively consistent (good); 50x+ means single-viral-driven (risky)

When to skip the DIY approach

RedNote's public surface requires sustained anti-bot engineering to extract reliably at scale. The schema also evolves regularly, so a scraper that worked last month can quietly start returning empty arrays this month without raising any error you'd notice in your pipeline.

That's why I maintain the RedNote Scraper on Apify — six modes (search, user posts, profiles, post details, comments, video) with consistent output schemas across them. The infrastructure work (session handling, rate limiting, schema stability) is already done.

Pricing is pay-per-event: $0.005 per result. A typical influencer batch (50 candidates, profile + 50 posts each) costs about $13. The Apify free tier ($5 monthly) covers ~1,000 items.

FAQ

Is the data from open profiles only? Yes — public-facing profile and post data. Private/locked accounts are not accessible. Same content any anonymous browser user can see.

Does it work on xhslink.com short links? Yes, those resolve automatically.

Can I get post-level comments? Use mode: comments on individual post URLs.

What about RedNote vs Xiaohongshu vs Little Red Book? All the same platform — the Actor handles all three name conventions and URL formats.

Is scraping Xiaohongshu legal? This Actor accesses publicly visible content only. No authentication is bypassed. Always consult your local laws.

Full Chinese intelligence stack

Brand teams running campaigns across Chinese platforms typically pair this with:

RedNote Scraper — (this one, social side)
RedNote Shop Scraper — Xiaohongshu e-commerce (products, vendors, prices)
Weibo Scraper — microblogging, brand mentions, hot search
Bilibili Scraper — video creator analytics

Vetting more than 100 creators per month? I offer custom output schemas, dedicated proxy pools, SLA support, and volume discounts. DM me on Apify or open an Issue titled "Enterprise inquiry".

Bug? Issues are typically fixed within 48 hours.

If this saved you time, a 30-second review on the Apify Store helps a lot. ⭐