DEV Community: Sami

Google Ads can spend up to 2x your daily budget. I built a Chrome extension that catches it before it happens.

Sami — Thu, 30 Apr 2026 15:17:28 +0000

If you've ever opened Google Ads and noticed your campaign spent way more than the daily budget you set, you're not imagining it. Google's documentation explicitly says they may spend up to twice your daily budget on any given day, evening it out across the month. That's not a bug — it's how their pacing engine has always worked.

What changed in March 2026: Google now aggressively targets 100% of your monthly limit — which is 30.4× your daily budget. Even with ad scheduling. So if your campaigns only run 22 days a month (weekdays only, for example), Google can push up to 38% more spend per active day than you'd expect from your daily budget setting.

Most PPC managers don't notice until the damage is done. The Campaigns tab in Google Ads doesn't tell you whether you're on pace or headed for overspend. You'd need a spreadsheet, a calendar, and a calculator open in another window — or a SaaS tool that costs $49 to $749 per month.

I got tired of the spreadsheet route. So I built a Chrome extension that does it inside Google Ads, in real time, for free up to 3 campaigns. Walking through the build because the technical approach is interesting and the pricing math vs SaaS tools is genuinely lopsided.

What budget pacing actually requires

The math is simple. For each campaign:

expected_spend_today = daily_budget × (days_elapsed_in_month / total_days_in_month)
pacing_ratio = actual_spend_today / expected_spend_today

# pacing_ratio < 1.10 → on pace
# pacing_ratio 1.10–1.20 → slight overspend
# pacing_ratio > 1.20 → overspend risk

That's the whole core logic. SaaS tools wrap this in dashboards, alerts, multi-account aggregation, and reporting. But the underlying calculation is six lines of code.

The reason SaaS tools charge $49+/month isn't the math — it's the data plumbing. They connect to the Google Ads API (OAuth, refresh tokens, quota management), run server-side jobs to pull your accounts on a schedule, store results in a database, render charts. Real infrastructure cost.

But here's the thing: your campaign data is already visible on your Google Ads screen. Names, budgets, costs, statuses — the information is sitting in the DOM right there. If you're already looking at Google Ads, why does anyone need to call an API to tell you what you're already looking at?

The Chrome extension approach

I built AdPacer as a Manifest V3 Chrome extension that reads the campaign data from the Google Ads page DOM and overlays three pacing indicators directly on the interface you're already using. Architecture:

Content script runs on ads.google.com URLs
MutationObserver detects when the campaigns table renders or updates (Google Ads is a heavy SPA so this matters)
DOM parsing extracts campaign name, daily budget, current spend per row
Pacing math runs locally on the extracted values
DOM injection adds the colored pacing bars and projected-spend badges next to each campaign row
Notifications API for the periodic overspend checks

Zero API calls. Zero authentication flows. Zero backend. Zero data leaves the user's browser. Everything runs in the page's content-script context.

The privacy implication is meaningful: AdPacer cannot exfiltrate your Google Ads data even if it wanted to. There's no network request to anywhere. SaaS tools, however privacy-conscious their privacy policies are, send your campaign data to their servers as a fundamental part of how they work.

What you actually see in Google Ads after install

Three additions to the standard Campaigns tab:

1. Pacing bars — a color-coded bar next to each campaign:

Green: on pace, within 10% of expected spend
Yellow: ahead of pace, 10–20% over expected
Red: overspend risk, 20%+ over expected

2. Projected end-of-month spend — a badge showing what your monthly spend will be if you continue at the current daily run rate. Updates as the page data updates. No spreadsheet required.

3. Browser notifications — when any campaign crosses your threshold (configurable from 10% to 25%). Checks automatically every 30 minutes. Catch problems early instead of at month-end reconciliation.

That's it. Install, open Google Ads, see your pacing.

Pricing — and why this is structured the way it is

I deliberately wanted to make this accessible to freelancers and small teams, not enterprise-priced.

Tier	Price	Limit
Free	$0	Up to 3 campaigns. All core features. No credit card, no trial expiration.
Pro	$14/mo	Unlimited campaigns, custom thresholds, priority support.
Agency	$29/mo	Multi-account support, PDF pacing reports, team sharing.

The free tier covers most freelancers managing 1-3 client accounts at a time, or small e-commerce teams running a couple of brand/generic/shopping campaigns. Pro is for in-house PPC managers running 5-50 campaigns. Agency is for teams managing multiple clients.

For comparison with the SaaS landscape:

Tool	Lowest tier	Notes
TrueClicks	$49/mo	Broader PPC management
Optmyzr	$129/mo	Optimization suite
WordStream	$299/mo+	Enterprise tier
AdPacer	$0–$14/mo	Pacing only

If you need full PPC management — bid optimization, A/B testing, audience suggestions, the whole stack — the SaaS tools are doing a lot more than pacing. But if all you actually need is "tell me when a campaign is going to overspend," paying $49-299/month for that single feature is overkill.

Who I built this for

PPC managers running Google Ads daily who want instant budget visibility without context-switching to another tool. Freelancers managing 1-5 client accounts where SaaS pricing eats too much of the margin. Agency teams who need quick pacing checks across multiple campaigns. E-commerce advertisers watching ROAS and budget efficiency in real time.

If you're an enterprise team running 200+ campaigns with complex bid strategies, this isn't for you — you probably already have an Optmyzr-class tool. If you're somewhere between "spreadsheet" and "expensive SaaS," this fills the gap.

What it doesn't do (yet)

Being honest about scope:

No Microsoft Ads / Bing Ads support yet (Google Ads only)
No Meta / TikTok Ads (different DOMs, different challenges, would be a separate extension)
No historical pacing trends beyond current month
No bid suggestions or campaign optimization (that's a different problem space)

Pacing is a single, focused use case. The extension does that one thing well rather than trying to be a half-decent everything tool.

Install link

AdPacer on the Chrome Web Store

Free tier covers up to 3 campaigns with no credit card and no trial expiration — install and see if it solves your problem before paying anything.

If you're a PPC manager and the spending pattern Google introduced in March 2026 has been causing you headaches, this is the lowest-friction way to catch overspend before it happens. If you're a developer reading this for the technical approach: yes, the entire thing runs client-side via DOM parsing — no API key, no backend, no data leaves the browser.

Happy to answer questions about either side.

How to scrape Weibo (微博) data with Python in 2026 — the Sina Visitor System and how to handle it

Sami — Thu, 30 Apr 2026 14:58:19 +0000

Weibo is China's Twitter — the platform where Chinese public opinion forms, brand crises break first, and government statements land. 580M+ monthly active users, mostly mainstream demographics. If you're doing China market intelligence, brand monitoring, or PR analytics, Weibo is one of the platforms you can't skip.

The challenge: Weibo's developer API requires a Chinese business license, has severe rate limits, and exposes very limited data. For Western teams, web scraping is the practical option. The interesting twist is Weibo's Sina Visitor System — an auth flow that makes anonymous access possible for some endpoints but not others. Understanding which is which matters for what you can actually scrape.

This article covers the technical landscape (with real Python code) and points to a hosted scraper if you'd rather skip the maintenance.

What Weibo serves

A Weibo post is structured similarly to a tweet but with longer character limits and more structured engagement signals:

Post text (140 to 2,000 characters depending on user level)
Repost chain — Weibo's quote-tweet equivalent, central to virality tracking
Engagement metrics — attitudes_count (likes), comments_count, reposts_count
Hashtags and mentions
Geolocation if disclosed by user
Author profile — follower count, verification status, verified reason (e.g., "新浪科技 official Weibo")
Media — images, videos

A Weibo user profile gives you:

User ID (numeric)
Screen name (display name)
Description / bio
Followers / friends counts
Statuses count (total posts)
Verification status with reason text — this is gold for identifying official accounts vs personal vs corporate

For monitoring use cases, the metric that matters most depends on your goal:

Crisis monitoring: track comments_count and repost velocity. A spike in either signals viral attention.
Brand presence: track post frequency from verified accounts in your category.
KOL identification: filter by verified=true + follower count above a threshold.

The Sina Visitor System

This is the key technical concept for scraping Weibo without a Chinese business license.

When you visit Weibo without logging in, Sina automatically issues you a "visitor cookie" via what they call the Sina Visitor System (SVS). This cookie lets you access limited public data — specifically:

Hot search / trending topics: full access
Post comments: full access for any public post
Post viewing: limited

For these endpoints, scraping is straightforward — get a visitor cookie, hit the AJAX endpoint, parse JSON.

What the visitor cookie does NOT give you:

Search by keyword (returns hot timeline as a fallback instead of true search results)
User posts beyond profile basics (you get the profile, not the user's post history)

For those, you need a real logged-in cookie — specifically the SUB cookie value from a logged-in browser session. We'll get to that.

Approach 1: Build it yourself

The Sina Visitor System flow looks roughly like this:

import httpx

# Step 1: Hit the visitor system to get a tid (temporary ID)
visitor_url = "https://passport.weibo.com/visitor/genvisitor"
resp = httpx.post(visitor_url, data={
    "cb": "gen_callback",
    "fp": '{"os":"1","browser":"Chrome","fonts":"undefined","screenInfo":"1920*1080*24","plugins":""}',
})
# The response is a JSONP-wrapped JSON. Strip the wrapper, parse, extract tid.

# Step 2: Use tid to get the SUB visitor cookie
incarnate_url = "https://passport.weibo.com/visitor/visitor"
resp = httpx.get(incarnate_url, params={
    "a": "incarnate",
    "t": tid,
    "w": "2",
    "c": "100",
})
# Response sets cookies. Extract SUB and SUBP from response.cookies.

# Step 3: Use those cookies to call AJAX endpoints
hot_search_url = "https://weibo.com/ajax/side/hotSearch"
resp = httpx.get(hot_search_url, cookies={"SUB": sub, "SUBP": subp})
data = resp.json()
# data["data"]["realtime"] is the hot search list

That's the rough shape. In practice you'll handle:

Rate limit responses (HTTP 418, 429) with exponential backoff
Cookie expiration (visitor cookies last hours, not days)
AJAX endpoint changes (Weibo periodically reshuffles paths)
Anti-scraping fingerprint checks (less aggressive than RedNote, but still present)

For the keyword-search and user-posts endpoints, you'll need a real SUB cookie from a logged-in account:

# Get SUB from your browser DevTools → Application → Cookies → weibo.com
# Look for the cookie named "SUB"
sub_cookie = "SUB=_2A25Fxxxxxx..."

resp = httpx.get(
    "https://weibo.com/ajax/side/searchAll",
    params={"q": "人工智能"},
    cookies={"SUB": sub_cookie.split("=", 1)[1]},
)

Cookies typically last several days before expiring, depending on Weibo's session policies.

DIY cost breakdown

Cost	Estimate
Initial setup (visitor system, hot search, comments)	4-8 hours
User session cookie management	1-2 hours/week
Maintenance when Weibo changes endpoints	2-4 hours, every 2-3 months
No proxy needed for most endpoints	$0

Weibo is genuinely the easiest of the major Chinese platforms to scrape if you stay within visitor-system endpoints. RedNote and Bilibili both have more complex auth.

Approach 2: Use a hosted scraper

If you don't want to maintain visitor-system handling and cookie management, the zhorex/weibo-scraper Apify actor handles it.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

# Hot search (no cookie needed)
run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "hot_search",
    "maxResults": 50,
})

for topic in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"#{topic['rank']}: {topic['title']} (heat: {topic['hotValue']:,})")

Output format:

{
  "rank": 1,
  "title": "人工智能最新突破",
  "category": "科技",
  "hotValue": 2847562,
  "labelName": "热",
  "isHot": true,
  "url": "https://s.weibo.com/weibo?q=...",
  "scrapedAt": "2026-04-25T12:00:00Z"
}

For brand monitoring, search mode is what you want — though note the search-vs-cookie tradeoff:

# Without cookie: returns hot timeline as fallback
run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "CeraVe",
    "maxResults": 50,
})

# With cookie: returns true keyword-matched results
run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "CeraVe",
    "maxResults": 50,
    "cookieString": "SUB=your_logged_in_cookie",
})

The hosted actor handles the visitor system, exponential backoff, and rate limit recovery internally. Pricing: $5 per 1,000 results.

Honest stats on the actor right now: 4 paying users, 11 free-tier users, 92.5% success rate, 3,768 result extractions to date. Average issue response time when something breaks: under a few hours.

When DIY vs hosted

DIY makes sense when:

You're processing > 1M posts/month (per-result cost adds up)
You have ops capacity to refresh SUB cookies regularly
You need to scrape behind login at scale
You have specific endpoints not covered by hosted scrapers

Hosted makes sense when:

You don't have a dedicated scraper engineer
Volume is moderate (< 500k posts/month)
You want the visitor-system handling to be someone else's problem
You're prototyping and want to validate the use case before committing

What you do with the data downstream

Sentiment analysis on Chinese text is the obvious next layer. Off-the-shelf Chinese BERT models work reasonably for Weibo's discourse style — Weibo posts tend to be more formal than RedNote slang, so general Chinese sentiment models accuracy is higher (typical 75-85% on neutral/positive/negative classification).

For brand crisis detection, the signal you usually want is *velocity

Coin-per-view: the Bilibili metric that beats subscriber count for creator vetting

Sami — Wed, 29 Apr 2026 17:48:32 +0000

If you've ever sponsored a YouTube creator and been disappointed by the ROI, you've already lived through what subscriber count actually measures: not engagement, not influence, not purchase intent. Just historical clicks on a follow button. Many of those followers stopped opening videos two years ago. Some are inactive accounts. Some followed for a single piece of content that has nothing to do with your brand.

This is universally true on creator platforms, but it's especially true on Bilibili — China's YouTube. With 300M+ monthly active users skewed Gen Z and millennials, Bilibili is where Chinese creator marketing happens. And Bilibili exposes three engagement signals that YouTube doesn't, which together let you cut through the noise of follower counts and identify creators whose audiences actually engage.

The single most useful one is coin-per-view ratio. This post explains what it is, why it matters, what threshold to use, and how to compute it for any Chinese creator in a few lines of code.

Why follower count is a lying signal

Three reasons follower counts mislead in creator marketing:

1. Followers are a lagging indicator. Someone followed a creator in 2023 because they liked one video. That doesn't tell you whether they still watch in 2026, whether they engage, or whether they trust the creator's recommendations enough to buy.

2. Followers are gameable. Not everyone games them, but enough creators do that you can't trust raw counts without other signals. Bot followers, follow-for-follow campaigns, paid follower services. China specifically has a robust market for these.

3. The follower-to-engagement ratio varies wildly. A creator with 100k followers and 1M average views per video has fundamentally different audience economics than another creator with 100k followers and 5k average views per video. Both have the same "follower count" — the engagement quality is the actual signal.

This is why every serious creator marketing tool talks about "engagement rate" — which on YouTube is usually computed as (likes + comments) / views. It's better than raw follower count, but on Bilibili you can do meaningfully better.

The three Bilibili-native metrics

Bilibili was designed by anime fans for anime fans, and the engagement system reflects values around quality and creator support that YouTube's flat "like" button never captured. Three metrics that come back from any Bilibili video scrape:

Danmaku (弹幕) — real-time scrolling comments overlaid on the video as users watch. Think livestream chat, but for pre-recorded video. The danmaku count tells you how many people were engaged enough mid-watch to type something. It's a leading indicator of viewing time and attention.

Favorites (收藏) — equivalent to "save for later" or YouTube's bookmark. Strong long-term value signal: high favorites relative to views means people return to this video. Tutorials, references, and definitive content score high here.

Coins (投币) — Bilibili's tipping system. This is the interesting one. Each user gets a small daily allocation of coins (typically 5 per day for active users), and they can "throw" them at videos they want to support. Because coins are scarce by design — you only have a few to spend, ever — coin counts are a strong genuine-appreciation signal.

A user gives a coin to a video they love. They give a coin to a creator they want to keep making content. They don't give a coin to a video they passively watched and forgot. The cost is real (relative to the user's daily allocation), so the signal is real.

Coin-per-view ratio: the single best signal

If I had to pick one metric to evaluate a Bilibili creator, it would be median coin-per-view ratio across their last 20-30 videos.

The math is simple:

coin_per_view = coin_count / view_count * 100  # express as percentage

What I've found from looking at hundreds of Bilibili creators across categories:

Coin/View %	Audience quality
< 0.5%	Passive viewers. Casual scrolling traffic, not engaged.
0.5% – 1%	Average. Normal Bilibili content, decent audience.
1% – 2%	Strong. Genuinely engaged audience. Worth sponsoring.
> 2%	Exceptional. Users actively spending limited resources on this content.

Above 2% is rare. It typically indicates either: (a) genuinely high-quality educational/tutorial content that people return to, (b) a creator with a deeply loyal niche audience, or (c) content that struck a strong emotional/cultural nerve.

For creator vetting, my heuristic is: if median coin-per-view is below 1%, the audience is more passive than the follower count suggests; sponsorship ROI will probably disappoint.

How to compute this for any creator

The data you need: a creator's recent videos with their view and coin counts. Bilibili exposes this through their public API — no auth required. You can use the open-source bilibili-api Python library, or call their /x/space/wbi/arc/search endpoint directly.

If you'd rather skip the API integration entirely, I built a hosted scraper on Apify Store: zhorex/bilibili-scraper. $5 per 1,000 results, free tier covers ~1,000 results.

from apify_client import ApifyClient
from statistics import median

client = ApifyClient("YOUR_APIFY_API_TOKEN")

# Get a creator's last 30 videos
# user_id (mid) is the number in their profile URL: space.bilibili.com/{mid}
run = client.actor("zhorex/bilibili-scraper").call(run_input={
    "mode": "user_videos",
    "userIds": ["546195"],   # 老番茄 (a well-known Bilibili gamer)
    "maxResults": 30,
})

videos = []
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("type") == "video":
        videos.append(item)

# Compute coin-per-view ratio per video
ratios = []
for v in videos:
    views = v.get("viewCount", 0)
    if views < 1000:  # skip videos with too few views to be meaningful
        continue
    coins = v.get("coinCount", 0)
    ratios.append(coins / views * 100)

print(f"Videos analyzed: {len(ratios)}")
print(f"Median coin-per-view: {median(ratios):.2f}%")
print(f"Best video coin-per-view: {max(ratios):.2f}%")

# Categorize
median_ratio = median(ratios)
if median_ratio >= 2.0:
    label = "EXCEPTIONAL"
elif median_ratio >= 1.0:
    label = "STRONG"
elif median_ratio >= 0.5:
    label = "AVERAGE"
else:
    label = "PASSIVE"

print(f"Audience quality: {label}")

Run this against any Bilibili creator's user ID and you have a concrete answer about audience engagement quality.

A workflow for vetting creators at scale

If you're building a creator marketing program for the Chinese market, the workflow that works for the teams I've seen using this:

Gather candidates. From competitor sponsorship lists, from category trending, or from agency recommendations. Aim for 30-50 candidates per round.
Pull their recent video portfolios. Use user_videos mode to get the last 20-30 videos per creator.
Compute aggregate metrics. For each creator: median coin-per-view, median favorite-per-view, median danmaku-per-view, view consistency (standard deviation).
Filter on quality threshold. Drop anyone with median coin-per-view below 1%. This usually cuts the candidate list by 40-60%.
Manual review of the survivors. Watch a sample of their videos. Check for content fit. Evaluate sponsorship history (do their sponsored posts feel native or forced?).
Negotiate from the qualified shortlist.

Total cost using a hosted scraper: ~$5-10 in scraping for a 50-creator vetting round. Compared to agency rates for the same work ($500-2000 per round), the math is obvious once you do it.

Cross-platform creator vetting

Bilibili is not the whole story. If you're vetting creators for a comprehensive China presence:

Bilibili for video content (gaming, tech, anime, education)
RedNote (Xiaohongshu) for product-discovery content (beauty, fashion, lifestyle, food)
Weibo for public discourse and broad reach campaigns

Each platform has different engagement signals. Bilibili has coins; RedNote has saves (similarly scarce intent-to-buy signal); Weibo has reposts and verified-account hierarchy. A creator strong on one isn't necessarily strong on others.

I maintain scrapers for all three on Apify Store under the zhorex profile, with consistent output schemas across the suite. Same pricing model ($5/1000 results), same Apify infrastructure. If you're doing cross-platform creator analytics, the consistency saves integration time.

When this approach fails

Two cases where coin-per-view ratio is a misleading signal:

1. Brand-new creators with very few videos. If a creator has uploaded 3 videos and one went viral with high coins, the ratio looks artificial. Wait until you have 15-20 videos to compute median.

2. Live-stream-focused creators. Bilibili lets creators upload archived live streams. Coin economics are different in livestream context (gifts replace coins). For livestream-heavy creators, you need different analysis.

For everyone else, coin-per-view ratio is the single best signal I've found for vetting Bilibili creator quality at scale.

What this won't tell you

Whether the audience is geographically right for your campaign (need follower demographics, which require auth)
Whether the creator has done sponsorships before that flopped (need to scrape their content for promo patterns)
Whether their audience overlaps with your target customer profile (need cross-reference with other platforms)

Treat coin-per-view as the engagement-quality filter. Everything else still requires manual review or additional data sources.

If you're working on creator marketing for the Chinese market and want to compare notes on what works — drop a comment. I write about Chinese platform analytics (Bilibili, RedNote, Weibo) and the build-vs-buy tradeoffs around them.

Hosted Bilibili scraper: apify.com/zhorex/bilibili-scraper
Other Chinese platform scrapers in the same suite: RedNote for product-discovery content, Weibo for public discourse.

The easiest Chinese platform to scrape in Python in 2026: Bilibili in under 30 lines

Sami — Wed, 29 Apr 2026 17:41:42 +0000

If you've ever tried to scrape RedNote (Xiaohongshu), you know the pain. Request signing that rotates monthly, TLS fingerprinting that blocks requests immediately, residential proxies required, the whole tour. Weibo isn't quite as bad but you still need the Sina Visitor System dance to even hit a public endpoint.

Bilibili is the outlier. No API key. No browser. No proxy. No request signing rotation worth worrying about. Pure HTTP. Runs in 256MB RAM.

If you only need to monitor one Chinese platform — say you're tracking a gaming brand launch, or doing creator research, or analyzing tech trends — Bilibili is where you should start. This post walks through how to scrape it from scratch in Python, what data you actually get, and when to switch from DIY to a hosted scraper.

Why Bilibili is unusually scrape-friendly

Bilibili (哔哩哔哩) is China's YouTube — 300M+ monthly active users, skewed Gen Z and millennials, dominant in anime, gaming, tech, and educational content. From a scraping perspective, three things make it different from RedNote and Weibo:

1. Internal HTTP APIs are mostly stable. Bilibili exposes JSON endpoints for search, video metadata, user info, popular/trending, and comments. Most don't require auth for public content. The endpoints change rarely (months between meaningful updates) compared to RedNote where signing rotates monthly.

2. No TLS fingerprinting. Plain httpx or requests works. You don't need curl_cffi or any Chrome impersonation library to get past anti-bot.

3. Generous rate limits from non-Chinese IPs. I've sustained 1-2 requests per second from a single datacenter IP without getting throttled. RedNote would have banned the IP within minutes at that rate.

There's one caveat I'll cover at the end: comments scraping is throttled from datacenter IPs. Everything else works fine.

A 30-line Bilibili scraper

Here's the smallest useful Bilibili scraper I'd actually run in production:

import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Referer": "https://www.bilibili.com/",
}


def get_video_detail(bvid: str) -> dict:
    """Fetch full metadata for a Bilibili video by BVID."""
    url = "https://api.bilibili.com/x/web-interface/view"
    response = httpx.get(url, params={"bvid": bvid}, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json().get("data", {})


def get_popular(category_rid: int = 0, page_size: int = 20) -> list:
    """Fetch trending videos. category_rid=0 means all categories."""
    url = "https://api.bilibili.com/x/web-interface/popular"
    params = {"ps": page_size}
    if category_rid:
        params["rid"] = category_rid
    response = httpx.get(url, params=params, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json().get("data", {}).get("list", [])


# Use it
trending = get_popular(page_size=10)
for video in trending:
    stat = video.get("stat", {})
    print(f"{video.get('title')[:60]}")
    print(f"  Views: {stat.get('view'):,} | Coins: {stat.get('coin'):,} | "
          f"Favorites: {stat.get('favorite'):,} | Danmaku: {stat.get('danmaku'):,}")

That's it. No auth, no proxy, no signing. You can paste this into a Python REPL and have working data in 5 seconds.

The five things you can scrape

Bilibili's web client exposes endpoints for five modes. Each is useful for different use cases:

Mode	Endpoint	Use case
Search	`/x/web-interface/wbi/search/type`	Find videos by keyword (Chinese or English). Filterable by sort order, duration, date range.
Video detail	`/x/web-interface/view`	Full metadata for a specific video including all engagement metrics
Comments	`/x/v2/reply/main`	Comments on a video (caveats below)
User videos	`/x/space/wbi/arc/search`	All recent uploads from a creator
Popular	`/x/web-interface/popular`	Trending feed, optionally filtered by category

The search endpoint requires a small WBI signing scheme — slightly more involved than the others. Open-source library bilibili-api on GitHub handles it if you don't want to reverse-engineer it yourself. For everything else, plain HTTP works.

What data you actually get back

Each video from Bilibili comes with three categories of engagement metrics:

Standard metrics (similar to YouTube):

View count
Like count
Share count
Reply count (comments)

Bilibili-specific metrics (these don't exist on YouTube):

Danmaku count (弹幕) — the count of real-time scrolling comments overlaid on the video as users watch
Coin count (投币) — Bilibili's tipping system. Users get a few coins per day and "throw" them at videos. Coins are scarce by design.
Favorite count (收藏) — equivalent to "save for later"

These extra metrics are non-trivial. A video with 1M views, 50k coins, 30k favorites is meaningfully different from 1M views, 1k coins, 5k favorites — even with identical view counts. The first has an audience that's actively engaged enough to spend their daily coin allocation; the second has passive viewers.

If you're doing creator analytics or content strategy, the engagement quality signals coins/favorites give you are worth the integration effort.

The comments caveat

There's one Bilibili-side restriction worth knowing: /x/v2/reply/main is throttled when called from datacenter IPs (AWS, GCP, Azure, etc.). You'll get the top ~3 pinned comments per video and then nothing. Full pagination requires either:

Authenticated session cookies, or
Residential IPs

If you need full comments at scale, this is the one part where you'll run into infrastructure cost. Other modes are unaffected.

Performance you can expect

Real numbers from running this on Apify's serverless infrastructure (256MB RAM, no proxy, no auth):

Mode	Input	Duration	Throughput
Search	max=50	~7-8 seconds	~7 items/sec
Popular	max=40	~5 seconds	~8 items/sec
Video detail	10 BVIDs	~5 seconds	~2 items/sec (with tag enrichment)
User videos	3 users, max=15	~4 seconds	~4 items/sec

Fast enough for monitoring use cases. For heavy bulk extraction (millions of videos) you'd want to parallelize across multiple workers — easy because there's no IP throttling on most endpoints.

DIY vs hosted

DIY is genuinely viable for Bilibili. The maintenance burden is low — endpoints don't change much, there's no signing rotation eating your time. If you only need Bilibili and you're already comfortable with Python HTTP, build it yourself.

But if you're already monitoring multiple Chinese platforms (RedNote + Weibo + Bilibili is the typical full-suite use case), a hosted scraper that handles all three with consistent output schema is worth the convenience. I built and maintain one on Apify Store: zhorex/bilibili-scraper. Honest current state:

16 users
9 monthly active
100% success rate over 2,635 result extractions
Average issue response: 4.4 hours
$5 per 1,000 results (Apify free tier covers ~1,000/month at no cost)

It supports all five modes above with consistent output JSON, handles WBI signing for search internally, and is paired with companion scrapers for the other Chinese platforms — RedNote (Xiaohongshu) and Weibo. Same pricing model across the suite.

Using it from Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run = client.actor("zhorex/bilibili-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "AI tutorial",
    "sortOrder": "click",       # most-viewed first
    "durationFilter": "medium", # 10-30 min videos
    "maxResults": 50,
})

for video in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{video['title']} — {video['viewCount']:,} views, {video['coinCount']:,} coins")

When to use Bilibili specifically

Even if you're already covering YouTube, Twitter, etc., Bilibili captures things those platforms don't:

Chinese gaming and esports content — game launches, walkthroughs, and esports event reactions live here
Tutorial and educational content — Knowledge category is huge; replaces YouTube tutorials for Chinese audiences
Anime and otaku culture — central hub for the Chinese anime community
Creator economy in China — the coin/favorite metrics give better creator quality signals than subscriber counts (more on that in a follow-up post)

If your use case touches Chinese-market gaming, anime, tech, or education, Bilibili is irreplaceable.

FAQ

Is scraping Bilibili legal?

Public-data scraping legality varies by jurisdiction. Bilibili's ToS prohibits automated access. The scraping approach treats public web pages as accessible (the same content any logged-out browser visitor can see). Consult legal counsel for your specific use case. Not legal advice.

Do I need a Chinese IP?

For most endpoints, no. Bilibili is globally accessible. The exception is comments scraping (covered above). Some licensed video content (anime, dramas) may be geo-restricted but metadata and engagement metrics are accessible from any IP.

What's a BVID?

Bilibili's video ID format. Looks like BV1YXDfBUETP. Replaced the older numeric aid format. URLs use BVIDs: bilibili.com/video/BV1YXDfBUETP.

How does this compare to YouTube Data API?

YouTube has standardized API access for verified channel owners with quota limits. Bilibili has no equivalent for international developers. For Chinese-market analytics, scraping is the only practical option. The upside: more granular engagement signals (coins, favorites, danmaku) than YouTube exposes.

Why did bilibili-api (the Python lib) work for me a year ago and break now?

Probably the WBI signing scheme rotated. Check the library's commit history; if there's been a recent fix, update. If the project looks abandoned, consider switching libraries or going hosted.

If you're working on Chinese-market analytics, brand monitoring, or creator research and want to compare notes — drop a comment. I write about the build-vs-buy tradeoffs for Chinese platform scraping (RedNote, Weibo, Bilibili).

Hosted actor: apify.com/zhorex/bilibili-scraper

How to scrape RedNote (Xiaohongshu) with Python in 2026 — the auth/signing problem and how to handle it

Sami — Sat, 25 Apr 2026 01:13:23 +0000

RedNote (Xiaohongshu, 小红书, sometimes "Little Red Book" or just XHS) is the platform a lot of Western teams realized they needed to monitor in 2024-2025, when the TikTok regulatory mess in the US sent millions of users — and brand attention — toward Chinese platforms. It's now China's #1 lifestyle and product-discovery network, with 300M+ monthly active users and a search-driven discovery model that makes it different from every other Chinese social platform.

The problem: there's no official public API. Western teams who try to monitor it usually end up either (a) paying enterprise vendors $20-50k/year for limited China coverage, or (b) trying to scrape it themselves and discovering that RedNote has one of the more aggressive anti-scraping stacks in Chinese social.

This article walks through the actual technical challenges and shows you both DIY and hosted approaches with real Python code. I've shipped a hosted RedNote scraper on Apify that I'll mention later — but the goal here is for you to understand the problem space well enough to make an informed build-vs-buy decision, not to sell you anything.

What RedNote actually serves

Before we go technical: what data does RedNote expose, and what's actually useful?

A RedNote post is structured roughly like this:

Title (often very short, sometimes empty)
Body text — long-form description with product mentions, hashtags, location tags
Image carousel — 1-9 images. Critical: a non-trivial portion of product info lives in image text overlays, not in the body
Engagement metrics — likes, saves, comments, shares
Author profile — username, avatar, follower/following counts, bio, verification, location
Tags / categories — hashtags and platform-assigned categories

For most monitoring use cases, the metric that matters more than likes is saves. Saves on RedNote are the closest equivalent to "I want to buy this later" — they correlate with purchase intent. Likes on RedNote are casual engagement, similar to Twitter likes.

Profile data is structured similarly:

User ID, RedNote ID (red ID), nickname, avatar
Bio / description
Follower / following counts
Location, gender, profile tags
Total likes received across all posts
Verification status

The technical challenges (why this is harder than scraping Twitter)

If you've scraped Western social platforms, your default toolkit is probably httpx or requests plus maybe a residential proxy. RedNote is going to break each of those defaults.

Challenge 1: TLS fingerprinting

RedNote uses TLS fingerprinting (specifically JA3/JA4) to identify and block requests that don't come from real browsers. The requests library has a Python-specific TLS fingerprint that RedNote's bot-detection layer recognizes immediately.

The standard fix is to use curl_cffi, which lets you spoof a Chrome or Safari TLS fingerprint:

from curl_cffi import requests as curl_requests

# Spoof Chrome 120's TLS fingerprint
response = curl_requests.get(
    "https://www.xiaohongshu.com/explore",
    impersonate="chrome120"
)

This alone gets you past the first layer of detection.

Challenge 2: Request signing

RedNote signs every API request with a value called x-s (sometimes seen as xs) plus other parameters like x-t and x-s-common. These are computed client-side from a JavaScript function in their web app.

The signing function changes roughly monthly. When it changes, every scraper using the old signing logic breaks until someone reverse-engineers the new function.

Here's roughly what you need to do:

# Pseudo-code — actual signing logic is more complex

import time
import json
import hashlib

def generate_signing_headers(url_path: str, body: dict) -> dict:
    """
    The actual logic is reverse-engineered from RedNote's web client JS.
    This requires reading their obfuscated bundle and reproducing it in Python.
    """
    timestamp = str(int(time.time() * 1000))
    body_str = json.dumps(body, separators=(',', ':')) if body else ""

    # Real implementation involves:
    # - Specific input concatenation order
    # - Custom hashing scheme (not standard HMAC)
    # - Several "magic constants" that change when RedNote rotates
    # - Sometimes a captcha-derived token

    raw = f"url={url_path}&data={body_str}&t={timestamp}"
    x_s = hashlib.md5(raw.encode()).hexdigest()  # This is NOT the actual algorithm

    return {
        "x-s": x_s,
        "x-t": timestamp,
        # x-s-common is computed separately
    }

The actual signing algorithm is more complex than what I've shown. There are open-source libraries that have reverse-engineered it (xhs-api and similar on GitHub) — they get you most of the way there, but expect to patch them when RedNote rotates.

Challenge 3: IP-level rate limiting and datacenter blocking

RedNote blocks requests from datacenter IPs (AWS, GCP, Azure, DigitalOcean, etc.) within minutes. You need residential proxies, ideally with Chinese geolocation or at least Asia-Pacific.

Even with residential IPs, there's a per-IP rate limit. Realistic throughput is around 10-20 requests per minute per IP before you start getting 412/418 errors and eventually IP bans.

Challenge 4: SPA / dynamic rendering for some endpoints

Search and the explore feed are loaded via AJAX after initial page load, but a few endpoints (some user pages, certain post types) only render their data in the Vue.js application state. You either need to extract data from the inline <script> tag (look for window.__INITIAL_STATE__) or render with Playwright.

Challenge 5: Login walls on certain features

True keyword-filtered search requires login. Without login, you get the explore feed (trending/recommended for your keyword), which is useful but not the same. This is a structural product limitation, not a scraping limitation — you can scrape it the same way logged-in users see it, you just need to either provide cookies or accept the explore-feed fallback.

Approach 1: Build it yourself

If you have ops capacity to maintain it (someone who can read JavaScript and reverse-engineer signing functions monthly), DIY is feasible. Here's a minimal example using curl_cffi plus an open-source signing library.

First install:

pip install curl_cffi xhs

Note: xhs is one of several open-source libraries on GitHub that wrap RedNote's API. Check their commit history before depending on one — the abandoned ones break monthly.

from xhs import XhsClient
from xhs.exceptions import DataFetchError

# You need to provide your own cookies and signing function URL
# The 'sign' function comes from the library's reverse-engineered JS
def sign(uri, data=None, a1="", web_session=""):
    # Implementation provided by the library
    # When RedNote rotates, you'll need to update this
    pass

client = XhsClient(
    cookie="abRequestId=...; webBuild=...; xsecappid=xhs-pc-web; ...",
    sign=sign
)

try:
    # Get a user's posts
    user_id = "5cfbc3f10000000018023ebb"
    posts = client.get_user_notes(user_id)

    for post in posts.get("notes", []):
        print(f"Title: {post.get('display_title')}")
        print(f"Likes: {post.get('interact_info', {}).get('liked_count')}")
        print("---")

except DataFetchError as e:
    print(f"RedNote rejected the request: {e}")
    # Common causes:
    # - Signing function out of date (update from upstream)
    # - Cookie expired (re-login)
    # - IP throttled (rotate proxy)

The cookie you need comes from logging into RedNote in a browser and copying the relevant cookies from DevTools. The cookies expire — typically a few days — so you'll need to refresh them periodically.

Here's the honest cost breakdown for DIY:

Cost component	Estimate
Initial setup (researching libraries, getting first scrape working)	8-16 hours
Residential proxy (Bright Data, Oxylabs, etc.)	$50-200/month for moderate volume
Per-incident maintenance when RedNote rotates	4-8 hours, 1-2x/month
Ongoing: cookie refresh, error handling	1-2 hours/week

If you have a developer whose time is worth $50-100/hour, DIY is around $400-1000/month all-in for moderate scraping volumes.

Approach 2: Use a hosted scraper

The build-vs-buy math changes if you don't have someone on the team who can read reverse-engineered JavaScript and patch signing logic. Hosted Apify Actors handle that for you.

Several developers (including me) maintain RedNote scrapers on Apify. Mine is:

apify.com/zhorex/rednote-xiaohongshu-scraper
$5 per 1,000 results
14 paying users currently, 38 on free tier
Average issue response time: 1.6 hours
88.8% success rate (the gap is mostly RedNote-side transient errors)

Using it from Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

# Search
run = client.actor("zhorex/rednote-xiaohongshu-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "skincare routine",
    "maxResults": 50,
    "filterByMinLikes": 100  # Only return posts with 100+ likes
})

# Iterate over results
for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Title: {post['title']}")
    print(f"Likes: {post['likes']}")
    print(f"Author: {post['author']['nickname']}")
    print(f"URL: {post['postUrl']}")
    print("---")

The output JSON is flat:

{
  "mode": "search",
  "postId": "69d269310000000023017e07",
  "postUrl": "https://www.xiaohongshu.com/explore/69d269310000000023017e07",
  "type": "normal",
  "title": "Morning skincare routine for dry skin",
  "images": ["https://sns-webpic-qc.xhscdn.com/..."],
  "likes": 15234,
  "author": {
    "userId": "575d32285e87e733f0162c0a",
    "nickname": "BeautyQueen",
    "avatar": "https://sns-avatar-qc.xhscdn.com/..."
  },
  "scrapedAt": "2026-04-25T21:14:30Z"
}

This is one option among several on Apify Store. EasyApi has the most users by volume; OrbitData Labs has a different all-in-one approach. Pricing is roughly the same across them ($5/1000 ± $1). Differences are in:

Output schema (some return RedNote's raw nested API response, some flatten it)
Update frequency (some are abandoned and break for weeks at a time)
Mode coverage (some only do search; others handle profiles, comments, videos, etc.)
Issue response time

If you're evaluating, run a free-tier test on 2-3 of them with the same input and compare what you get back. The free tier costs you nothing.

When does each approach make sense?

DIY (build it yourself):

You have a Chinese-language ops team and can monitor breakages
You're processing > 1M posts/month (the per-result cost of hosted starts to add up)
You need to scrape behind login (which means you need cookies from logged-in accounts you control)
You have specific data needs that no hosted scraper covers

Hosted Apify actor:

You don't have a dedicated scraper engineer
Volume is variable or moderate (< 500k posts/month)
You want to outsource the cat-and-mouse with RedNote's anti-bot updates
You're prototyping and want to validate the approach before committing

The middle ground that often makes sense: use a hosted actor for production data flow, build a thin DIY layer for any specific endpoints the hosted version doesn't cover. The hosted scraper handles the maintenance burden on the parts that break most often (search, profile, posts), and you keep custom DIY logic for the edges.

What you do with the data downstream

Scraping is one third of the problem. The other two:

Sentiment analysis on Chinese text. Off-the-shelf Chinese BERT models (like bert-base-chinese from Huggingface) are a starting point but accuracy varies wildly by domain. RedNote slang, in particular, doesn't appear in the training data of general Chinese sentiment models — fine-tuning on RedNote-specific labeled samples gets you significant accuracy lift if accuracy matters.

Image text extraction. A non-trivial portion of product mentions on RedNote live in image text overlays (Chinese users frequently put product names visible in images, not in the post body). PaddleOCR is the open-source standard for Chinese OCR. Slow (~30 seconds per image) but reliable. Adds significant cost to processing pipelines but you'll miss a measurable percentage of product mentions without it.

Both of these are downstream of scraping — solve scraping first, then layer.

FAQ

Is scraping RedNote legal?

Public-data scraping legality varies by jurisdiction. RedNote's ToS prohibits automated access (as do most platform ToS). The Apify approach (and most public-scraping infrastructure) treats public web pages as accessible, the same way Google's crawler would. You should consult legal counsel for your specific use case. Not legal advice.

How fast can I scrape RedNote?

Realistic sustained throughput per IP is around 10-20 requests per minute before triggering rate limits. With residential proxy rotation and proper backoff, you can scale this horizontally — Apify's actor handles this internally. For DIY, plan for ~1 result per second per IP as a conservative number.

Do I need a Chinese IP?

Not strictly required, but residential IPs (Asian residential preferred) have notably higher success rates than US/European residential. Datacenter IPs are blocked outright.

What's xsec_token and why does it matter?

When users share posts via the RedNote app or copy URLs, those URLs include an xsec_token query parameter that authenticates the link request. Some scrapers don't handle URLs with xsec_token correctly and return errors. If you're scraping URLs collected from real users, make sure your tooling supports this.

Can I scrape video files from RedNote posts?

Yes. Video URLs are returned in the post metadata. Direct download from those URLs works without authentication for public videos.

How often does the request signing change?

Roughly monthly, sometimes more frequently around major RedNote app updates. If you're doing DIY, plan to dedicate 4-8 hours per rotation to update your signing function, or rely on an actively maintained library that pushes updates fast.

What's the difference between RedNote and Xiaohongshu?

They're the same platform. "Xiaohongshu" (小红书) is the Chinese name and means "Little Red Book". "RedNote" is the English brand they pushed during the 2024-2025 TikTok migration to be more accessible to global users. Same app, same data, same API endpoints.

If you're working on China market intelligence, brand monitoring, or competitive research and want to compare notes — drop me a comment. I write about Chinese platform scraping (Weibo, Bilibili, RedNote) and the build-vs-buy trade-offs around them.

The actor I mentioned: apify.com/zhorex/rednote-xiaohongshu-scraper. Free tier covers ~1,000 results, which is enough to validate against your specific use case before committing.

Monitoring the Chinese Social Media Ecosystem: RedNote, Weibo & Bilibili Data Pipeline

Sami — Thu, 23 Apr 2026 14:56:48 +0000

If you are doing market research, brand monitoring, competitive intelligence, or academic research on China, no single platform tells the whole story. Chinese internet users spread across specialized platforms the way Western users split between Twitter, Instagram, YouTube, and Reddit — but the Chinese platforms are larger, more fragmented, and harder to access from outside the country.

This article maps the three most important Chinese social platforms for data collection, explains what each one covers, and shows how to build a unified monitoring pipeline using three Apify Actors that share a common architecture: no browser, no proxy, no API keys.

The Three Pillars of Chinese Social Media Intelligence

1. RedNote / Xiaohongshu (小红书) — Social Commerce & Lifestyle

What it is: China's Instagram meets Pinterest. A social commerce platform where users share product reviews, lifestyle content, travel diaries, and beauty routines. Known for its influence on Chinese consumer purchasing decisions.

User base: 200M+ monthly active users

Why it matters for research:

Product sentiment and unfiltered consumer reviews
Trend detection in beauty, fashion, food, travel, and lifestyle
Influencer (KOL/KOC) discovery for Chinese market campaigns
Brand perception monitoring among Chinese consumers

Actor: zhorex/rednote-scraper

Modes: Search posts, user profiles, post comments, trending topics

2. Weibo (微博) — Microblogging & Public Opinion

What it is: China's Twitter. The platform where Chinese public opinion forms in real-time. Breaking news, celebrity drama, government communications, and brand PR all happen on Weibo.

User base: 580M+ monthly active users

Why it matters for research:

Real-time trending topics with heat scores and categories
Public opinion tracking on policy, brands, and international events
Celebrity and KOL influence measurement
Crisis monitoring and brand sentiment during PR events

Actor: zhorex/weibo-scraper

Modes: Hot search/trending, post comments, keyword search, user posts

Auth note: Trending topics and post comments work without login. Search and user posts require a Weibo SUB cookie (easily obtained from a browser session).

3. Bilibili (哔哩哔哩) — Video & Creator Analytics

What it is: China's YouTube, with a strong focus on anime, gaming, tech education, and Gen Z culture. Known for its unique danmaku (弹幕) system — scrolling comments that overlay the video in real-time.

User base: 300M+ monthly active users

Why it matters for research:

Creator analytics with Bilibili-specific metrics (danmaku, coins, favorites)
Gaming and anime industry monitoring
Chinese Gen Z content trends and preferences
Tech and education content analysis

Actor: zhorex/bilibili-scraper

Modes: Search videos, video details, video comments, user/creator videos, popular/trending

Platform Comparison

Dimension	RedNote	Weibo	Bilibili
Platform type	Social commerce	Microblogging	Video
Primary audience	Women 18-35, consumers	All demographics	Gen Z, gamers, students
Content format	Photos + short text	Short posts (tweets)	Long-form video
Best for	Product sentiment, lifestyle trends	Breaking news, public opinion	Creator analytics, youth culture
Unique data	Purchase intent, product reviews	Hot search rankings, heat scores	Danmaku counts, coin tipping
Auth required	No	Partial (trending: no, search: yes)	No
MAU	200M+	580M+	300M+
Actor	`rednote-scraper`	`weibo-scraper`	`bilibili-scraper`

Building a Unified Pipeline

All three actors share common design principles:

Pure HTTP — no browser needed, minimal compute
256MB RAM — cheap to run
Pay-per-event — $5 per 1,000 items for all three
JSON/CSV/Excel output — same export formats
Apify integrations — Google Sheets, S3, webhooks, Zapier, Make, n8n

Example: Daily China Brand Monitor

Here is a practical pipeline architecture using all three actors:

Step 1: Schedule daily runs for each actor via Apify Schedules.

RedNote — search for your brand name in Chinese:

{
  "mode": "search",
  "searchQuery": "你的品牌名",
  "maxResults": 50
}

Weibo — monitor trending mentions:

{
  "mode": "hot_search",
  "maxResults": 50
}

Bilibili — track video mentions:

{
  "mode": "search",
  "searchQuery": "你的品牌名",
  "maxResults": 50
}

Step 2: Use Apify webhooks to push results to your data warehouse (S3 → Snowflake/BigQuery) or directly to Google Sheets for a lightweight dashboard.

Step 3: Set up alerting — use Zapier or Make to send Slack notifications when specific keywords appear in trending topics or when engagement spikes on brand-related content.

Use Case Examples

Brand Entering the Chinese Market

A Western consumer brand preparing to launch in China runs all three actors to understand:

RedNote: What do Chinese consumers say about the product category? What do they value? Which competitor products are reviewed positively?
Weibo: Is the brand already discussed in Chinese media? What is the general sentiment?
Bilibili: Are there video reviews or unboxing content for the brand or competitors?

Academic Research on Chinese Digital Culture

A researcher studying Chinese Gen Z media consumption uses:

Bilibili: Trending video content, danmaku engagement patterns, creator growth trajectories
Weibo: Public discourse topics, hashtag trends over time
RedNote: Lifestyle and consumption trends, product preference signals

PR Crisis Monitoring

A multinational company monitors all three platforms for brand mentions after a negative news cycle:

Weibo: Real-time trending topics and comment sentiment (the fastest-moving platform)
RedNote: Consumer reaction and product boycott signals
Bilibili: Video commentary and creator opinion pieces

Technical Architecture

All three actors share:

Feature	Detail
Runtime	Pure HTTP (no Playwright/Puppeteer)
Memory	256MB RAM
Proxy	Not required
API keys	Not required
Output	JSON, CSV, XLSX, JSONL
Rate limiting	Built-in backoff and retry
Language	Content in original Simplified Chinese
Translation	Pipe through Google Translate, DeepL, or Claude for English

Pricing

All three actors use the same pricing model:

Volume	Cost per actor
1,000 items	$5
10,000 items	$50
100,000 items	$500

Running all three actors daily with 50 items each = 150 items/day × 30 days = 4,500 items/month ≈ $22.50/month for comprehensive daily China monitoring across three platforms.

Apify's free plan includes $5 of monthly credits — enough to test all three actors.

FAQ

Do I need a VPN to scrape Chinese platforms?
No. All three actors use public HTTP endpoints that are globally accessible. No VPN or proxy is required.

Is the data in Chinese?
Yes. All content is returned in the original Simplified Chinese. For English translations, pipe the output through a translation service.

Can I combine data from all three actors?
Yes. All three output JSON with similar schemas. Use Apify's dataset API or export to a shared warehouse for unified analysis.

Is scraping Chinese social media legal?
These actors only access publicly available data through public web endpoints. No authentication is bypassed and no private data is accessed. Always review your local laws and each platform's terms of service.

Get Started

RedNote: apify.com/zhorex/rednote-scraper
Weibo: apify.com/zhorex/weibo-scraper
Bilibili: apify.com/zhorex/bilibili-scraper

Start with one platform, validate the data quality, then expand to all three for comprehensive Chinese social media intelligence.

Built by Zhorex — the only developer on Apify specializing in Chinese platform intelligence.

Automating Perplexity AI Searches for Content Research and Brand Monitoring

Sami — Thu, 23 Apr 2026 14:55:14 +0000

Perplexity AI has become one of the most popular AI-powered search engines, giving users synthesized answers with cited sources instead of a list of blue links. For marketers, content strategists, and SEO professionals, a new discipline has emerged: Answer Engine Optimization (AEO) — the practice of getting your brand mentioned and cited in AI-generated search results.

The challenge is that monitoring how Perplexity answers queries about your brand, competitors, or industry requires manually searching one query at a time. If you want to track visibility across dozens or hundreds of queries on a regular schedule, you need automation.

This article walks through how to automate Perplexity AI searches at scale using the zhorex/perplexity-ai-scraper Actor on Apify — no Perplexity API key required.

TL;DR

What: Extract AI-generated answers, cited sources, and related questions from Perplexity AI
How: zhorex/perplexity-ai-scraper on Apify
Cost: $0.02 per query ($20 per 1,000 queries)
No API key: Scrapes the public web interface directly via headless browser
Two modes: Full search results, or brand mention monitoring

Why Automate Perplexity Searches?

Answer Engine Optimization (AEO)

As more users shift from Google to AI search engines like Perplexity, ChatGPT, and Google AI Overviews, monitoring your visibility in AI-generated answers is becoming critical. AEO focuses on:

Is your brand mentioned when users ask about your product category?
What position does your brand appear in the AI answer?
Which competitors are mentioned alongside you?
What sources does Perplexity cite — and is your content among them?

Content Strategy

Perplexity curates answers from multiple sources and cites them. By analyzing which URLs get cited for your target keywords, you can identify content gaps: if your competitors are cited but you are not, that signals where to invest in content.

Competitive Intelligence

Track how Perplexity positions your competitors for your target keywords. Which brands does it recommend? How does it describe them? This gives you a direct view into how AI search engines perceive your market.

Two Modes of Operation

Mode 1: Search — Extract Full AI Answers

Submit any query and get the full AI-generated answer with all cited sources.

Input:

{
  "mode": "search",
  "queries": [
    "best CRM software for small business 2026",
    "top project management tools for startups"
  ]
}

Output:

{
  "query": "best CRM software for small business 2026",
  "answer": "For 2026, the best CRM overall is...",
  "sources": [
    {
      "position": 1,
      "title": "The Best CRM Software for 2026",
      "url": "https://www.pcmag.com/picks/the-best-crm-software",
      "domain": "pcmag.com",
      "snippet": "Customers are vital to any business..."
    }
  ],
  "relatedQuestions": [
    "Zoho CRM vs Salesforce which is better for small businesses",
    "Best free or low-cost CRM alternatives for startups 2026"
  ],
  "totalSources": 10,
  "answerLength": 1542,
  "searchUrl": "https://www.perplexity.ai/search?q=...",
  "scrapedAt": "2026-04-09T15:00:00+00:00"
}

Mode 2: Brand Monitor — Track Brand Mentions in AI Answers

Automatically detect whether your brand appears in Perplexity's answer for a set of queries, and see which competitors are mentioned.

Input:

{
  "mode": "brand_monitor",
  "brandName": "HubSpot",
  "queries": [
    "best CRM software 2026",
    "top email marketing platforms for ecommerce"
  ]
}

Output:

{
  "brand": "HubSpot",
  "query": "best CRM software 2026",
  "mentioned": true,
  "mentionContext": "HubSpot is the easiest all-around option for teams that want a simple start...",
  "position": "Mentioned early in the answer (top section)",
  "competitorsMentioned": ["Zoho CRM", "Salesforce", "Pipedrive"],
  "sourcesCount": 10,
  "scrapedAt": "2026-04-09T15:00:00+00:00"
}

Practical Use Cases

1. Weekly AEO Dashboard

Schedule the actor to run weekly with your core keyword list (e.g., "best [your category] software 2026"). Track over time whether your brand is being recommended, and whether your visibility is improving after content investments.

2. Competitor Monitoring

Run brand_monitor mode with your top competitor names. See which queries they appear for that you do not — those are opportunities.

3. Content Gap Analysis

For your target queries, look at which sources Perplexity cites. If your content is not cited, study what the cited sources cover that you do not. Use this to prioritize content creation.

4. Market Research at Scale

Need AI-curated answers to dozens of research questions? Use search mode with a list of queries and get structured, sourced answers in bulk — with full attribution for verification.

Python and JavaScript Integration

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("zhorex/perplexity-ai-scraper").call(run_input={
    "mode": "brand_monitor",
    "brandName": "YourBrand",
    "queries": ["best project management tools 2026"]
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["mentioned"], item["competitorsMentioned"])

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('zhorex/perplexity-ai-scraper').call({
    mode: 'search',
    queries: ['best CRM software 2026'],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].answer);

How It Works Under the Hood

The actor opens Perplexity.ai in a headless Chromium browser (Playwright), navigates to the search URL for each query, waits for the AI to finish generating its streamed answer, then extracts the full answer text, cited sources with URLs, and related questions. Results are pushed to a structured Apify dataset.

Performance:

~15-30 seconds per query (AI answer generation takes time)
1024 MB RAM recommended (Playwright + Chromium)
Sequential queries with 5s delay to avoid rate limiting

Perplexity API vs. This Actor

Aspect	Perplexity Sonar API	This Actor
API key required	Yes (paid subscription)	No
Cost	Sonar Pro pricing	$0.02 per query
Speed	Fast (direct API)	~15-30s per query
Sources/citations	Included	Included
Brand monitoring	Build it yourself	Built-in mode
Related questions	Not included	Included

The trade-off is speed: the API is faster, but this actor requires no API key or subscription.

Pricing

$0.02 per query ($20 per 1,000 queries). You can test with Apify's free tier which gives you $5 of monthly usage — enough for 250 queries.

FAQ

How is this different from the Perplexity API?
The official Perplexity API (Sonar) gives programmatic access but requires a paid API key. This Actor scrapes the free public web interface — same answers, same sources, no API key required. The trade-off is speed (~15-30s per query).

What is AEO?
Answer Engine Optimization is the practice of getting your brand mentioned in AI search results. As more users shift to AI search engines, monitoring your visibility in AI-generated answers is becoming as important as traditional SEO.

Can I schedule regular monitoring runs?
Yes. Use Apify Schedules to run the actor daily, weekly, or at any interval. Combine with webhooks to send Slack or email alerts when your brand visibility changes.

Are the answers always the same?
No. AI answers are non-deterministic — the same query can produce slightly different answers on different runs. This is a characteristic of AI search, not a limitation of the actor.

Get Started

https://apify.com/zhorex/perplexity-ai-scraper

Try a single query in search mode to see the output quality, then build out your keyword list for systematic monitoring.

Scraping Bilibili Videos and Creators for Market Research in 2026

Sami — Thu, 23 Apr 2026 14:53:41 +0000

Bilibili (哔哩哔哩) is China's premier video platform with 300M+ monthly active users. Think YouTube, but with a younger demographic (Gen Z and millennials), a thriving anime and gaming community, and a unique feature called danmaku (弹幕) — real-time scrolling comments that overlay the video as it plays.

For anyone doing market research on Chinese youth culture, gaming audiences, tech content, or creator economics, Bilibili is one of the richest data sources available. The problem is that there is no official public Bilibili API for international developers. Bilibili's internal APIs are undocumented, require Chinese phone verification, and change frequently.

This article shows how to extract Bilibili videos, comments, creator profiles, and trending content using the zhorex/bilibili-scraper Actor on Apify — no API key, no browser, no proxy required.

TL;DR

What: Extract videos, comments, creator profiles, and trending content from Bilibili
How: zhorex/bilibili-scraper on Apify — pure HTTP, 256MB RAM
Cost: $5 per 1,000 items scraped
Auth: None required — all endpoints are public
Unique data: Danmaku counts, coin counts (投币), favorites, plus standard engagement metrics

What Makes Bilibili Data Unique

Unlike YouTube or TikTok, Bilibili has platform-specific metrics that this actor captures:

Danmaku count (弹幕) — live scrolling comments overlaid on the video. High danmaku signals active community engagement, not just passive viewing
Coin count (投币) — Bilibili's tipping system where users "throw coins" at creators. A direct signal of audience appreciation
Favorite count (收藏) — equivalent to "save" on other platforms
Standard metrics: views, likes, shares, replies

These metrics together give a much richer picture of content performance than views alone.

Five Scraping Modes

1. Search Videos

Search by keyword (Chinese or English). Supports sort and filter options.

{
  "mode": "search",
  "searchQuery": "人工智能教程",
  "sortOrder": "pubdate",
  "maxResults": 50
}

Sort options: totalrank (relevance), click (most views), pubdate (newest), dm (most danmaku), stow (most favorites), scores (most comments).

Duration filters: short (<10min), `medium` (10-30min), `long` (30-60min), `verylong` (>60min).

2. Video Details

Full video info with all engagement metrics and tags.

{
  "mode": "video_detail",
  "videoUrls": [
    "https://www.bilibili.com/video/BV1GJ411x7h7",
    "BV1xx411c7mD"
  ]
}

3. Video Comments

Extract comments with author info and likes.

{
  "mode": "video_comments",
  "videoUrls": ["https://www.bilibili.com/video/BV1GJ411x7h7"],
  "maxComments": 50,
  "sortComments": "hot"
}

Sort options: hot (top/most-liked), time (newest), likes (by like count).

Note: Bilibili throttles comment pagination from datacenter IPs, returning only top/pinned comments. Full comment pagination requires residential IPs or authenticated sessions. Other modes are unaffected.

4. Creator/User Videos

Get user profile info plus their recent videos.

{
  "mode": "user_videos",
  "userIds": ["546195", "1340190821"],
  "maxResults": 30
}

Find a user's mid in their profile URL: space.bilibili.com/{mid}. Multiple users are processed in parallel (up to 3 concurrent).

5. Popular/Trending Videos

Trending videos, filterable by Bilibili category.

{
  "mode": "popular",
  "category": "game",
  "maxResults": 20
}

Available categories: Animation, Music, Dance, Gaming, Knowledge, Tech, Sports, Cars, Life, Food, Animal, Fashion, Entertainment.

Output Example

Video:

{
  "type": "video",
  "bvid": "BV1YXDfBUETP",
  "title": "Example Video Title",
  "url": "https://www.bilibili.com/video/BV1YXDfBUETP",
  "duration": 167,
  "durationFormatted": "2:47",
  "viewCount": 1570113,
  "likeCount": 182455,
  "coinCount": 110535,
  "favoriteCount": 63471,
  "shareCount": 45918,
  "danmakuCount": 7466,
  "replyCount": 17276,
  "authorName": "Creator Name",
  "authorMid": 1340190821,
  "publishDate": "2026-04-08T12:00:00+00:00",
  "category": "Gaming",
  "tags": ["anime", "action", "review"],
  "scrapedAt": "2026-04-10T10:00:00+00:00"
}

User Profile:

{
  "type": "user",
  "mid": 546195,
  "name": "老番茄",
  "fans": 20189060,
  "archiveCount": 652,
  "verified": false,
  "profileUrl": "https://space.bilibili.com/546195",
  "scrapedAt": "2026-04-10T10:00:00+00:00"
}

Use Cases for Market Research

Gaming/anime brand monitoring — Track game launches and anime reactions on China's largest anime community
Content trend analysis — Identify trending topics in Chinese youth culture and Gen Z interests
Creator evaluation — Analyze Bilibili KOLs (Key Opinion Leaders) for partnerships and sponsorships using follower counts, engagement ratios, and content frequency
Ad placement research — Understand which categories and content types perform best by danmaku density, coin rates, and view-to-engagement ratios
Academic research — Study Chinese digital culture, danmaku behavior, and content consumption patterns
Product launch monitoring — Track brand mentions and competitor content in China

Python and JavaScript Integration

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("zhorex/bilibili-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "人工智能教程",
    "maxResults": 50
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["viewCount"], item["danmakuCount"])

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('zhorex/bilibili-scraper').call({
    mode: 'search',
    searchQuery: '人工智能教程',
    maxResults: 50,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => console.log(item.title, item.viewCount));

Technical Details

No browser — pure HTTP requests to Bilibili's public APIs
No proxy — Bilibili is accessible globally (some licensed content may be geo-restricted)
No API key — all endpoints are public
256MB RAM — lightweight and efficient
Concurrent fetching — video_detail: up to 5 in parallel; user_videos and video_comments: up to 3 in parallel

Pricing

$0.005 per item scraped ($5 per 1,000 results). Start with Apify's free plan which includes $5 of monthly credits.

Part of the Chinese Digital Intelligence Suite

Bilibili covers video and creator analytics. For full China market coverage, combine with:

zhorex/weibo-scraper — Weibo microblogging, trending topics, public opinion (580M+ MAU)
zhorex/rednote-scraper — RedNote/Xiaohongshu social commerce, lifestyle content (200M+ MAU)

All actors: no browser, no proxy, no API keys. Built by Zhorex.

Get Started

https://apify.com/zhorex/bilibili-scraper

Try the popular mode first (no auth needed) to see what is trending on Chinese video right now.

How to Scrape Weibo Without Login in 2026: The Complete Guide

Sami — Thu, 23 Apr 2026 14:51:50 +0000

Weibo (微博) is China's dominant microblogging platform — think Twitter meets Instagram, with 580M+ monthly active users. It is where Chinese public opinion forms, brands communicate, celebrities post, and news breaks. Government officials, industry leaders, and major brands all maintain active Weibo accounts.

For anyone doing China market research, PR monitoring, influencer analysis, or geopolitical tracking, Weibo data is essential. The problem is that there is no official public Weibo API available for international developers. Weibo's developer platform requires a Chinese business license, imposes strict rate limits, and returns limited data.

This guide walks through how to extract Weibo posts, trending topics, comments, and creator profiles using the zhorex/weibo-scraper Actor on Apify — no API key, no browser, no VPN required.

TL;DR

What: Extract posts, trending topics, comments, and user profiles from Weibo
How: zhorex/weibo-scraper on Apify — pure HTTP, no browser needed
Cost: $5 per 1,000 items scraped
Auth: Trending topics and post comments work without any login. Search and user posts require a Weibo SUB cookie
No VPN needed: All endpoints are globally accessible

Why Weibo Data Matters

Who	Why
PR & Communications	Track brand mentions in real-time on China's public square
Market Research	Monitor what is trending among Chinese consumers
Influencer Marketing	Find and evaluate KOLs by followers, engagement, verification
Competitive Intelligence	Track Chinese competitor announcements and campaigns
Geopolitical Analysis	Monitor public discourse on policy and international topics
Journalism	Access Chinese public opinion data for reporting
Academic Research	Study Chinese social media behavior and trends

Four Scraping Modes

The actor supports four distinct modes, each targeting a different type of Weibo data:

1. Hot Search / Trending Topics (no login needed)

Get the real-time pulse of the Chinese internet. Returns trending topics with heat scores and categories.

{
  "mode": "hot_search",
  "maxResults": 50
}

Output:

{
  "rank": 1,
  "title": "人工智能最新突破",
  "category": "科技",
  "hotValue": 2847562,
  "labelName": "热",
  "isHot": true,
  "url": "https://s.weibo.com/weibo?q=...",
  "scrapedAt": "2026-04-10T12:00:00Z"
}

2. Post Comments (no login needed)

Extract comments from specific posts. Provide post IDs (mid) or detail URLs.

{
  "mode": "post_comments",
  "postIds": ["5285773987283226"],
  "maxComments": 50
}

3. Search Posts (login optional)

Search by keyword. Without cookies, returns hot timeline posts as a fallback. With cookies, searches the full index.

{
  "mode": "search",
  "searchQuery": "人工智能",
  "maxResults": 50,
  "cookieString": "SUB=your_sub_cookie_value"
}

4. User Posts (login needed for posts)

Get profile info (always works without login) plus posts (requires cookies). Provide numeric user IDs or profile URLs.

{
  "mode": "user_posts",
  "userIds": ["1642634100"],
  "maxResults": 50,
  "cookieString": "SUB=your_sub_cookie_value"
}

How to Get Cookies

For search and user posts modes, you need a Weibo SUB cookie:

Open weibo.com in your browser and log in
Open DevTools (F12) → Application → Cookies → weibo.com
Copy the value of the SUB cookie
Paste it in the cookieString field as: SUB=your_value_here

The cookie typically lasts several days before expiring.

Python and JavaScript Examples

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "hot_search",
    "maxResults": 50
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('zhorex/weibo-scraper').call({
    mode: 'hot_search',
    maxResults: 50,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => console.log(item));

Technical Details

No browser needed — pure HTTP requests using httpx, runs in 256MB RAM
No VPN needed — Weibo endpoints are globally accessible
Automatic session — visitor cookies obtained automatically via the Sina Visitor System
Rate-limit handling — exponential backoff on 418/429 errors
Chinese text preserved — all content returned as-is in original Simplified Chinese

Pricing

Volume	Cost
1,000 items	$5
10,000 items	$50
100,000 items	$500

Each scraped item (post, comment, trending topic, or profile) counts as one result. You can start with Apify's free plan, which includes $5 of monthly credits — enough for 1,000 data points.

Part of the Chinese Digital Intelligence Suite

Weibo covers microblogging and public opinion, but for comprehensive China market intelligence you need the full picture:

Platform	Users	What it covers	Actor
Weibo	580M+ MAU	Microblogging, trending, celebrity content	`zhorex/weibo-scraper`
RedNote (Xiaohongshu)	200M+ MAU	Social commerce, lifestyle, product reviews	`zhorex/rednote-scraper`
Bilibili	300M+ MAU	Video content, danmaku, creator analytics	`zhorex/bilibili-scraper`

All three actors: no browser, no proxy, no API keys. Built by Zhorex — the only developer on Apify specializing in Chinese platform intelligence.

FAQ

Is there a Weibo API?
There is no official public Weibo API available for international developers. Weibo's developer platform requires a Chinese business license and imposes strict rate limits. This scraper is the best alternative.

Do I need a VPN?
No. The Weibo endpoints used by this actor are globally accessible without a VPN or proxy.

Is the content in Chinese?
Yes. Weibo is a Chinese-language platform — all content is returned in the original Simplified Chinese. If you need English translations, pipe the output through a translation API (Google Translate, DeepL, or Claude).

Is scraping Weibo legal?
This scraper only accesses publicly available data through Weibo's public web endpoints. It does not bypass authentication or access private/locked accounts. Always review your local laws and Weibo's terms of service.

Get Started

The Actor page, full input schema, and a free trial run are at:

https://apify.com/zhorex/weibo-scraper

Start with trending topics (no login needed) to see the data quality, then expand to search and user profiles as needed.

Scraping G2 Reviews Without Anti-Bot Headaches: A SaaS Competitive Intelligence Pipeline With 41 Fields Per Review

Sami — Fri, 17 Apr 2026 17:51:52 +0000

G2 hosts verified software reviews across tens of thousands of SaaS categories. For anyone doing competitive intelligence, sales prospecting, or product research in the B2B software space, it is one of the single highest-signal public datasets on the internet. The problem is that scraping it has become almost comically painful in 2026.

If you have tried hitting g2.com with requests lately, you already know the story. Cloudflare challenges, DataDome fingerprinting, TLS fingerprint checks, behavioral JavaScript puzzles, and invisible CAPTCHAs. Even well-configured headless browsers with residential proxies get flagged within the first dozen pages. The G2 review data is public, but getting to it at scale has turned into a cat-and-mouse game that burns engineering hours and proxy budget in equal measure.

This article walks through a cleaner path: using the zhorex/g2-reviews-scraper Actor on Apify to pull structured reviews, 41 fields deep, without running a single browser or rotating a single proxy.

Why G2 Data Is Worth the Effort

Before getting into the scraper itself, it helps to articulate what the data is actually good for. G2 reviews are structured, long-form, and written by verified business users. That combination makes them uniquely useful for:

Sales intelligence. A review that complains about vendor X's integration with Salesforce is a warm lead for vendor Y whose Salesforce integration is its headline feature.
Win/loss analysis. Reviews of competitors often name the alternatives the reviewer evaluated. This is free narrative market research.
Feature gap detection. Aggregating thousands of "what do you dislike" fields across a category surfaces the roadmap items customers actually care about.
Churn signals. Negative sub-ratings on "support" or "ease of setup" for a specific competitor, trended over quarters, predict defection windows.

The value is there. The delivery mechanism is the bottleneck.

The Problem With the Official Path

G2 does offer a paid API, but it is gated behind an enterprise contract, requires a seat license on the G2 side, and typically restricts you to data about your own product and a handful of named competitors. Pulling the full set of reviews for a category like "CRM Software" or "Marketing Automation" across all vendors is not on the menu.

The community workarounds are worse. DIY scrapers hit Cloudflare + DataDome within minutes. Proxy bills for rotating residential IPs can run into the hundreds or thousands per month. Every time G2 rolls a new JS challenge, your Playwright script breaks at 3 AM and you spend a Saturday fingerprinting headers.

What the Actor Does

The g2-reviews-scraper Actor bypasses all of that by calling G2's public review feed directly. No browser, no proxy rotation, no anti-bot bypass hacks. You give it a product URL or slug and it returns structured JSON.

Feature set:

Scrape reviews for any G2 product URL or slug
41 fields per review including sub-ratings, reviewer job title, company size, industry, and verification status
Pagination handled internally, up to the full review history of a product
Filters for star rating, date range, and review language
JSON, CSV, Excel, or JSONL output
Runs on Apify infrastructure, so no local Node/Python setup required for the scraping itself

Comparison Table

Capability	G2 Official API	DIY scraper + residential proxies	`zhorex/g2-reviews-scraper`
Access to competitor reviews	No (own product only)	Yes, if you can keep it running	Yes
Anti-bot challenge handling	N/A	Your problem, breaks weekly	Handled, no browser needed
Setup time	Weeks (contract + provisioning)	Days to weeks	Minutes
Cost at 10k reviews/month	Custom enterprise quote	~$300-600 proxies + eng time	$50
Sub-ratings included	Yes	Usually not	Yes (41 fields)
Export formats	JSON via API	Whatever you build	JSON, CSV, XLSX, JSONL
Maintenance burden	Low	High	None

Input Example

A realistic starter config for pulling reviews across three CRM competitors:

{
  "productUrls": [
    "https://www.g2.com/products/salesforce-sales-cloud/reviews",
    "https://www.g2.com/products/hubspot-sales-hub/reviews",
    "https://www.g2.com/products/pipedrive/reviews"
  ],
  "maxReviewsPerProduct": 500,
  "minRating": 1,
  "maxRating": 5,
  "dateFrom": "2024-01-01",
  "language": "en",
  "includeSubRatings": true,
  "includeReviewerProfile": true
}

Output Example

Here is one review item, trimmed to the fields most people care about. The full object has all 41 fields.

{
  "reviewId": "g2-8421930",
  "productSlug": "hubspot-sales-hub",
  "productName": "HubSpot Sales Hub",
  "reviewTitle": "Great for SMB, feels cramped above 50 reps",
  "starRating": 3.5,
  "subRatings": {
    "easeOfUse": 4.5,
    "qualityOfSupport": 4.0,
    "easeOfSetup": 4.0,
    "meetsRequirements": 3.5
  },
  "reviewLikes": "Pipeline view is clean, sequences are easy to build, and the free tier got us started without procurement.",
  "reviewDislikes": "Once we hit 60 reps the reporting module struggled. Forecasting is weaker than Salesforce and custom objects are limited.",
  "recommendations": "Fine for teams under 50. Above that, evaluate Salesforce or Dynamics.",
  "reviewer": {
    "displayName": "Verified User in Software",
    "jobTitle": "RevOps Manager",
    "industry": "Computer Software",
    "companySize": "51-200 employees",
    "isVerified": true
  },
  "publishedAt": "2025-11-08T14:22:10Z",
  "language": "en",
  "helpfulCount": 12,
  "source": "organic",
  "reviewUrl": "https://www.g2.com/products/hubspot-sales-hub/reviews/hubspot-sales-hub-review-8421930"
}

Fields include reviewId, productSlug, productName, starRating, five sub-ratings, reviewLikes, reviewDislikes, recommendations, problemsSolved, benefitsRealized, reviewer display name, job title, industry, company size, region, validation status, publishedAt, updatedAt, language, helpfulCount, source, incentive (if the review was incentivized), and the canonical reviewUrl.

Four Real Use Cases

1. SaaS Sales Displacement Plays

A sales team selling a CRM builds a nightly job that pulls all 1- and 2-star reviews for three major competitors. Each review is piped through an LLM that extracts the specific complaint and the reviewer's company. The result is a prioritized outbound list where every lead comes with a documented pain point in their own words. Open rates on personalized sequences built from real G2 complaints routinely run 2-3x generic cold outbound.

2. Category-Level Feature Gap Analysis

A product manager at a marketing automation vendor scrapes every review in the "Marketing Automation" category filed in the last 12 months, thousands of reviews across dozens of products. She clusters the "dislikes" text with embeddings and counts cluster frequency per vendor. The result is a heatmap showing which features are consistent weak spots across the category (great roadmap input) and which are only weak for specific competitors (great competitive collateral).

3. Churn and Renewal Risk Signals

A customer success team at an observability vendor subscribes to a rolling scrape of their own product's reviews plus the top five competitors. Any new 1- or 2-star review mentioning an integration or feature their product covers gets routed to a Slack channel. It acts as an early-warning system for account risk and a real-time queue of switch-ready prospects.

4. Private Equity Due Diligence

A PE analyst evaluating a SaaS acquisition scrapes 5 years of G2 history for the target and three comparable vendors. The trend of monthly average star rating, sub-rating deltas, and review volume growth becomes part of the investment memo. This is one of the few ways to reality-check the seller's narrative about product quality and market position.

Pricing

The Actor is priced at $0.005 per review scraped, billed through Apify. Platform compute usage is negligible because there is no browser.

Worked examples:

1,000 reviews: $5
10,000 reviews (a large product's full history): $50
100,000 reviews (a full category sweep): $500
1,000,000 reviews (multi-category enterprise pull): $5,000

Compare that to a DIY build where residential proxies for similar volumes can cost hundreds to thousands per month, plus engineering time to keep the anti-bot bypass alive. For most teams the break-even point is well under a week.

FAQ

Is scraping G2 reviews legal?
The Actor accesses G2's public Elasticsearch API — no authentication is bypassed and no terms of service are circumvented. That said, how you use and redistribute the data is on you. Do not republish full review text as your own content, and respect GDPR if you process reviewer metadata for EU subjects. Always consult legal counsel for your specific use case.

Do I need proxies or an anti-bot solver?
No. The Actor uses G2's public review feed directly and does not trigger Cloudflare or DataDome. You do not need to supply proxies, browser fingerprints, or CAPTCHA solver tokens.

How fresh is the data?
Reviews are scraped live at run time. If a review was published five minutes before your run, it will be included. For ongoing monitoring, schedule the Actor hourly or daily via Apify Schedules.

What is the rate limit?
Practically speaking, you are limited by Apify concurrency and the Actor's internal pacing, not by G2 blocking. Expect roughly 500-1,000 reviews per minute per run. Runs can be parallelized across products.

Can I get all 41 fields or is that a premium tier?
All fields are included at the flat $0.005 per review rate. There is no feature-gated premium tier.

How do I export to my warehouse?
Apify exposes datasets as JSON, CSV, XLSX, JSONL, RSS, and HTML table, and has native integrations for S3, Google Drive, and webhooks. A common pattern is JSONL to S3, then COPY into Snowflake or BigQuery.

Pair It With Capterra for Full B2B Coverage

G2 skews toward mid-market and enterprise SaaS. Capterra, owned by Gartner, leans more toward SMB and has broader coverage of vertical software (construction, healthcare, legal). For any serious competitive intel project, you want both. The companion Actor zhorex/capterra-reviews-scraper uses the same schema philosophy and pairs cleanly with this one in a single pipeline. If you are also tracking sentiment on Chinese-language software or consumer platforms, zhorex/weibo-scraper covers the APAC side.

Get Started

The Actor page, full input schema, and a free trial run live at:

https://apify.com/zhorex/g2-reviews-scraper

Drop in a product URL, run it, and you will have a clean JSON dataset in the Apify console in a couple of minutes. No proxy contract, no anti-bot cat-and-mouse, no maintenance bill at 3 AM.

Telegram Channel Scraper: Extract Messages, Media & Metadata Without API Keys or Phone Numbers

Sami — Wed, 15 Apr 2026 17:15:49 +0000

Telegram has 950+ million monthly active users and has become the go-to platform for crypto communities, news channels, research groups, and brand communications. But extracting data from Telegram channels programmatically is surprisingly difficult — until now.

This guide covers why Telegram scraping matters, the technical challenges involved, and how to extract messages, views, reactions, and media from any public Telegram channel without needing API keys, phone numbers, or Telegram bot tokens.

Why Scrape Telegram Channels?

Telegram isn't just a messaging app — it's a broadcasting platform where organizations, communities, and influencers share real-time information. The data inside Telegram channels is valuable for:

Business Intelligence

Crypto & DeFi research — Track token announcements, project updates, and community sentiment across hundreds of channels
Brand monitoring — See what's being said about your brand in Telegram communities
Competitive intelligence — Monitor competitor announcements and marketing strategies

Research & OSINT

Open-source intelligence (OSINT) — Investigate public channels for geopolitical, security, or journalistic research
Academic research — Study information spread, community dynamics, and content patterns
Market research — Analyze consumer conversations and product discussions

Marketing & Content

Influencer discovery — Find active Telegram channels in your niche with engaged audiences
Content strategy — Analyze which message formats, topics, and posting times drive the most views and engagement
Lead generation — Identify potential customers discussing relevant topics

The Challenge: Why Telegram Scraping Is Hard

If you've tried to build a Telegram scraper, you know the pain:

The Official API Requires Authentication

Telegram's Bot API and MTProto API both require:

A phone number to create an account
An API key from my.telegram.org
Session management with 2FA complications
Compliance with strict rate limits

For bulk data extraction, you risk account bans if you exceed rate limits or trigger anti-abuse systems.

The Telethon/Pyrogram Approach

The most common DIY approach uses Python libraries like Telethon:

from telethon import TelegramClient

# Requires API ID, API hash, AND a phone number
client = TelegramClient('session', api_id, api_hash)

async def scrape_channel(channel_name):
    await client.start(phone='+1234567890')  # Phone verification required
    channel = await client.get_entity(channel_name)
    messages = await client.get_messages(channel, limit=100)
    return messages

Problems with this approach:

You need to provide a real phone number and verify with SMS
Your account can get banned if you scrape too aggressively
Session management is fragile — sessions expire, require re-auth
You need to handle flood wait errors and implement backoff logic
Media downloads require additional API calls per file

The Easy Way: No API Keys, No Phone Numbers

The Telegram Channel Scraper on Apify takes a completely different approach. Instead of using Telegram's authenticated API, it accesses public channel data through Telegram's web preview endpoints — the same ones that power t.me preview pages.

What This Means For You

No API key needed — zero authentication required
No phone number — no account registration or SMS verification
No proxy needed — direct HTTP requests work without rotation
No account ban risk — doesn't use an authenticated Telegram session
No browser needed — pure HTTP requests, fast and cheap

What Data Can You Extract?

From any public Telegram channel, you get:

Data Field	Description
Message text	Full message content including formatting
Views	Number of views per message
Reactions	Emoji reactions with counts
Date & time	Publication timestamp
Media URLs	Links to photos, videos, documents
Forward info	Original source if message was forwarded
Reply info	Which message it replies to
Channel metadata	Name, description, subscriber count, avatar

How To Use It

{
  "channelNames": ["duaborev", "crypto_news"],
  "maxMessages": 100,
  "includeMedia": true
}

That's it. Enter channel names (or full t.me URLs), set your message limit, and hit Start.

Real-World Use Cases

Crypto Research Dashboard

Step 1: List 50 crypto project Telegram channels
Step 2: Schedule daily scraping → extract latest messages
Step 3: Run NLP sentiment analysis on message text
Step 4: Build dashboard tracking project activity and community sentiment
Step 5: Alert when channels mention specific tokens or events

OSINT Investigation

Step 1: Identify target public channels
Step 2: Extract full message history with timestamps
Step 3: Analyze posting patterns, forwarded sources, media
Step 4: Map information networks and content origin

Brand Monitoring

Step 1: Search for channels mentioning your brand/product
Step 2: Schedule weekly scraping of relevant channels
Step 3: Track mention frequency, sentiment, and reach (views)
Step 4: Compare share of voice vs competitors

Content Marketing Research

Step 1: Scrape top channels in your niche
Step 2: Analyze which posts get highest views and reactions
Step 3: Identify optimal posting times and content formats
Step 4: Apply insights to your own Telegram channel strategy

Why This Scraper vs. Alternatives

Method	API Key	Phone #	Ban Risk	Speed	Cost
Telethon (DIY)	Yes	Yes	High	Slow (rate limits)	Free + server costs
Pyrogram (DIY)	Yes	Yes	High	Slow (rate limits)	Free + server costs
Bot API	Yes	No	Medium	Medium	Free + server costs
SaaS tools	Varies	Varies	Low	Medium	$50-200+/mo
Telegram Channel Scraper	No	No	None	Fast	$5/1K messages

Key Advantages

Zero setup friction — No API registration, no phone verification, no OAuth flow. Enter channel names and go.
No ban risk — Since it doesn't use an authenticated session, there's no account to ban.
Pay per result — $0.005 per message ($5 per 1,000). No monthly subscription.
Scheduled runs — Set up daily/weekly/hourly scraping directly in Apify.
Export anywhere — JSON, CSV, Excel, XML. Integrate with Google Sheets, Zapier, Make, n8n.

Pricing

Volume	Cost	Per-Message
100 messages	$0.50	$0.005
1,000 messages	$5.00	$0.005
10,000 messages	$50.00	$0.005

Free Apify trial includes credits to test before committing. No monthly fees.

FAQ

Does this scrape private Telegram channels?

No. This scraper only accesses publicly available Telegram channels — the same content anyone can view at t.me/channelname without a Telegram account.

Can I scrape Telegram groups (not channels)?

Currently, the scraper is optimized for public channels (one-way broadcast). Group chat scraping requires authenticated API access and is not supported.

Does it download media files?

It extracts media URLs (photos, videos, documents) so you can download them separately. The URLs point to Telegram's CDN and are publicly accessible.

How often can I run it?

As often as you need. Schedule runs every hour for near-real-time monitoring, or daily/weekly for trend analysis.

Is scraping Telegram channels legal?

This scraper only accesses publicly available content that anyone can view without a Telegram account. It does not bypass authentication or access private data. Always comply with local regulations when using scraped data.

Get Started in 60 Seconds

Go to the Telegram Channel Scraper →
Click "Try for free"
Enter your channel names
Hit Start → get structured JSON data

No API keys. No phone numbers. No account bans. Just clean Telegram channel data ready for analysis.

Try it free on Apify →

Part of the Zhorex scraper suite on Apify. Also check out: RedNote Xiaohongshu Scraper · Kick.com Analytics · G2 Reviews Scraper

Kick.com API Alternative: Extract Streamer Analytics, Live Streams & VODs Without Authentication

Sami — Wed, 15 Apr 2026 17:07:41 +0000

Kick.com doesn't offer a public API. But if you need Kick streamer data — profiles, live streams, VODs, clips, or channel rankings — there's a clean, structured way to get it without building your own scraper from scratch.

This guide covers what data is available from Kick.com, how the platform's internal API works, and the fastest way to extract Kick streaming analytics at scale.

Why Kick.com Data Is Valuable in 2026

Kick.com has positioned itself as the creator-friendly alternative to Twitch, offering streamers a 95/5 revenue split (vs Twitch's 50/50). This has attracted a wave of major creators, making Kick one of the fastest-growing live streaming platforms globally.

For businesses and analysts, this growth creates demand for structured Kick.com data:

Who Needs Kick Data?

Influencer marketing agencies — Find the right Kick streamers for brand deals based on viewer counts, categories, and engagement patterns
Esports organizations — Track competitive gaming viewership and identify rising talent
Brand sponsors — Monitor ROI of sponsorship deals across Kick channels in real time
Content analysts — Discover which categories, stream formats, and clip styles perform best
Competitive intelligence teams — Compare Kick vs Twitch vs YouTube Gaming performance for specific creators

What Data Can You Extract?

Data Type	Examples
Channel profiles	Follower count, bio, avatar, social links, verified status, creation date
Live streams	Current viewers, stream title, category, start time, language, tags
VODs	Past broadcasts with duration, views, category, thumbnail
Clips	Clip title, views, likes, duration, creator, video URL
Rankings	Top channels by viewer count, filtered by category

The Problem: No Public Kick API

Unlike Twitch (which has a documented API), Kick.com doesn't offer a public developer API. The platform does use internal API endpoints that return JSON, but:

Endpoints change without notice — Kick updates their internal routes regularly
TLS fingerprinting — Standard HTTP clients get blocked; you need browser-level TLS impersonation
Rate limiting — Too many requests from the same IP get throttled
No documentation — You're reverse-engineering endpoints from browser network traffic

Building a DIY Kick scraper means maintaining code that breaks every few weeks when Kick updates their internal API structure.

The Solution: 4-in-1 Kick.com Analytics Actor

The Kick.com Streamer & Channel Analytics actor on Apify solves all of this. It handles TLS fingerprinting, rate limiting, and endpoint changes — and delivers clean JSON data through 4 scraping modes.

Mode 1: Channel Profiles (`channel_details`)

Get complete Kick streamer profiles:

{
  "mode": "channel_details",
  "channelNames": ["xqc", "amouranth", "trainwreckstv"]
}

Returns: display name, bio, avatar, banner, follower count, live status, current viewers, category, stream title, verified badge, subscriber badges, social links (Instagram, Twitter, YouTube, Discord, TikTok), and creation date.

Use case: Build a Kick influencer database with accurate follower counts and cross-platform social links for outreach campaigns.

Mode 2: Live Streams (`live_streams`)

Discover who's streaming live on Kick right now:

{
  "mode": "live_streams",
  "category": "just-chatting",
  "minViewers": 100,
  "maxResults": 50,
  "sortBy": "viewers"
}

Returns: channel name, viewer count, stream title, category, start time, thumbnail, language, tags, and maturity rating.

Use case: Real-time Kick viewership monitoring — schedule runs every 5 minutes for a live dashboard of who's streaming and how many viewers they have.

Mode 3: VODs & Clips (`channel_videos`)

Extract past broadcasts and Kick clips:

{
  "mode": "channel_videos",
  "channelNames": ["xqc"],
  "videoType": "clips",
  "maxResults": 50
}

Returns: clip ID, title, duration, views, likes, category, thumbnail, video URL, creator, and creation date.

Use case: Analyze what content formats get the most views and engagement on Kick — identify optimal clip lengths, trending categories, and viral content patterns.

Mode 4: Channel Rankings (`top_channels`)

Get a ranked list of top Kick channels by live viewers:

{
  "mode": "top_channels",
  "sortBy": "viewers",
  "category": "fortnite",
  "maxResults": 25
}

Returns: rank, channel name, bio, avatar, current viewers, stream title, category, affiliate status, and social links.

Use case: Competitive streaming intelligence — see who's dominating each category on Kick, and how viewer distribution compares to Twitch.

Why This Actor vs. DIY Scraping

Feature	DIY Python Scraper	Kick.com Analytics
Setup time	Hours to days	2 minutes
TLS fingerprinting	Must handle yourself	Built-in
Rate limit handling	Your problem	Handled automatically
API change updates	You maintain it	Maintained for you
4 scraping modes	Build each separately	All included
Data normalization	Manual parsing	Clean JSON output
Scheduling	Cron jobs + servers	Built-in Apify scheduler
Cost	Server + dev time	$0.005/result ($5/1K)

Key Technical Advantages

No API key needed — accesses Kick.com's public API endpoints directly
No proxy needed — direct HTTP requests work without proxy rotation
No browser needed — pure HTTP with TLS impersonation, no Playwright/Puppeteer overhead
Lightweight — runs on 256 MB RAM
Structured output — clean JSON, CSV, Excel, or XML export

Real-World Workflow Examples

Influencer Marketing: Find & Vet Kick Streamers

1. top_channels mode → get top 100 streamers in target category
2. channel_details mode → bulk-pull profiles for all 100
3. Filter by follower count, verified status, social links
4. channel_videos (clips) → check engagement on recent content
5. Build shortlist with data-backed partnership recommendations

Esports: Track Tournament Viewership

1. live_streams mode → filter by game category (e.g., "valorant")
2. Schedule every 5 min during tournament hours
3. Export to Google Sheets via Apify integration
4. Build real-time viewership dashboard

Content Strategy: Analyze What Works on Kick

1. channel_videos (clips) → extract top clips from 20 channels
2. Analyze: avg views by duration, category, time of day
3. Identify content patterns that drive viral clips
4. Apply insights to your own streaming or client strategy

Cross-Platform Streaming Intelligence

The same developer also offers a Twitch Streamer & Channel Analytics actor with 6 scraping modes. Combined with the Kick actor, you can build a complete cross-platform streaming analytics pipeline:

Compare creator performance across Kick and Twitch
Track viewer migration between platforms
Identify creators who are growing faster on one platform vs. another
Monitor category trends across both ecosystems

Pricing

Pay-per-result with no monthly fees:

Volume	Cost
1,000 results	$5.00
10,000 results	$50.00
100,000 results	$500.00

Free Apify trial includes credits to test before committing.

Get Started

Go to the Kick.com Streamer & Channel Analytics actor →
Click "Try for free"
Select your mode, enter parameters, hit Start
Get structured JSON data in seconds

No API key. No proxy. No browser. Just clean Kick.com data ready for analysis, dashboards, or integration with your existing tools.

Try it free on Apify →