DEV Community: Mohamed Senator

How to scrape Bilibili search, comments and trending boards without logging in

Mohamed Senator — Wed, 21 Oct 2026 09:05:44 +0000

Disclosure: I built the Actor used in this tutorial, Bilibili Scraper on Apify.

Bilibili (哔哩哔哩, "B站") is where a big part of China's gaming, anime, tech and creator conversation happens. If you track a brand, a game launch or a topic in the Chinese market, you want three things from it: which videos match your keywords, what people say in the comments, and what's on the trending boards. Doing that by hand means a Chinese UI, login walls and pagination quirks.

This tutorial shows how to pull all three as clean JSON with one Actor, no Bilibili account needed.

What you get

Every row has the same shape, whatever mode you run:

{
  "platform": "bilibili",
  "type": "video",
  "id": "BV1Wenc6aExW",
  "url": "https://www.bilibili.com/video/BV1Wenc6aExW",
  "text": "【原神】判测胜券本征，原魔战队表现排行 ...",
  "authorName": "并非云某人",
  "authorUrl": "https://space.bilibili.com/130929912",
  "createdAt": "2026-10-20T18:01:45+08:00",
  "views": 21338,
  "likes": 810,
  "comments": 239,
  "favorites": 89,
  "platformMetrics": { "danmaku": 273, "duration": 384, "tags": "原神UP主激励计划,搞笑,..." },
  "keyword": "原神"
}

That's a real row from a search run for 原神 (Genshin Impact) on Oct 20, 2026. Comments come back as type: "comment" with a parentId pointing at the video, and trending items as type: "trend".

1. Search videos by keyword

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("s3nafps/bilibili-scraper").call(run_input={
    "mode": "search",
    "keywords": ["原神"],
    "maxItems": 100,
    "sort": "relevance",
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["views"], item["likes"], item["text"][:60])

In my test run this returned 88 videos in about 31 seconds.

2. Pull comments (and replies) for a video

Pass a BV id or a video URL:

run = client.actor("s3nafps/bilibili-scraper").call(run_input={
    "mode": "comments",
    "urls": ["BV12ez4BHEDM"],
    "maxItems": 20,
    "includeReplies": True,
    "commentSort": "hot",
})

You can also run mode: "details" with includeComments: true to get video stats and the top comments in one go.

3. Read a trending board

run = client.actor("s3nafps/bilibili-scraper").call(run_input={
    "mode": "trending",
    "trendingCategory": "game",
    "maxItems": 10,
})

The game board run took about 14 seconds for 10 items.

4. Daily brand monitoring with only-new mode

Set onlyNew: true and give the job a monitorName. The Actor remembers what it already returned for that monitor, so a daily schedule on Apify only hands you videos you haven't seen yet. Pipe that into Slack, a sheet, or your agent.

run_input = {
    "mode": "search",
    "keywords": ["你的品牌名"],
    "onlyNew": True,
    "monitorName": "my-brand-daily",
    "maxItems": 100,
}

Creator profiles

mode: "profiles" takes space.bilibili.com links or plain UIDs and returns one profile row each.

What it costs

It's pay per result, with no subscription. On Apify's free plan it's $2.50 per 1,000 videos or trending items, $1 per 1,000 comments, and $3 per 1,000 profiles, and it gets cheaper on higher Apify plans. So the 88-video search above works out to about $0.22.

Use it from an AI agent

Like any Apify Actor, it's callable over Apify's MCP server, so Claude, Cursor or your own agent can search Bilibili and read comments as a tool call.

Try it here: Bilibili Scraper on Apify. If a field you need is missing, tell me in the comments.

How to compare 500 keywords in Google Trends on one scale (not 5 at a time)

Mohamed Senator — Tue, 20 Oct 2026 12:59:09 +0000

Watch the full video

If you have ever tried to rank a long keyword list with Google Trends, you have hit the wall: the website compares 5 terms at a time, and every chart rescales its own biggest term to 100. A "60" in one chart and a "60" in another don't mean the same thing.

This post shows how to get around that and put hundreds of keywords (up to 500) on one shared 0–100 scale, with a real input, real output and code. Disclosure: the tool I use is my own Apify Actor, Google Trends Scraper: Bulk Compare 100+ Keywords.

The 5-keyword problem

Say you want to know which of 20 coffee drinks people search for most. On the Google Trends website you run 4 comparisons of 5. In each one, the top term becomes 100 and everything else is relative to it. So:

You can't compare "matcha" from chart 1 with "oat milk" from chart 4.
If one huge term (like "coffee") sits in a chart, everything small next to it rounds down to 0 or 1, because Google reports whole numbers.

The manual workaround is to put one shared keyword in every chart and rescale by hand. It works, but it is slow, and it breaks when the shared keyword is much bigger or smaller than the others.

How the Actor solves it

It does the linking automatically:

A quick first pass sorts your keywords by size.
Keywords are compared in groups of 5. Each group shares one keyword with the next group, so neighbours are always of similar size and small keywords keep their precision.
Every group is rescaled through its shared keyword, and the single biggest value across all your keywords becomes 100.

The result is one ranked table where a score means the same thing in every row.

If you'd rather measure everything against one reference, such as your brand, set anchorKeyword and that keyword is included in every comparison instead.

Walkthrough

1. The input

This is the input I ran, the same 20 drinks the Actor's input form is prefilled with:

{
  "keywords": [
    "coffee", "tea", "matcha", "espresso", "latte",
    "cold brew", "cappuccino", "green tea", "chai", "iced coffee",
    "mocha", "flat white", "french press", "decaf", "oat milk",
    "bubble tea", "kombucha", "hot chocolate", "energy drink", "yerba mate"
  ],
  "geo": "",
  "timeRange": "today 12-m",
  "searchType": "web"
}

The input fields:

Field	Default	What it does
`keywords`	required	1 to 500 keywords. Duplicates and blank lines are removed. Any language.
`geo`	`""` (worldwide)	Country code, for example `US`, `SA` or `AE`.
`timeRange`	`today 12-m`	`now 1-H`, `now 7-d`, `today 3-m`, `today 12-m`, `today 5-y`, `all` (since 2004) and more.
`customTimeRange`	none	For example `2024-01-01 2024-06-30`. Overrides `timeRange`.
`searchType`	`web`	`web`, `youtube`, `news`, `images` or `shopping`.
`category`	`0` (all)	Limit to one Google category.
`anchorKeyword`	none	Put one keyword (your brand, say) in every comparison.
`includeRelated`	`false`	Top and rising related searches per keyword.
`includeRegions`	`false`	Interest by country, or by region inside the chosen country.
`language`	`en`	Language for related searches and place names, for example `ar`.

2. The output

Here is the ranked table from that run (worldwide, past 12 months, run on October 18, 2026). averageScore is the keyword's average over the period on the shared scale:

Rank	Keyword	averageScore	trend	change3m	change12m
1	coffee	73.9	rising	−7%	+16%
2	tea	39.1	flat	−10%	+1%
3	espresso	9.2	flat	−7%	+14%
4	matcha	8.3	rising	−9%	+14%
5	latte	7.7	rising	+13%	+14%
6	chai	5.5	rising	+12%	+24%
7	green tea	3.6	flat	−21%	−1%
8	energy drink	2.6	rising	−4%	+15%
9	mocha	2.0	flat	+10%	+1%
10	cold brew	1.8	rising	−21%	+75%
…	…	…	…	…	…
20	yerba mate	0.5	rising	−42%	+5%

Your numbers will differ, because they come from Google at run time.

Notice that "yerba mate" gets a usable 0.5 even though "coffee" is about 150 times bigger. In a single 5-term chart next to "coffee", it would round to 0 or 1.

Each dataset row looks like this (a real row from a separate US run on October 20, 2026, with the weekly timeline shortened):

{
  "keyword": "matcha",
  "rank": 4,
  "averageScore": 6.1,
  "peakScore": 8,
  "peakDate": "2026-04-05",
  "latestScore": 5,
  "change3m": -31,
  "change12m": 18,
  "trend": "rising",
  "peakMonth": null,
  "breakout": false,
  "lowVolume": false,
  "noData": false,
  "timeline": [
    { "date": "2025-10-19", "score": 4 },
    { "date": "2025-10-26", "score": 4 }
  ],
  "searchType": "web",
  "geo": "US",
  "timeRange": "today 12-m",
  "anchorKeyword": null
}

What the fields mean:

trend is rising, falling, flat or seasonal. It describes the whole period, while change3m only looks at the last 3 months, so they can point in different directions (cold brew above: rising over the year, down 21% over 3 months).
peakMonth is filled for ranges of 2 years or more.
breakout is true when the last few points are more than double the keyword's average.
lowVolume is true when Google rounds a keyword to 0 or 1, so its exact size is uncertain. Compare it again in a list of similar-sized keywords.
noData: true means Google has no data for that keyword. These keywords aren't charged.

The run also saves a SUMMARY record in the key-value store with the ranked table and one sentence. For the 20-drink run it was:

Top by interest: coffee (73.9), tea (39.1), espresso (9.2). Fastest rising over 3 months: hot chocolate (+28%), latte (+13%), chai (+12%).

That's short enough to paste straight into an LLM prompt or a Slack message.

3. What 500 keywords looks like

I also ran a 500-keyword list (worldwide, past 12 months): broad terms like "chatgpt", "iphone" and "coffee" plus long-tail variants like "lipstick deals". It finished in about 7 minutes. Every keyword came back on the same scale, from 71.1 for "chatgpt" at the top down to 0.00033 for "lipstick deals" at rank 500. On the Google Trends website, anything below 1 in a chart just shows as 0, so you couldn't rank those long-tail terms against each other.

Run it from code

Python (pip install apify-client):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

keywords = [line.strip() for line in open("keywords.txt") if line.strip()]  # up to 500

run = client.actor("s3nafps/google-trends-bulk-compare").call(run_input={
    "keywords": keywords,
    "geo": "US",
    "timeRange": "today 12-m",
    "searchType": "web",
})

rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
for row in rows[:20]:
    print(row["rank"], row["keyword"], row["averageScore"], row["trend"], row["change3m"])

summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("SUMMARY")
print(summary["value"]["text"])

To get a spreadsheet, export the dataset as CSV or Excel from the run's Output tab, or with pandas:

import pandas as pd

df = pd.DataFrame(rows).drop(columns=["timeline"])
df.to_csv("trends_ranked.csv", index=False)

You can also run it from n8n, Make or Zapier with the Apify "Run Actor" node, or give it to an AI agent through MCP at https://mcp.apify.com/?actors=s3nafps/google-trends-bulk-compare.

Use cases

SEO keyword research: rank 50–500 keyword ideas by real search interest before you write a page, and use lowVolume to spot terms that are too small to bother with.
Content planning: find the topics with positive change3m / change12m or a breakout flag, and use peakMonth (with a 2-year+ range) to time seasonal posts.
Market research: compare the same product list across countries by changing geo, or set searchType to shopping (Google Shopping) or youtube to see product demand. Add includeRegions to see where interest is concentrated.
Brand tracking: set your brand as anchorKeyword, add competitors, and schedule a weekly run.

Pricing

Pay per keyword, checked today on the Store:

$0.002 per keyword compared on the Free and Bronze plans (Silver $0.0018, Gold and above $0.0015).
$0.00005 start fee per run.
Related searches and regions are included at no extra cost.
Keywords that fail, can't be linked to the shared scale, or have no data aren't charged.

That's about $0.20 for 100 keywords and $1.00 for 500 on the free plan.

FAQ

Why don't my scores match the Google Trends website?
On the website, each chart scales its own biggest term to 100. Here, the biggest term across all your keywords is 100. If you compare the same 5 keywords on their own, the ratios between them match the website.

How long does a run take?
A 100-keyword run usually takes 1 to 3 minutes. My 500-keyword run took about 7 minutes.

Why did one keyword get an error instead of a score?
A keyword more than about 200 times bigger or smaller than everything it was compared with shows as 0 on Google and can't be linked to the scale. The Actor keeps the biggest set of keywords it can link, reports the outlier with an error and doesn't charge for it. Compare it in a separate list of similar-sized keywords.

Which countries and languages work?
Every country Google Trends supports, and keywords in any language, including Arabic.

Can I schedule it?
Yes. Use Apify Schedules to run the same list daily or weekly, and connect a webhook, Google Sheets or Slack to get the new ranking automatically.

Is it allowed?
The Actor only reads public, aggregated Google Trends data, with no personal data. Check that your use complies with Google's terms and the laws that apply to you.

Try it

Google Trends Scraper: Bulk Compare 100+ Keywords on the Apify Store. The input form is prefilled with the 20 drinks above, so you can see a full ranked table for about $0.04.

If a keyword list gives odd results, open an issue on the Actor page with the input so I can look at it.

Turn any docs site into RAG-ready Markdown (without writing a scraper)

Mohamed Senator — Tue, 20 Oct 2026 12:59:09 +0000

Watch the full video

Every RAG project I have seen starts the same way: someone wants a chatbot that answers questions from a documentation site, and the first week disappears into scraping. Menus end up in your embeddings, code blocks lose their language, JavaScript-rendered pages come back empty, and chunks split in the middle of a sentence.

I got tired of redoing this, so I built a small tool for it: Website to Markdown Crawler for LLMs & RAG on the Apify Store. Disclosure: it is my own Actor. This post shows the approach, the exact input I use and how to load the result into a vector store. The ideas apply to any scraper.

What "RAG-ready" actually means

Raw HTML is the wrong input for an embedding model. What you want per page is:

Only the main content. No nav bars, cookie banners, footers, share buttons or sidebars.
Structure kept as Markdown. Headings, lists, tables and fenced code blocks with their language, so an LLM can read it and you can chunk on headings.
Chunks that respect headings. Each chunk should know which section it came from (for example First crawler > Building a crawler), stay under a size limit, and overlap a little with the previous chunk so context isn't lost at the boundary.
Metadata. Title, description, canonical URL, language and dates, for filtering and for citing sources in answers.

The three problems that cost the most time

JavaScript-rendered pages. Many docs sites are React or Next.js apps, and a plain HTTP fetch returns an almost empty shell. The cheap fix is to fetch with plain HTTP first, check whether the extracted content looks missing, and only then open the page in a real browser. Rendering every page in a browser works too, but it is much slower.

Finding every page. Following links works, but a sitemap is usually faster and more complete. Some sites have no usable sitemap, so you need a fallback to link-following.

Chunking. Splitting every N characters cuts code blocks and sentences in half. Splitting on headings first and then packing up to a token limit keeps each chunk coherent.

The Actor handles all three. renderJavaScript: "auto" (the default) does the HTTP-first, browser-only-when-needed fetch; crawlMode: "sitemap" falls back to following links when a site has no sitemap; and chunkContent: true adds heading-aware chunks.

Step by step: one docs section into chunks

1. Pick the start URL and a filter

Use the docs homepage as the start URL and urlMustContain to stay inside the section you care about. Start URLs are always extracted; crawled or sitemap URLs must contain the filter text.

2. Choose a crawl mode

none (default): only the URLs you list.
sameDomain: also follow links on the same site, up to maxCrawlDepth (default 2).
sitemap: use the site's sitemap. Usually the fastest way to get a whole docs site.

3. Turn on chunking and set a page cap

This is the input I use:

{
  "urls": ["https://docs.apify.com"],
  "crawlMode": "sitemap",
  "urlMustContain": "docs.apify.com/actors/",
  "maxPages": 200,
  "chunkContent": true,
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 100
}

Other useful options: outputFormat (markdown, text or both), removeBoilerplate (on by default), includeLinks, includeTables and includeMetadata (all on by default), renderJavaScript (auto, never or always) and maxConcurrency (default 5).

4. Run it from code

With the Python client (pip install apify-client):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("s3nafps/ai-ready-webpage-extractor").call(run_input={
    "urls": ["https://docs.apify.com"],
    "crawlMode": "sitemap",
    "urlMustContain": "docs.apify.com/actors/",
    "maxPages": 50,
    "chunkContent": True,
    "chunkSizeTokens": 800,
    "chunkOverlapTokens": 100,
})

pages = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(len(pages), "pages")
for page in pages[:5]:
    print(page["title"], page["estimatedTokens"], len(page["chunks"]))

Or in JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('s3nafps/ai-ready-webpage-extractor').call({
  urls: ['https://docs.apify.com'],
  crawlMode: 'sitemap',
  urlMustContain: 'docs.apify.com/actors/',
  maxPages: 50,
  chunkContent: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, 'pages');

5. What comes back

One dataset item per page. Here is a real item from a crawl of the Crawlee docs, shortened. The chunk starts with text from the end of the previous chunk; that is the overlap.

{
  "sourceUrl": "https://crawlee.dev/js/docs/introduction/first-crawler",
  "title": "First crawler | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "description": "Your first steps into the world of scraping with Crawlee",
  "canonicalUrl": "https://crawlee.dev/js/docs/introduction/first-crawler",
  "language": "en",
  "mainContentMarkdown": "Version: 3.18\n\n# First crawler\n\nNow, you will build your first crawler...",
  "chunks": [
    {
      "index": 4,
      "headingPath": ["First crawler", "Building a crawler", "Building a CheerioCrawler"],
      "content": "Bulk add requests\n\nThe code above is for illustration of the request queue concept...",
      "estimatedTokens": 647
    }
  ],
  "headings": [{ "level": 1, "text": "First crawler" }],
  "wordCount": 1219,
  "estimatedTokens": 2236,
  "renderedWithBrowser": false,
  "crawlDepth": 1,
  "extractionStatus": "succeeded"
}

Each item also has finalUrl, httpStatus, author, publishedDate, modifiedDate, links, tables, images, schemaOrgData and scrapedAt. The dataset has two views: Overview (one row per page) and RAG chunks (one row per chunk). Failed pages and their reasons go into the OUTPUT record in the run's key-value store.

That Crawlee crawl (sameDomain mode, urlMustContain: "/js/docs/", 25 pages) is also a good warning: 9 of the 25 pages were copies of the Quick Start page under different version folders (/docs/3.11/, /docs/3.12/ and so on). Check your first run for versioned duplicates and tighten urlMustContain before crawling more.

6. Embed the chunks

From here, embedding is a loop over chunks, using headingPath and sourceUrl as metadata:

for page in pages:
    for chunk in page["chunks"]:
        vector_store.add(
            text=chunk["content"],
            metadata={
                "url": page["canonicalUrl"] or page["sourceUrl"],
                "title": page["title"],
                "section": " > ".join(h.replace("\u200b", "") for h in chunk["headingPath"]),
            },
        )

The replace("\u200b", "") is there because some docs themes put an invisible zero-width space in heading anchors, and it ends up in the heading text.

If you use LangChain, ApifyDatasetLoader from langchain_community.document_loaders can load the dataset directly; map mainContentMarkdown (or each chunk) to a Document.

Pricing

Pay per page, no subscription. Prices I checked today on the Store:

$0.003 per extracted page on the free plan (Bronze $0.0027, Silver $0.0024, Gold and above $0.002).
$0.00005 start fee per run (per GB of memory, default 1 GB).
Failed pages are not charged. Browser rendering and chunking are included.

So a 100-page docs section costs about $0.30 on the free plan, and 1,000 pages about $3. Set maxPages, and Maximum cost per run in the run options, to cap any run.

Things to know before you use it

Public pages only. No logins, cookies, paywalls or CAPTCHA solving.
Non-HTML files such as PDFs are skipped.
Main-content detection is heuristic. Unusual layouts can include a little extra text, so spot-check a few pages before you embed thousands.
Token counts are estimates (about 4 characters per token).

FAQ

Does it work on JavaScript sites?
Yes. With renderJavaScript: "auto", pages are fetched as plain HTML first and opened in a real browser only when the content comes back missing. Each item's renderedWithBrowser field tells you which path was used. For app-like sites that still come back short, set "always".

What if the site has no sitemap?
Sitemap mode falls back to following links from your start URL. You can also use sameDomain mode with maxCrawlDepth.

How do I keep costs predictable?
Set maxPages and a maximum cost per run. The run stops cleanly at the limit, and failed pages are never charged.

Which vector databases does it work with?
Any. The output is plain JSON with text, heading paths and token estimates, so it works with Pinecone, Qdrant, Weaviate, pgvector or a local store.

Can an AI agent call it directly?
Yes, through the Apify MCP server: add https://mcp.apify.com/?actors=s3nafps/ai-ready-webpage-extractor to Claude, Cursor or another MCP client.

Try it

Website to Markdown Crawler for LLMs & RAG on the Apify Store. The input form is prefilled with one URL, so a first test run costs less than a cent.

If you have a site that doesn't extract cleanly, open an issue on the Actor page with the URL. That is the fastest way for me to improve it.

How to replace Google's Custom Search JSON API before it shuts down (Jan 1, 2027)

Mohamed Senator — Mon, 19 Oct 2026 14:46:39 +0000

Watch the full video

Google's Custom Search JSON API (customsearch/v1) is closed to new customers and will stop answering on January 1, 2027. If your app, chatbot or AI agent calls it, you have until then to move.

The official successor, Vertex AI Search, is a different product: new setup, new response format, new pricing. If you just want your existing code to keep working, a drop-in replacement is less work.

This guide shows how to switch in about a minute, without changing how you read the results.

What "drop-in" means here

The replacement I'll use is an open actor on Apify that answers with the same JSON as Google's API: kind: customsearch#search, items[].title, items[].link, items[].snippet, htmlSnippet, searchInformation, and queries.nextPage. It accepts the same parameters: q, num, start, gl, hl, lr, cr, safe, dateRestrict, siteSearch, exactTerms, excludeTerms, orTerms and fileType. Errors come back in Google's shape too, so your error handling keeps working.

Disclosure: I built it, because I needed it myself.

Step 1: Get a token

Create a free Apify account and copy your API token from Settings → Integrations. The free plan includes $5 of monthly credit, which covers roughly 1,000 searches.

Step 2: Change one line

Before:

GET https://www.googleapis.com/customsearch/v1?key=GOOGLE_API_KEY&cx=ENGINE_ID&q=coffee
~~~

After:

~~~http
GET https://s3nafps--google-custom-search-api.apify.actor/customsearch/v1?token=APIFY_TOKEN&q=coffee
~~~

`key` becomes your Apify token (or send it as an `Authorization: Bearer` header). `cx` is accepted and ignored, so old URLs don't break.

Python:

~~~python
import os, requests

r = requests.get(
    "https://s3nafps--google-custom-search-api.apify.actor/customsearch/v1",
    params={"q": "best espresso machine", "num": 5, "gl": "us"},
    headers={"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"},
    timeout=60,
)
for item in r.json().get("items", []):
    print(item["title"], item["link"])
~~~

JavaScript:

~~~js
const url = new URL('https://s3nafps--google-custom-search-api.apify.actor/customsearch/v1');
url.search = new URLSearchParams({ q: 'best espresso machine', num: '5' });
const res = await fetch(url, { headers: { Authorization: `Bearer ${process.env.APIFY_TOKEN}` } });
console.log((await res.json()).items?.map((i) => i.link));
~~~

## Step 3: Check the differences

A few things work differently, so test before you switch production traffic:

- Results come from google.com, not from a Programmable Search Engine you configured. To limit results to one site, use `siteSearch=example.com`.
- Speed: half of searches answer within about 7 seconds, most within 15. Google's API was faster. Set your client timeout to 60 seconds.
- Image search (`searchType=image`) isn't supported yet and returns a clear 400 error.
- `pagemap` only has `metatags`, and only when you ask for Markdown (below).

## Bonus for AI agents: page text in the same call

Add `markdown=true` and each result also carries the clean text of its page as Markdown, ready for an LLM or a RAG pipeline. That saves a second scraping step:

~~~python
params={"q": query, "num": 5, "markdown": "true", "markdownMaxChars": 4000}
~~~

Some sites block automated visitors, so expect text for about half the pages; the others return `markdownError` and aren't charged.

## Cost

Google charged $5 per 1,000 queries. This replacement charges $1.50 per 1,000 searches plus Apify's platform cost (mostly the Google search proxy, about $2.50 per 1,000), so about $4 per 1,000 in total. Searches that fail aren't charged.

## Lists of searches

For batch jobs, run the actor with a list instead of calling the web address:

~~~json
{ "queries": ["best coffee grinder 2026", "how to make cold brew"], "num": 10 }
~~~

Each search becomes one row in Google's format.

Actor page with all parameters: https://apify.com/s3nafps/google-custom-search-api

If you're migrating and something your code relies on is missing, open an issue on the actor page and I'll look at it.

How to get Aqar.fm property listings into Google Sheets (Riyadh, Jeddah, any Saudi city)

Mohamed Senator — Sat, 17 Oct 2026 01:52:49 +0000

Watch the full video

Aqar.fm is the biggest property site in Saudi Arabia, but it has no export button. If you want to compare rents across Riyadh districts or spot under-priced listings, you end up copying listings by hand.

Here's how to get clean listings into a spreadsheet in about two minutes, with no code.

Step 1: Open the scraper
Go to Aqar.fm Scraper on Apify and click Try for free.

Step 2: Pick your search

City: Riyadh, Jeddah, Dammam, Al Khobar, Mecca, Medina and more
Category: apartments, villas, land or buildings, for rent or for sale
Optional filters: district, price range, area, minimum bedrooms
Max results: how many listings you want

Step 3: Run it and export
Click Start. When the run finishes, open Storage → Dataset and export to CSV or Excel, then import the file into Google Sheets. To refresh automatically, add a schedule (for example, daily at 8:00).

What you get for each listing
English and Arabic title and description, price in SAR, rent period, area (m²), price per m², bedrooms, bathrooms, district, map coordinates, photos, and advertiser type.

The column most people miss is priceVsDistrictMedianPct: how far a listing's price per m² is above or below the median for its district.

Example: in my test, a 203 m² three-bedroom apartment for rent in Al Malqa, Riyadh, came in at SAR 369/m². The district median was SAR 622/m², so it was 40.6% below median. Sort by that column and the deals are at the top.

Cost: you pay per listing, and there's no subscription.

Want alerts instead of exports? Aqar.fm Monitor runs on a schedule and reports only new listings and price drops.

Turn any web page into typed JSON for your AI agent (with a source quote for every value)

Mohamed Senator — Sat, 17 Oct 2026 01:48:24 +0000

Watch the full video

LLM agents are great at reasoning and bad at reading messy HTML. The usual fix is a custom scraper per site, which breaks every time the site changes.

Webpage to JSON for AI Agents takes a URL and a plain-English description of the fields you want, and returns typed JSON. Every value comes with a quote from the page, so you can check it isn't made up.

Input:

{
  "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
  "instruction": "Extract the book title, price as a number, currency code, and whether it is in stock."
}

Output (real run):

{
  "data": { "title": "A Light in the Attic", "price": 51.77, "currencyCode": "GBP", "inStock": true },
  "evidence": { "title": "# A Light in the Attic", "price": "£51.77" },
  "unverifiedFields": []
}

Useful options

JSON schema: pass extractionSchema when you need an exact shape.
List mode: set listMode: true to get many records from one page, such as product grids or job boards.
Strict evidence: requireEvidence is on by default. A value without a source quote is flagged in unverifiedFields.

Call it from code

from apify_client import ApifyClient
client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("s3nafps/webpage-to-json-for-ai-agents").call(run_input={
    "urls": ["https://example.com/pricing"],
    "instruction": "List each plan with name, monthly price and included seats."
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["data"])

Use it as an MCP tool: add https://mcp.apify.com/?actors=s3nafps/webpage-to-json-for-ai-agents to Claude, Cursor or any MCP client, and your agent can call it directly.

The LLM cost is included. You pay $0.03 per page with data and nothing for pages where no data was found.