DEV Community

voiddo
voiddo

Posted on Originally published at voidd0.github.io

Similarweb Scraper: how to get Top Websites by Category as JSON or CSV (Python, API, no code)

Scrape Similarweb top websites rankings by category: global rank, domain, category, rank change, average visit duration, pages per visit, and bounce rate.

This tutorial shows what the Actor v0iddo/similarweb-scraper returns, how to run it on the Apify Store without code, and how to call it from Python and cURL.

What data you get

One row per result. Every row can be downloaded as JSON, CSV, Excel or HTML, or read through the API.

Field Label Type Example
rank Rank number 1
domain Domain text google.com
category Category text Computers Electronics and Technology > Search Engines
categorySlug Category Slug text computers-electronics-and-technology/search-engines
rankingList Ranking List text global
rankChange Rank Change number 0
avgVisitDuration Avg Visit Duration text 00:10:00
avgVisitDurationSeconds Avg Visit Seconds number 600
pagesPerVisit Pages per Visit number 8.68
bounceRate Bounce Rate % number 28.4
url Source URL URL https://www.similarweb.com/top-websites/

Real rows from a run of the Actor (fields that name people are left out):

[
  {
    "rank": 1,
    "domain": "google.com",
    "category": "Computers Electronics and Technology > Search Engines",
    "categorySlug": "computers-electronics-and-technology/search-engines",
    "rankChange": 0,
    "avgVisitDuration": "00:10:00",
    "avgVisitDurationSeconds": 600,
    "pagesPerVisit": 8.68,
    "bounceRate": 28.4,
    "rankingList": "global",
    "url": "https://www.similarweb.com/top-websites/"
  },
  {
    "rank": 2,
    "domain": "youtube.com",
    "category": "Arts & Entertainment > Streaming & Online TV",
    "categorySlug": "arts-and-entertainment/tv-movies-and-streaming",
    "rankChange": 0,
    "avgVisitDuration": "00:18:50",
    "avgVisitDurationSeconds": 1130,
    "pagesPerVisit": 11.45,
    "bounceRate": 23.61,
    "rankingList": "global",
    "url": "https://www.similarweb.com/top-websites/"
  }
]
Enter fullscreen mode Exit fullscreen mode

No code: run it on the Apify Store

  1. Open the Similarweb Scraper page on Apify.
  2. Fill in the input form. An example input is already filled in.
  3. Press Start, wait for the run to finish, and download the results as JSON, CSV or Excel from the Output tab.

The input that is pre-filled on the Actor page:

{
  "categorySlugs": [
    "",
    "computers-electronics-and-technology",
    "arts-and-entertainment"
  ],
  "maxItems": 20
}
Enter fullscreen mode Exit fullscreen mode

Ready-made examples

Saved inputs on Apify: run one as it is, or copy it and change the values.

Python (apify-client)

Get your API token in Apify Console under Settings, then Integrations. The Actor id is v0iddo/similarweb-scraper.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {'categorySlugs': ['',
                   'computers-electronics-and-technology',
                   'arts-and-entertainment'],
 'maxItems': 20}

run = client.actor("v0iddo/similarweb-scraper").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
Enter fullscreen mode Exit fullscreen mode

cURL

curl -X POST "https://api.apify.com/v2/acts/v0iddo~similarweb-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"categorySlugs": ["", "computers-electronics-and-technology", "arts-and-entertainment"], "maxItems": 20}'
Enter fullscreen mode Exit fullscreen mode

Good to know

  • Each Similarweb category page lists up to 50 ranked websites. Use an empty string in categorySlugs to get the global top 50.
  • rankingList names the ranking each row comes from: the category slug you asked for, or global for the global top 50. categorySlug is the site's own category, which is often a subcategory of the list.
  • Rankings are updated monthly by Similarweb and reflect the previous calendar month.
  • Bounce rate, pages per visit, and visit duration may be unavailable for some entries.

Price

$1.70 per 1,000 results. You pay per result row. There is no separate charge per run. Higher Apify plans pay less per result, down to $1.19 per 1,000.

FAQ

What data does the Similarweb Scraper return?

Each result is one row in an Apify dataset. Fields include rank, domain, category, categorySlug, rankingList, rankChange, avgVisitDuration, avgVisitDurationSeconds, and more. The table above shows every field with an example value.

Can I use it without writing code?

Yes. Open the Actor page on Apify, fill in the input form (an example is already filled in), press Start, and download the results as JSON, CSV, Excel or HTML. You can also schedule runs there.

How much does it cost?

$1.70 per 1,000 results. You pay per result row. There is no separate charge per run. Higher Apify plans pay less per result, down to $1.19 per 1,000. A new Apify account gets $5 of platform credit every month.

Links

Top comments (0)