DEV Community

Cover image for How to Scrape note.com Articles and Creators by Hashtag with Python
ReventaData
ReventaData

Posted on Originally published at apify.com

How to Scrape note.com Articles and Creators by Hashtag with Python

#python #webscraping #apify #automation

By the end of this tutorial, you will have a short Python script that turns a note.com hashtag into structured rows: article title and URL, publication time, price in JPY, free, paid or membership flags, likes, and the creator's handle, name and X account. With one switch on, each row also gets the readable text, the paywalled character count and the creator's public profile.

note.com is a Japanese creator and blog platform. In this guide, we use Python and the Apify Actor note.com Hashtag Scraper (reventadata/note-scraper). You will configure a hashtag, run the Actor and filter rows by price, likes, date and keyword. The sample row below is a sample from a capture dated 2026-10-22, parsed with the Actor's own parser.

⏱️ Time estimate: one sitting: install one package, set one token, run three short Python blocks.

Prerequisites

To follow this tutorial, you will need:

  1. Python 3.11+ (apify-client 3.3.0 requires it).
  2. The Apify API client for Python.
  3. An Apify account and its API token, which sits on the API & Integrations page in Apify Console.

Install the client:

pip install "apify-client~=3.0"
Enter fullscreen mode Exit fullscreen mode

The code reads the token from an environment variable:

export APIFY_API_TOKEN="your_apify_token_here"
Enter fullscreen mode Exit fullscreen mode

What you'll achieve

You will turn one or more note.com hashtags into one row per article. Each row carries id, key, title and url, the tag and list (listMode) it was found on, publishedAt in Japan time, price in JPY, isFree, hasSubscription, isPaywalled, likeCount, and the creator's creatorUrlname, creatorName and creatorTwitter.

Note: The Actor reads only the public hashtag pages and article pages a logged-out browser sees. Per its README, no emails are extracted; the rows carry the URLs, X handle and external links a creator publishes.

Example use case: Suppose you want free AI articles with at least 100 likes, published since October. You would set hashtags: ["AI"], freeOnly: true, minLikes: 100 and publishedAfter: "2026-10-01", and every saved row would pass all three.

Step 1: Access note.com Hashtag Scraper

Go to the note.com Hashtag Scraper page on Apify Store. The README lists every input and output field. In this tutorial we drive this Apify Actor from Python, so everything after this step happens in code.

Step 2: Configure the hashtags

The main input is hashtags: tags as written on note.com (AI, #日記) or tag URLs such as https://note.com/tag/ChatGPT. Two more inputs decide how much each tag yields:

  • mode: popular (the tag page as it opens), new (newest first) or both (two pages per tag, duplicates removed). Default both.
  • expandRelatedTags: 0 to 10. Each tag page lists up to 10 related keywords (the AI page lists AGI, AI Agent, AI Act and others); this many of them are read too.

Why the second one? The Actor reads one page per tag and list: 20 articles on tags served as a search list, about 40-50 on large tags served as a timeline list. The site pages further only through its JSON API, which note.com's robots.txt disallows, so related tags are how you reach more rows.

Here is the setup. It reads AI (popular and new), opens each article page and saves at most 10 rows:

import os
from apify_client import ApifyClient

# 1. Initialize ApifyClient with the token from the environment
client = ApifyClient(os.environ["APIFY_API_TOKEN"])

# 2. Define the input: one hashtag, both lists, article pages on, at most 10 rows
run_input = {
    "hashtags": ["AI"],
    "mode": "both",
    "includeDetails": True,
    "maxItems": 10,
}
Enter fullscreen mode Exit fullscreen mode

💡 Tip: paste hashtags exactly as note.com shows them. Japanese tags work as written; # and /hashtag/ URLs are accepted too.

No proxy setting is needed in run_input: when none is given, the Actor uses Apify residential proxy in Japan, the transport that returned full pages in testing on 2026-10-22.

Step 3: Add optional filters

These filters run on the rows after the page is read. Rows a filter drops never reach your dataset:

Parameter Type What it keeps
keywords array Articles whose title contains any term (with includeDetails, also the text, description or creator bio). Case- and width-insensitive: ＡＩ matches AI.
excludeKeywords array Drops articles containing any term in the same fields.
freeOnly boolean Articles the page marks free: no price, no membership gate.
minLikes integer Articles with at least this many likes.
publishedAfter string YYYY-MM-DD (read as Japan time) or an ISO timestamp.
includeDetails boolean Not a filter: opens each article page for text, paywall count, categories and the creator profile.

Three combinations that work:

  • Newest posts under a tag: mode: "new", no filters.
  • Popular free reads: mode: "popular", freeOnly: true.
  • Creator research: includeDetails: true, minLikes: 500, to get follower counts and bios of well-liked authors.

⚠️ A mode outside both, popular and new is refused before the run starts. A publishedAfter that is not a date stops the run with an error at its start.

Step 4: Run the Actor

Now start the run with client.actor("reventadata/note-scraper").call(run_input=run_input). The call starts the Actor on Apify and waits for it to finish before returning the run:

# 3. Start the Actor run and wait for it to finish
print("Starting note.com extraction...")
run = client.actor("reventadata/note-scraper").call(run_input=run_input)
if run is None or run.status != "SUCCEEDED":
    status = run.status if run else "unknown"
    raise SystemExit(f"Run did not succeed (status: {status}).")

# 4. Fetch the saved articles
items = client.dataset(run.default_dataset_id).list_items().items
print(f"Retrieved {len(items)} articles.\n")
Enter fullscreen mode Exit fullscreen mode

Behind the scenes:

  • The Actor opens https://note.com/tag/AI and, for new, the same URL with ?mode=new. With includeDetails, it also opens each article at https://note.com/<creator>/n/<key>. Those are the only two URL shapes it requests; /api/* and /search are robots-disallowed and never called.
  • note.com sometimes answers a tag page without its article list. The Actor counts the articles it parsed, and on zero it retries on a fresh proxy exit.
  • It stops at maxItems or when the tags run out.

Step 5: View and download results

Loop over the rows to print the parts you need:

for item in items:
    print(item["title"])
    print(f"  URL: {item['url']}")
    print(f"  Creator: {item.get('creatorName')} (@{item.get('creatorUrlname')})")
    if item.get("isFree"):
        print("  Free to read")
    else:
        print(f"  Price: {item.get('price')} JPY, membership: {item.get('hasSubscription')}")
    print(f"  Likes: {item.get('likeCount')}, published: {item.get('publishedAt')}")
    if item.get("detailsStatus") == "ok":
        print(f"  Followers: {item.get('creatorFollowers')}, paywalled chars: {item.get('paywalledChars')}")
    print("-" * 40)
Enter fullscreen mode Exit fullscreen mode

Here is one row, a sample from a capture dated 2026-10-22 (the AI tag's popular list plus that article's page), parsed with the Actor's own parser. Null and empty fields, bodyHtml, the two image URLs and scrapedAt (added at save time) are left out:

{
    "id": 185275155,
    "key": "n6ccbac30a263",
    "title": "【朝メモ】 Meta 新AIモデル発表",
    "url": "https://note.com/goto_finance/n/n6ccbac30a263",
    "tag": "AI",
    "listMode": "popular",
    "publishedAt": "2026-10-13T06:59:35.000+09:00",
    "price": 0,
    "currency": "JPY",
    "isFree": false,
    "hasSubscription": true,
    "isPaywalled": true,
    "canRead": false,
    "likeCount": 750,
    "commentCount": 10,
    "imageCount": 3,
    "format": "4.0",
    "creatorUrlname": "goto_finance",
    "creatorName": "後藤達也",
    "creatorUrl": "https://note.com/goto_finance",
    "creatorTwitter": "goto_finance",
    "bodyText": "おはようございます。\n米国はコロンバスデーで、債券市場は休場でした。株式市場ではメモリ関連株が下落したものの、S&P500はほぼ横ばいでした。\n\n今朝はMetaの新しいAIと、今晩から本格化する米企業決算についてまとめます。\n◆ Meta、AI戦略に変化",
    "bodyChars": 127,
    "paywalledChars": 1022,
    "anonymousLikeCount": 8,
    "creatorFollowers": 94430,
    "creatorFollowing": 19,
    "creatorNoteCount": 1136,
    "creatorProfile": "経済ニュースを「わかりやすく、おもしろく」。経済や投資になじみのない方を念頭に、偏りのない情報発信を目指しています。国民の金融リテラシーの健全な向上に少しでも貢献できればと思っています。YouTube、Twitter、noteが3本柱。プラットフォーム間のシナジーも探ります。",
    "creatorSocials": {
        "twitter": "goto_finance"
    },
    "creatorCreatedAt": "2022-03-06T04:00:44.000+09:00",
    "detailsStatus": "ok"
}
Enter fullscreen mode Exit fullscreen mode

Apify datasets export as JSON, CSV, XML, Excel, HTML table, RSS or JSONL.

Understanding your results

The fields that matter most:

  • price, isFree, hasSubscription: price is the single-purchase price in JPY, 0 when there is none. A price of 0 is not "free": the sample row costs 0 but sits behind a membership, so isFree is false. In the 20-article AI popular capture, all 20 were not free and 19 were membership articles.
  • isPaywalled / canRead: whether a logged-out visitor can read the whole article, as the page flags it.
  • paywalledChars and bodyChars (details): characters behind the paywall and characters you can read. The sample shows 127 readable and 1,022 behind the paywall.
  • likeCount, commentCount: the page's counts. Large tags served as a timeline list carry likes only; commentCount is null there.
  • creatorFollowers, creatorNoteCount, creatorProfile (details): the creator's public profile. A creator who hides follow counts gets null, not 0.
  • detailsStatus: ok, skipped (details off), redirect (custom-domain creator: listing fields only) or failed.

Pro tips

  1. Use both for range. In the AI captures from 2026-10-22, the popular list (likes 174 to 1,272) and the newest list (likes 0 to 2, all posted within two minutes) shared no article.
  2. Grow with related tags. One page per tag and list is all the site's robots-allowed pages give. expandRelatedTags: 5 reads five related tags from each seed tag's page.
  3. Filter on the title first. Without includeDetails, keywords checks only the title. Turn details on and the text and bio count too, at one extra request per article.
  4. Batch tags in one run. Put several in hashtags. An article kept under one tag is not saved again under the next.
  5. Start small. Keep maxItems: 10 until your downstream code handles the fields, then raise it.
  6. Read the membership flag, not the price. In the AI popular capture, 15 of 20 articles had price 0, and all 15 were membership articles that a logged-out visitor cannot read in full.

Troubleshooting

  • Run FAILED with zero rows: a run ends FAILED when a tag page cannot be read or when every tag is unknown on note.com (HTTP 404, check the spelling). A run that saves no rows ends FAILED even when the tag pages loaded normally and your filters dropped every article. A single unknown tag among several is logged and skipped.
  • Run stops at once: no entry in hashtags was a tag or a note.com tag URL, or publishedAfter is not a date.
  • Fewer rows than expected: read the summary line, Run finished: N tag pages and N article pages fetched, N rows pushed. Tally: ...; it counts each filter's drops (not_free, too_few_likes, too_old, keyword_mismatch). From Python: client.log(run.id).get().

For help, use the Actor's Store page.

Pricing and next steps

For current pricing and details, see the Store page.

To repeat a search, save the input as a task and add an Apify schedule, which runs an Actor or task on a cron expression (0 8 * * * is every day at 8 AM). To react when a run finishes, add a webhook on ACTOR.RUN.SUCCEEDED; its payload carries the run object, including defaultDatasetId.

Which hashtags are you planning to track on note.com? Drop a comment! 💬

Top comments (0)