DEV Community

drew
drew

Posted on Edited on

Scrape Bluesky posts and profiles without an API key

#bluesky #webscraping #api #tutorial

Bluesky is unusually open: almost everything public is available through the AT Protocol's public AppView API at public.api.bsky.app / api.bsky.app, with no login. (Tested 30 Oct 2026: post search returned 403 on public.api.bsky.app but worked on api.bsky.app. Profiles and feeds worked on both.) Here's how to use it directly, and how to get the same data with no code at all.

Disclosure: I built the Apify Actor shown in part 2.

Part 1: The raw API (no key needed)

Search recent posts:

curl "https://api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=web%20scraping&sort=latest&limit=5"
Enter fullscreen mode Exit fullscreen mode

Look up a profile:

curl "https://public.api.bsky.app/xrpc/app.bsky.actor.getProfile?actor=bsky.app"
Enter fullscreen mode Exit fullscreen mode

Get an account's posts:

curl "https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed?actor=bsky.app&limit=10"
Enter fullscreen mode Exit fullscreen mode

This works fine for one-off calls. You'll start writing plumbing once you need cursor pagination, rate-limit backoff, flattening nested record/embed objects, extracting links and hashtags from facets, and remembering which posts you've already seen.

Part 2: No-code with the Bluesky Scraper Actor

Bluesky Scraper on Apify handles that plumbing. Pick a mode (Search posts, Search profiles or Posts from accounts) and enter queries. Bluesky's search syntax works, e.g. from:handle, lang:en and #hashtags.

Input from a real run:

{
  "mode": "searchPosts",
  "queries": ["web scraping"],
  "maxItems": 20,
  "sort": "latest",
  "outputFormat": "both"
}
Enter fullscreen mode Exit fullscreen mode

That run finished in about 3 seconds. One of the items, exactly as returned:

{
  "type": "post",
  "url": "https://bsky.app/profile/hugovalters.bsky.social/post/3mz3rucjn3h2o",
  "text": "Fill web forms from a JSON file using Playwright. 136 lines, intermediate level, takes a screenshot after submit so you can verify. Built for QA and smoke tests. #python #playwright #scraping https://www.valtersit.com/python/playwright-browser-automation-for-form-filling/",
  "createdAt": "2026-10-30T12:30:00.600753Z",
  "authorHandle": "hugovalters.bsky.social",
  "likeCount": 0,
  "repostCount": 0,
  "replyCount": 0,
  "quoteCount": 0,
  "links": ["https://www.valtersit.com/python/playwright-browser-automation-for-form-filling/"],
  "imageCount": 0,
  "isReply": false,
  "query": "web scraping",
  "markdown": "## Post by @hugovalters.bsky.social\n- **url:** https://bsky.app/profile/hugovalters.bsky.social/post/3mz3rucjn3h2o\n- **date:** 2026-10-30T12:30:00.600753Z\n- **likes:** 0 ..."
}
Enter fullscreen mode Exit fullscreen mode

outputFormat: "both" adds the markdown field, a compact version that's easy to drop into an LLM prompt or a RAG index.

Monitoring a keyword or brand

Turn on incremental mode and add a schedule (for example, hourly). Each run returns only posts it hasn't returned before, so you can wire a webhook to Slack, a Google Sheet or your own endpoint without dealing with duplicates.

Exporting

Dataset → Export → CSV/Excel/JSON, or call the run's dataset API from your code.

Cost

$0.003 per run plus $0.001 per result, so 1,000 posts cost about $1. Small tests fit in Apify's free monthly credit.

Be a good citizen

It's public data, but it's still people's posts. Respect deletions, don't build profiles on individuals, and follow Bluesky's developer guidelines and UK/EU GDPR if you store personal data.

What would you monitor on Bluesky? Let me know in the comments.

Top comments (0)