How I built a Bluesky scraper using the AT Protocol API (and published it on Apify)

Daniel Ainsworth — Wed, 27 May 2026 18:37:19 +0000

Bluesky hit 40 million users earlier this year, and unlike Twitter, it runs on an open protocol — the AT Protocol — where public data is genuinely public and machine-readable by design. No $5,000/month enterprise API tier. No rate limits you need a lawyer to understand. Just a clean REST API that anyone can query.

I wanted to scrape it. Here's how I built a production-ready actor and what I learned along the way.

Why Bluesky is easy to scrape (legitimately)

Most social media scrapers are a fight against Cloudflare, rotating proxies, and terms of service grey areas. Bluesky is different. The AT Protocol was explicitly designed for third-party clients and data access. The public API at public.api.bsky.app serves unauthenticated read requests. There's no fingerprinting, no CAPTCHA, no DOM parsing.

The only wrinkle: the search endpoint (app.bsky.feed.searchPosts) now requires authentication via a free App Password. Everything else — author feeds, threads, profiles — works without a token.

The three modes I built

I wanted one actor that covered the main B2B use cases:

Search posts — keyword and hashtag search with date range, language filter, and sort order. Uses bsky.social/xrpc/app.bsky.feed.searchPosts with a Bearer token.

Author feed — pull all posts from one or more handles. No auth needed. Useful for competitor monitoring or auditing a creator's content history.

Thread — fetch a full conversation tree from a post URL. The API returns a nested tree; I flatten it depth-first so you get a clean ordered list of posts.

The one gotcha: API routing

This burned me. I was sending authenticated requests (with a JWT) to public.api.bsky.app. That endpoint is Cloudflare-fronted and returns 403 if you send auth tokens to it — it's for unauthenticated traffic only.

The fix: authenticated calls go to bsky.social. Unauthenticated reads go to public.api.bsky.app. You auth against bsky.social, get a JWT, then use that JWT only on subsequent bsky.social calls.

Monorepo deployment headache

I'm building a portfolio of Apify actors in a TypeScript monorepo with npm workspaces. The shared library (@apify-actors/shared) contains PPE charging helpers and error classes. Locally, workspace resolution handles it cleanly. On Apify's build servers, there's no monorepo — just the uploaded actor folder.

The solution: copy the shared source into src/shared/ inside each actor and use relative imports. tsup bundles it all into a single dist/main.js. The shared code stays in one canonical place in the repo; each actor gets its own copy baked in at build time.

Output schema

Every post comes back as a flat JSON record:

{
  "url": "https://bsky.app/profile/user.bsky.social/post/3lhxxxxxxxxx",
  "text": "Post content here",
  "authorHandle": "user.bsky.social",
  "authorDisplayName": "User Name",
  "likeCount": 142,
  "repostCount": 28,
  "replyCount": 19,
  "images": [{ "thumb": "...", "fullsize": "...", "alt": "..." }],
  "externalEmbed": { "uri": "...", "title": "...", "description": "..." },
  "createdAt": "2025-11-15T10:30:00.000Z"
}

Export as JSON, CSV, or Excel directly from Apify. Plug into Zapier or Make for no-code workflows.

The actor is live

If you want to use it without building anything: Bluesky Posts Scraper on Apify Store

PPE pricing: $0.25 per run + $0.003 per post ($3/1,000). No subscription.

The AT Protocol makes Bluesky one of the cleanest data sources you can work with right now. If your use case involves social listening, brand monitoring, or lead gen signals from a fast-growing tech-forward audience, it's worth adding to your stack.