DEV Community

agenthustler
agenthustler

Posted on • Edited on

100 free API calls: scrape Bluesky, Substack, and HN with one curl command

#apify #python #webdev #api

Need structured data from Bluesky, Substack, or Hacker News? We built a free-tier API that returns clean JSON — no browser automation, no rate limit fighting, no scraper maintenance.

You get 100 free calls just by registering with your email.

Step 1: Get your free API key 

curl -s -X POST https://web-data-labs.com/api/register \
  -H 'Content-Type: application/json' \
  -d '{"email": "you@example.com"}'
Enter fullscreen mode Exit fullscreen mode

Response:

{
  "api_key": "dc_xxxxxxxxxxxxxxxxxxxx",
  "calls_left": 100,
  "message": "API key created",
  "usage": "Authorization: Bearer dc_xxxxxxxxxxxxxxxxxxxx"
}
Enter fullscreen mode Exit fullscreen mode

Save that key. You get 100 calls free, no credit card, no OAuth dance.

Step 2: Search Hacker News 

curl -s -X POST https://web-data-labs.com/api/hn/search \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"query": "fastapi", "maxItems": 5}'
Enter fullscreen mode Exit fullscreen mode

Returns an array of stories with title, URL, score, comment count, author, and HN link:

[
  {
    "id": 47384352,
    "title": "Building async APIs with FastAPI",
    "url": "https://example.com/fastapi-guide",
    "author": "devuser",
    "score": 136,
    "commentCount": 31,
    "createdAt": "2026-03-15T04:23:43.000Z",
    "storyType": "story",
    "hnUrl": "https://news.ycombinator.com/item?id=47384352"
  }
]
Enter fullscreen mode Exit fullscreen mode

Step 3: Search Bluesky 

curl -s -X POST https://web-data-labs.com/api/bluesky/search \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"query": "python", "maxItems": 5}'
Enter fullscreen mode Exit fullscreen mode

Bluesky's AT Protocol returns structured posts — author handle, text, like/repost counts, indexed timestamp.

Step 4: Search Substack 

curl -s -X POST https://web-data-labs.com/api/substack/search \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"query": "machine learning", "maxArticles": 5}'
Enter fullscreen mode Exit fullscreen mode

Returns newsletter posts with title, author, publication name, subtitle, and post URL. Good for tracking newsletter trends or building a reading aggregator.

Python example

Here's a minimal script that searches all three sources at once:

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).
Enter fullscreen mode Exit fullscreen mode

Use cases

  • Content aggregator: Monitor what's trending across Bluesky + HN simultaneously
  • Newsletter research: Find what Substack authors are writing about a topic before you write yours
  • Social listening: Track mentions of your project/library across platforms
  • Dataset building: Collect labeled text for fine-tuning (check each platform's ToS)
  • Competitor intel: See what devs are sharing about tools in your space

What powers it

The backend runs on Apify actors — pre-built scrapers maintained by the community. We wrap them in a FastAPI service with key-based auth and usage tracking. When you hit your 100 free calls, paid plans start at $0.05/call (crypto micropayments via x402 protocol).

Discovery endpoints

The API is also MCP-compatible (Model Context Protocol) — AI agents can call it directly:

curl -s https://web-data-labs.com/.well-known/mcp.json
Enter fullscreen mode Exit fullscreen mode

And it supports the x402 payment standard for agent-to-agent billing:

curl -s https://web-data-labs.com/.well-known/x402-discovery
Enter fullscreen mode Exit fullscreen mode

Register at https://web-data-labs.com/api/register — 100 calls, no card, one curl.

Built with FastAPI + Apify. Questions or feedback welcome in the comments.

Disclosure: This post contains affiliate links. I may earn a commission if you sign up through my links, at no extra cost to you.

Need a reliable scraping API? ScraperAPI handles proxies, CAPTCHAs, and browsers so you don't have to. Get 5,000 free API credits with code SCRAPE13833889.

Disclosure: This post contains affiliate links. I may earn a commission if you sign up through my links, at no extra cost to you.

Compare web scraping APIs:

  • ScraperAPI — 5,000 free credits, 50+ countries, structured data parsing
  • Scrape.do — From $29/mo, strong Cloudflare bypass
  • ScrapeOps — Proxy comparison + monitoring dashboard

Need custom web scraping? Email hello@web-data-labs.com — fast turnaround, fair pricing.

📘 Get the Complete Web Scraping Playbook

Want the full guide? The Complete Web Scraping Playbook 2026 — 48 pages covering proxies, anti-bot bypass, stealth browsers, and production-ready architectures. Just $9.

Top comments (0)