DEV Community

Cover image for Reddit 403'd our stealth browser, so we stopped scraping reddit.com entirely
Simon
Simon

Posted on Originally published at crawlforge.dev

Reddit 403'd our stealth browser, so we stopped scraping reddit.com entirely

Ask any scraper developer what the hardest mainstream site is in 2026 and Reddit is a safe answer. We would know — reddit.com blocks every direct access path CrawlForge has, from a plain fetch to a full stealth browser.

So CrawlForge MCP v5.1.0 stops fighting the wall and goes around it: the new reddit_search tool — our 28th — searches Reddit posts and comments and reads entire threads through community-run archives, with no Reddit API key, no OAuth app, and no credentials of any kind.


TL;DR: v5.1.0 adds reddit_search: search Reddit posts and comments, or read a full nested comment thread, without ever touching reddit.com. It queries two community archives — Arctic Shift (near-real-time) and PullPush (Pushshift-compatible full-text search) — so there is nothing to sign up for on Reddit's side. Three modes, 2 credits per call. Upgrade: npm install -g crawlforge-mcp-server@latest.

Table of contents

What shipped

v5.1.0 is a focused minor release with one headline:

  • reddit_search — search Reddit posts (title + selftext) and comments, or read a post plus its nested comment tree, via the Arctic Shift and PullPush community archives. Three modes, scoped filters, normalized output. 2 credits per call.
  • Tool count goes from 27 to 28, and the MCP server card, tool instructions, and getting-started prompt all know about the newcomer.
  • 31 new unit tests (1,018 total in the suite) and 100% MCP protocol compliance across all 28 tools, live-verified over MCP stdio returning real r/ClaudeAI posts.

Why reddit.com cannot be scraped

We did not reach for archives out of preference — we measured the wall first. Before building the tool we ran every direct path against reddit.com, live:

  • fetch_url with a full browser User-Agent: 403
  • scrape against old.reddit.com: 403
  • scrape_template with the reddit-thread template: 403
  • stealth_mode at its most advanced level: 403

The block is layered — datacenter IP reputation first, TLS fingerprinting at the handshake, then a JavaScript challenge — which is why even a stealth browser on a server fails.

And the sanctioned door narrowed too: Reddit closed self-service API signup in November 2025 under its Responsible Builder Policy, so new applications for official API credentials go through an approval request rather than a signup form. Most Reddit MCP servers wrap that official API, which now makes "get credentials" the hardest step of their setup.

reddit_search sidesteps all of it. It never sends a single request to reddit.com.

One tool, three modes

  • posts (default) — keyword search across post titles and selftext, optionally scoped to a subreddit or author. Supports "quoted phrases", OR, and -exclusion.
  • comments — full-text search across comment bodies, something the official Reddit API cannot do at all.
  • thread — hand it a post ID and get the post plus its nested comment tree, with reddit-style collapse markers for branches too deep to expand.

Filters cover subreddit, author, after/before dates (ISO 8601, epoch seconds, or offsets like "7d"), limit (up to 100), and sort. Results come back normalized: full reddit.com permalinks, ISO dates, scores, comment counts, and text capped at 2,000 characters with truncation flags so a 100-result payload stays friendly to an LLM context window.

A typical call from an MCP client looks like this:

{
  "tool": "reddit_search",
  "arguments": {
    "query": "best mcp servers",
    "subreddit": "ClaudeAI",
    "mode": "posts",
    "limit": 10
  }
}
Enter fullscreen mode Exit fullscreen mode

Take any post id from the results, pass it back as link_id with mode: "thread", and you have the whole discussion — ready for summarize_content or analyze_content to turn into sentiment, topics, and a briefing.

The archives: Arctic Shift and PullPush

If you remember Pushshift — the beloved Reddit research archive that lost its API access in 2023 — these two projects are its successors, and they are what reddit_search queries:

  • Arctic Shift ingests Reddit in near-real-time. During live testing it returned a post created the same day, and it serves proper nested comment trees. Its one documented constraint: keyword search must be scoped to a subreddit or author.
  • PullPush is Pushshift-compatible and does what Arctic Shift will not — full-text search across all of Reddit. The trade-offs: documented gaps in its post-2023 archive and aggressive rate limits.

Both are free, both are community-run, and neither needs credentials. Every response includes provenance notes naming the archive that answered, so downstream consumers always know where the data came from.

How routing works

You never pick a backend unless you want to. In the default auto mode:

scoped search (subreddit/author)  --+--> Arctic Shift (fresher)
thread read                       --+      |
                                           +--> error? --> PullPush
                                                           (fallback_used: true)

unscoped full-text search  ----------> PullPush
                                       (only archive that supports it)
Enter fullscreen mode Exit fullscreen mode

The plumbing absorbs the archives' quirks: every request carries an identifying User-Agent (live testing showed Arctic Shift throttles anonymous clients into a shared bucket), transient throttle responses get one bounded retry, and PullPush's rate-limit messages pass through verbatim so you see the real reason instead of a generic error. A REDDIT_SEARCH_TIMEOUT_MS environment variable overrides the 30-second per-request cap if your pipeline needs a different budget.

Honest caveats

Archive data has edges, and we would rather document them than let you find them:

  • Very fresh content reads low. The archives capture posts the moment they appear, so scores and comment counts of content less than ~36 hours old often read 0 or 1. The content is there; the vote tallies have not caught up.
  • PullPush has post-2023 gaps. An empty unscoped search does not prove the content does not exist. Scope to a subreddit or author when you can — it routes you to the more complete archive.
  • Deleted content may persist in archives. That is inherent to how archives work, and it cuts both ways: it is also why researchers use them.

Using it from MCP and REST

In an MCP client — Claude Desktop, Claude Code, Cursor — just ask: "Search r/webdev for discussions about scraping infrastructure from the last month and summarize the top complaints." The agent picks reddit_search, scopes it, and chains the summary itself.

From the REST API, it is one authenticated POST:

const response = await fetch('https://crawlforge.dev/api/v1/tools/reddit_search', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    query: 'best mechanical keyboard',
    subreddit: 'MechanicalKeyboards',
    mode: 'posts',
    limit: 10,
  }),
});

const { data } = await response.json();
console.log(`${data.count} posts via ${data.source}`);
Enter fullscreen mode Exit fullscreen mode

You can also try it with zero setup in the playground — reddit_search is live there alongside the other 27 tools.

What about the Reddit thread template

scrape_template still ships its reddit-thread template, but it points at reddit.com — the very wall this release routes around — so it no longer works reliably. The scrape_template docs now say exactly that and point Reddit work at reddit_search instead. The other nine templates (Amazon, GitHub, YouTube, Hacker News, and friends) are unaffected.

Also since v5.0.4
If you skipped the patch between releases, v5.0.5 fixed two numbers serp_rank had been asserting wrongly: the DataForSEO request timeout (now 120 seconds by default, tunable via DATAFORSEO_TIMEOUT_MS, because live Google scrapes at depth 100 routinely take 30-60+ seconds) and the documented upstream cost (the old figure was the depth-10 price, 10× lower than the depth-100 default actually bills). It also fixed the Smithery listing, which now derives its tool table from the live registry instead of a hand-written card. The full story of the v5.0.x hardening cycle is in the v5.0.4 live-testing post.

Credit cost

reddit_search costs 2 credits per call — search or full thread read, same price. For scale: the free plan's 1,000 starter credits cover 500 Reddit searches, and a search plus a thread read plus summarize_content on the result is 8 credits end to end. As always, failed requests are not charged.

How to upgrade 

npm install -g crawlforge-mcp-server@latest
crawlforge --version   # 5.1.0
Enter fullscreen mode Exit fullscreen mode

Or, if your MCP client launches the server with npx, it picks up 5.1.0 automatically on the next restart. No schema, output-shape, or credit-cost changes to any existing tool — this is a drop-in upgrade. The package is crawlforge-mcp-server on npm.

Ready to mine Reddit discussions without an API key? The free plan's 1,000 credits are enough for 500 searches, and the reddit_search API reference covers every parameter and mode.

Start free with 1,000 credits

Top comments (0)