Stack Exchange tells you exactly how long to wait — most scrapers ignore it

Quick answer

The official Stack Overflow data dump has been paused since mid-2025, and even active it was a quarterly, multi-gigabyte XML export you had to ETL yourself before it was useful. The live Stack Exchange API (api.stackexchange.com/2.3) is the fresher alternative, but it caps unauthenticated traffic at 300 requests/day and tells every client, on every page, exactly how many seconds to wait before the next request — a signal most naive scrapers ignore. The Stack Exchange Questions Scraper reads that signal, paces itself against it, and returns 16 typed fields per question at $0.0015 per row plus a $0.005 run-start charge — $1.505 for 1,000 questions.

The API tells you the exact backoff — we obey it instead of guessing 🚦

Most rate-limit handling on the open web is reactive: you get a 429, you back off, you retry, you hope your guessed delay was long enough. Stack Exchange's API does something more precise — its JSON response body carries an optional backoff field, a number of seconds the server wants you to wait before your next request, sent proactively rather than as a punishment after you've already been throttled.

The scraper reads that field on every page and sleeps for exactly what it says, plus a one-second guard margin, before issuing the next request — not a fixed delay, not an exponentially-doubling guess, but the number the API itself hands back. Ignore that field and you'll eventually get throttled anyway; honor it and you stay inside quota without ever seeing a 429 in the first place.

300 requests a day, unauthenticated, is a real ceiling — not a formality 🔑

Every unauthenticated IP gets 300 API requests per day, and each page of 100 questions costs one request. That's 30,000 questions/day at the API's page-size ceiling before you hit the wall — plenty for a tag-scoped pull, thin for a broad multi-tag corpus. The Actor's apiKey input field passes a free key from stackapps.com straight through to the request, which lifts the same ceiling to 10,000 requests/day — over 30x more headroom, at zero cost, thirty seconds of signup. We don't hide this ceiling behind a vague "may be rate-limited" warning: the README states the exact numbers, because a contributor budgeting a 50,000-question pull needs the real math, not a euphemism.

Deleted accounts leave real gaps in the owner fields, and we pass that through honestly 👤

owner_user_id and owner_display_name come back null for questions asked by accounts that have since been deleted — a routine, ongoing fact of the Stack Exchange dataset, not a scraping defect. The parser doesn't infer a placeholder name or backfill from a cached snapshot; it reports exactly what the live API says about that question right now. If your pipeline joins on owner_user_id, expect a small, real null rate on any sufficiently large or old pull.

What we handle for you 🛡️

• We rotate browser fingerprints — curl-cffi impersonation across Chrome, Firefox, and Safari TLS profiles.
• We honor the API's own backoff signal, sleeping exactly what the server requests before the next page — proactive pacing, not reactive retries.
• We retry with exponential backoff on 408 / 429 / 5xx beyond that, up to 5 attempts per page, honoring Retry-After.
• We rotate residential proxies through Apify Proxy when enabled — fresh session and exit IP on every block.
• We keep the dataset clean — Pydantic-validated rows, ISO-8601 timestamps, CC BY-SA attribution fields on every row.
• You pay only for questions that land. No data, no charge, beyond the small run-start fee.

Full output schema 📦

Sixteen fields per question:

Field	Type	Notes
question_id	int	Stable across the whole Stack Exchange network
site	string	Site slug the question came from
title	string	Question title
body_html	string \	null
tags	list[string]	Tags applied to the question
score	int	Net upvotes minus downvotes
view_count	int	Question views
answer_count	int	Number of answers
is_answered	bool	Has an accepted or positive-score answer
accepted_answer_id	int | null	Accepted answer ID, when present
link	string	Canonical question URL
owner_user_id	int | null	Null when the asker's account was deleted
owner_display_name	string | null	Null when the asker's account was deleted
creation_date	int	Unix timestamp
posted_at	string	ISO-8601 UTC, derived from creation_date
scraped_at	string	ISO-8601 UTC row-creation timestamp

Who this is for

RAG corpus pipelines — feed tag-scoped Q&A bodies into a vector store for a domain-specific copilot, with attribution fields intact.

DevRel signal-tracking — diff unanswered questions in your product's tag daily to catch support gaps before they churn users.

Competitor-tag intelligence — compare question volume and score trends across react vs. vue vs. angular over time.

AI training corpora — a fresh, deduped alternative to the paused SO data dump, with the CC BY-SA attribution fields required for redistribution.

Frequently asked questions

Why use this instead of the Stack Overflow data dump?

The official dump has been paused since mid-2025, and even active releases were quarterly, unfiltered by tag, and shipped as multi-gigabyte XML requiring your own ETL. This Actor returns fresh, tag- or query-filtered questions as clean typed rows in minutes.

How does the Actor decide how long to wait between requests?

It reads the backoff field Stack Exchange's API returns in the response body — a server-specified number of seconds — and sleeps that long before the next page, rather than guessing a fixed or exponential delay.

Why is the daily quota so low without an API key?

Stack Exchange caps unauthenticated usage at 300 requests/day per IP. A free key from stackapps.com lifts that to 10,000 requests/day at no cost.

Why are some owner_user_id and owner_display_name fields null?

The asker's account has been deleted. The Stack Exchange API returns null for those fields on any question, current or historical — we pass that through as-is rather than inferring a placeholder.

What does 10,000 questions cost?

$15.005 — 10,000 × $0.0015, plus the $0.005 run-start charge.

Try it

Live on the Apify Store: Stack Exchange Questions Scraper.

Point it at any Stack Exchange site by tag, search query, or user, and get back typed rows with the attribution fields CC BY-SA 4.0 requires. Pay-per-event, no subscription.

---

Built by Devil Scrapes — we build scrapers for the targets that fight back.