DEV Community: Devil Scrapes

Equity Crowdfunding Leads: scrape 4,800+ Wefunder founders for $5/1K

Devil Scrapes — Sun, 31 May 2026 12:08:13 +0000

Quick answer: There is no unified API for Wefunder, Republic, or StartEngine. An equity crowdfunding leads scraper collects currently-raising and recently-funded campaign data — founder names, company taglines, raise progress, pre-money valuations — from all three platforms and returns them as one normalized dataset. The Apify Actor below does it for $0.005 per row (~$5.05 per 1,000), with the TLS fingerprinting, proxy rotation, and per-source parsing handled for you.

Wefunder alone lists 4,800+ currently-raising companies — founder names, taglines, raise totals, and pre-money valuations in one JSON payload. Republic has a trending carousel. StartEngine has an XML sitemap of 98 offering slugs. None has a download button; none shares a schema.

If you're a VC scout, an SDR targeting founders, or an analyst tracking what's raising in climate versus fintech, you're opening three browser tabs and copy-pasting. Here's what it takes to do that programmatically — and how I compressed it to one API call.

What is equity crowdfunding? 🔎

Equity crowdfunding under Regulation CF lets any US startup raise up to $5 million per year from the general public — not just accredited investors. The three dominant platforms are Wefunder (largest by volume), Republic (curated campaigns), and StartEngine (heavy on CPG and consumer brands).

Each platform requires issuers to file a Form C with the SEC before opening a round, so every active campaign has a verified company name, founding team, financial disclosures, and valuation on public record. That's the dataset: comprehensive, legally disclosed, and — until this Actor — only accessible by visiting three separate sites with three separate UX patterns.

Does Wefunder have an API? 📡

No public API. As of 2026, none of Wefunder, Republic, or StartEngine publishes an official data API or bulk export. Wefunder's SPA calls an internal JSON endpoint (/-/companies/explore) returning full campaign payloads — but it's undocumented, inspects your TLS fingerprint, and sits behind Cloudflare. Republic's backend GraphQL at api.republic.com rejects unauthenticated POSTs from datacenter IPs. StartEngine's offering detail pages require clearing a JavaScript-gated challenge first.

This is exactly why a hosted Actor earns its keep over a three-line requests snippet.

What the data looks like

Each row is a flat, typed record. A real one — RISE Robotics on Wefunder as of 2026-05-16:

{
  "source": "wefunder",
  "campaign_slug": "riserobotics",
  "company_name": "RISE Robotics",
  "tagline": "Electrifying heavy machines",
  "industry": null,
  "location": "MA",
  "founders": ["Hiten Sonpal"],
  "website_url": null,
  "target_amount_usd": null,
  "raised_amount_usd": 17448682.0,
  "num_investors": 417,
  "valuation_usd": 62100000.0,
  "revenue_usd": null,
  "funding_stage": "raising",
  "campaign_url": "https://wefunder.com/riserobotics",
  "scraped_at": "2026-05-16T13:40:00.000Z"
}

Sixteen fields, Pydantic-validated before they hit your dataset. valuation_usd comes from Wefunder's terms.nb shorthand ("$62.1M"), parsed into a float automatically. Republic and StartEngine rows land with the same shape; monetary fields are null there because that data is client-rendered (v2 plan — more below).

The naive approach (and why it falls apart) 🔧

The obvious move: open DevTools, find the XHR, replay it with requests.get(). It breaks fast, for a different reason on each platform.

Wefunder. The /-/companies/explore endpoint checks your TLS ClientHello fingerprint before it answers. Python's stdlib ssl and httpx look nothing like a real browser — the JA3/JA4 fingerprint reads as a script, and you hit a Cloudflare challenge before the JSON loads. We run curl-cffi with impersonate="chrome131", which replays the full Chrome 131 TLS handshake, ALPN extension order, and HTTP/2 SETTINGS frame, so at the TLS layer the connection is a browser.

Republic. The republic.com/companies page is SPA-rendered; the SSR shell carries only a ~10-item carousel of trending campaign links, and the backend GraphQL at api.republic.com rejects unauthenticated POSTs from datacenter IPs. We thread Apify residential proxies on every request so the connection arrives from a residential exit.

StartEngine. Their explore page is fully client-rendered. sitemap-private-offerings.xml carries the active slug list (98 entries as of 2026-05-16) — the only unauthenticated surface; detail pages return a bot-challenge body to non-browser clients. v1 emits slug + company name from the sitemap; Camoufox full-render is planned for v2.

We retry with exponential backoff (base 2 s, doubling, capped at 30 s, max 5 attempts) and honour Retry-After. On 429 or 503 we rotate the proxy session ID — fresh exit IP, fresh cookie jar. Partial success surfaces as an explicit status message; we never return an empty dataset under a green status. One source failing does not kill the run; all three failing exits non-zero with a clear error.

The Actor 🛠️

Equity Crowdfunding Leads on the Apify Store.

Open it in the Apify Console and click Start, or call it with the apify-client Python SDK:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("DevilScrapes/equity-crowdfunding-leads").call(
    run_input={
        "sources": ["wefunder"],
        "maxPerSource": 200,
        "statusFilter": "active",
        "industryFilter": "fintech",
        "useProxy": True,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["company_name"], item["raised_amount_usd"], item["founders"])

The key input parameters:

sources — any combination of wefunder, republic, startengine, or empty (= all three). Default: all three.
maxPerSource — hard cap per platform, 1–500. Default: 50.
statusFilter — "active", "funded", or "all". Wefunder-native; Republic and StartEngine emit currently-listed slugs regardless.
industryFilter — optional case-insensitive substring matched against tagline or industry. Pass "climate" for climate-tech campaigns.
useProxy — default true. Wefunder and Republic fingerprint datacenter IPs and block plain exits; leave it on.

What you'd actually use this for 💡

Four scenarios from the README and spec:

VC scout pipeline. Schedule a weekly Wefunder-only run, pull all active campaigns, join on founders[], enrich with LinkedIn. A live feed of sub-Series-A founders without waiting for Crunchbase. Scope it with industryFilter: "robotics" for your thesis vertical.

SDR founder outreach. Founders in active crowdfunding campaigns are fundraising — and buying. Filter by statusFilter: "active" and industryFilter: "fintech", drop founders[] into Apollo or Clay, and reach them while they're in motion.

Crowdfunding analytics. Schedule daily runs, persist to BigQuery or S3, and track raised_amount_usd trajectories. Wefunder publishes pre-money valuations Crunchbase never sees — the valuation_usd distribution by sector is a clean dataset for a leaderboard or research report.

Form C deep dives. This Actor surfaces campaign_slug and campaign_url. sec-edgar-filings-scraper (sibling Actor) takes it from there — issuer CIK on EDGAR, Form C / Form C-AR PDFs, audited revenue, SAFE terms. Two Actors, one Reg CF pipeline.

Pricing — exact numbers 💰

Pay-per-event. You pay for rows you receive, nothing for rows that don't come back.

Event	Price
Actor start (once per run)	$0.05
Per campaign row emitted	$0.005

Run size	Cost
50 rows (default, all 3 sources)	$0.30
150 rows (50/source × 3)	$0.80
1,000 rows	$5.05
5,000 rows	$25.05
10,000 rows	$50.05

For context, the nearest alternative — scraping Crunchbase via a third-party Apify Actor — typically runs around $30 per 1,000 rows, while covering fewer than 30% of Wefunder campaigns and zero Republic trending campaigns. This Actor is roughly 6× cheaper and sources from the campaigns directly, not from a derived database. Apify's $5 free trial credit covers your first ~990 rows with no credit card.

The part worth knowing before you build on this 🔍

Wefunder's internal /-/companies/explore endpoint is the same one the SPA calls on every page load — unauthenticated, returning full JSON payloads including pre-money valuation encoded as terms.nb dollar shorthand ("$62.1M", "$700K", "$1.2B"). This Actor parses that shorthand with multipliers K=1e3, M=1e6, B=1e9; malformed values emit null rather than crashing.

The design point worth knowing: the scraper doesn't infer valuations — it reads the exact payload the website reads and converts the display string to a typed float. The Pydantic v2 ResultRow model enforces the schema on every row before write, so type surprises are caught at write time, not at analysis time.

Limitations (the honest list) 🚧

Republic and StartEngine return sparse data in v1. Republic surfaces ~10 trending campaign slugs per run from the SSR shell; StartEngine emits slug + company name from the public sitemap. On both, raised amount, valuation, and investor count are client-rendered and stay null. For the richest rows, run Wefunder-only (sources: ["wefunder"]).
No historical archive. Every run is a fresh snapshot of currently-listed campaigns. Schedule runs and export to your own storage; Apify's default run-scoped storage is purged after 7 days on the free plan.
Status filter is Wefunder-native. funded and all only meaningfully change Wefunder results; Republic and StartEngine always emit their current listing surface regardless.
No investor identity data. Who invested and at what amount is private. This Actor emits only public-facing campaign metadata.
No SEC EDGAR Form C parsing. Revenue, expenses, share count, and SAFE terms from Form C filings are in scope for sec-edgar-filings-scraper, not this Actor.

FAQ

Is scraping Wefunder, Republic, and StartEngine legal?
All three host public-facing marketing pages built to attract investors. This Actor reads only what the public UI exposes — no authentication is bypassed, no private investor data is collected, and the request rate stays well under a human browsing the site. Form C filings are SEC-required public disclosures. Check your own jurisdiction and use case; nothing here is legal advice.

Does Wefunder, Republic, or StartEngine have an official API I should use instead?
No. As of 2026, none of the three offers a public data API or bulk export endpoint. Wefunder operates an internal JSON endpoint the SPA uses; Republic and StartEngine surface their data via their web UIs (or, for StartEngine, a sitemap).

Can I export the dataset to Google Sheets or a data warehouse?
Yes — export CSV, JSON, Excel, or XML from the Apify Console Export button after the run, webhook the dataset on ACTOR.RUN.SUCCEEDED into Make, Zapier, or n8n, or pull it via the Apify API.

Why does the Actor cost less than Crunchbase scrapers?
Different source, lower extraction cost. Crunchbase scraping hits a richer, more heavily defended site with far more fields. This Actor targets three smaller platforms and returns a narrower, well-defined schema. The 6× difference reflects the actual engineering complexity.

Try it

Live on the Apify Store: apify.com/DevilScrapes/equity-crowdfunding-leads.

Free $5 trial credit, no credit card. Run the defaults and you'll have 150 equity-crowdfunding leads across all three platforms in a couple of minutes. Need a fourth platform (NextSeed, MicroVentures), a field you wish was populated, or a parser that broke after a site restructure? Drop it in the comments. The devil's in the data; I ship based on what people actually find there.

Further reading:

Built by Devil Scrapes — Apify Actors for builders who want the data, not the drama. Pay-per-event, honest pricing, no junk fields. 😈

Discogs Price Scraper: pull every marketplace listing as clean JSON

Devil Scrapes — Sun, 31 May 2026 12:02:58 +0000

Quick answer: The Discogs marketplace is the world's largest secondary market for vinyl, CDs, and cassettes — but its free REST API exposes only a lowest_price aggregate and a num_for_sale count. To get the per-listing detail — who is selling, at what asking price, in what condition, and shipping from where — you have to scrape the Cloudflare-protected marketplace HTML. The Apify Actor at apify.com/DevilScrapes/discogs-sold-price does exactly that, joining the public REST API with paginated marketplace HTML into a single flat dataset. Price: $0.005 per row, ~$5.05 per 1,000 results.

If you have ever priced a record collection for resale, you know the workflow: open Discogs, search the release, squint at 25 listings per page, mentally adjust for condition grade and seller location. Multiply by 200 records and a weekend is gone. The data you want — every asking price, every seller rating, every condition grade — is sitting on a public web page, but Discogs exposes none of it in any published API, and the page sits behind Cloudflare.

What is Discogs? 🎵

Discogs is a community-built catalog of released music and an active secondary marketplace. Collectors use it to catalog inventory, buy and sell physical media, and settle "what is this copy worth" questions. The catalog side is free and has a REST API. The marketplace side — the live order book of who is selling what, at what price — is a Cloudflare-protected web app with no programmatic export.

The Discogs API exposes release metadata (title, artist, year, format, genres), community stats, and exactly two market-signal fields: lowest_price and num_for_sale. That is the entire API surface for marketplace pricing. Every column past those two — asking price by listing, condition grade, seller country, seller rating — lives only in the HTML.

Does Discogs have an API for marketplace listings?

No. The Discogs REST API at api.discogs.com exposes a GET /marketplace/stats/{release_id} endpoint that returns the floor ask (lowest_price) and the total listing count (num_for_sale). It does not return individual listings. The per-listing data — who is selling, at what price, with what media and sleeve grade, ships-from where — is available only on the paginated HTML page at www.discogs.com/sell/release/{release_id}. That page is served through Cloudflare and does not have a documented API counterpart.

This Actor fills that gap: it joins the two public REST endpoints (release metadata + marketplace stats) with the paginated listing HTML into one flat, typed row per listing.

What the data looks like 📤

Each marketplace listing comes back as one flat row. Here is a real one, from release ID 249504 (Rick Astley — Never Gonna Give You Up, 1987 UK 7"):

{
  "row_type": "listing",
  "release_id": 249504,
  "release_title": "Never Gonna Give You Up",
  "artist": "Rick Astley",
  "year": 1987,
  "country": "UK",
  "format_name": "Vinyl",
  "format_descriptions": ["7\"", "45 RPM", "Single", "Stereo"],
  "genres": ["Electronic", "Pop"],
  "master_id": 96559,
  "release_url": "https://www.discogs.com/release/249504",
  "listing_id": 3761251765,
  "listing_url": "https://www.discogs.com/sell/item/3761251765",
  "asking_price": 0.5,
  "asking_currency": "GBP",
  "shipping_text": "+£15.00",
  "condition_media": "Very Good Plus (VG+)",
  "condition_sleeve": "Generic",
  "seller_username": "Ronan266",
  "seller_rating_pct": 100.0,
  "seller_rating_count": 35,
  "seller_country": "United Kingdom",
  "stats_lowest_price": null,
  "stats_lowest_currency": null,
  "stats_num_for_sale": null,
  "stats_blocked_from_sale": null,
  "scraped_at": "2026-05-16T12:00:00.000Z"
}

That is the full 27-field schema. With includeStatsRow on (default), the Actor also emits one aggregate row per release — same schema, row_type: "stats", stats_* fields populated, listing fields null — so you can GROUP BY release_id for both per-listing detail and the market floor. Pydantic v2 validates every row before it is pushed: ISO-8601 timestamps, proper nulls, typed numerics — not stringified soup.

The naive approach (and why it falls apart) ⚠️

Every developer who finds the Discogs marketplace HTML eventually tries the same thing: open Chrome DevTools, find the request for /sell/release/{id}, replay it with requests.get(). Here is what breaks.

Cloudflare. The www.discogs.com/sell/release/ path sits behind Cloudflare's bot-management layer. Python's stdlib SSL stack, httpx, and plain requests all fail with a 403 and a JS challenge — the response looks like HTML but holds no listing data. We handle it with curl-cffi impersonation: AsyncSession(impersonate="chrome131") replays a real Chrome 131 TLS ClientHello, ALPN order, and HTTP/2 SETTINGS frame. Before any listing page we run a single homepage warm-up to seed the Cloudflare cookies, after which pages return 200 cleanly. We thread Apify residential proxies (BUYPROXIES94952) to give the session a clean exit IP — verified in cloud QA on real Apify datacenter IPs.

Rate limits. Discogs documents 60 req/min for anonymous API traffic; the marketplace HTML surface has an unwritten ceiling around 25 req/min. We throttle at one request per 1.5 seconds (~40 req/min) across all calls to stay under both, honour Retry-After on 429, and retry with exponential backoff on 408/429/503 and network errors (five attempts). When a release fails after all retries we log a WARNING, emit whatever rows we collected, and continue — one bad ID never kills the run.

The REST API needs a custom User-Agent. The Discogs API Terms require every request to carry an Application-Name/Version User-Agent; default browser-impersonation headers get a 403 on the JSON surface. The Actor sends DevilScrapes/0.1 (+https://apify.com/DevilScrapes) automatically — you configure nothing.

Parsing the condition grades. Condition text is free-form HTML. We extract it with a regex against the canonical Discogs grade vocabulary (Mint, Near Mint, VG+, VG, Generic, Not Graded, and the rest); anything outside it sets the field to null with a WARNING rather than emitting garbage. None of this is glamorous — all of it is the difference between a script that works once and a feed that survives Cloudflare's quarterly changes.

The Actor ⚙️

The Actor is live on the Apify Store: apify.com/DevilScrapes/discogs-sold-price. Run it from the Apify Console (paste release IDs, click Start) or via the Apify Python client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("DevilScrapes/discogs-sold-price").call(
    run_input={
        "releaseIds": [249504, 10843],
        "maxPagesPerRelease": 4,
        "maxListingsPerRelease": 100,
        "includeStatsRow": True,
        "useProxy": True,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["row_type"], item.get("asking_price"), item.get("condition_media"))

If you do not know the release ID, use searchQuery instead:

run = client.actor("DevilScrapes/discogs-sold-price").call(
    run_input={
        "searchQuery": "nirvana nevermind",
        "maxSearchResults": 5,
        "maxListingsPerRelease": 50,
    }
)

The search path resolves the top maxSearchResults hits into release IDs, then fetches listings for each. Exactly one of releaseIds or searchQuery is required — passing both, or neither, raises a Pydantic ValidationError before any network call goes out.

Key input parameters:

Field	Default	Max	Notes
`releaseIds`	—	100 IDs per run	XOR with `searchQuery`
`searchQuery`	—	200 chars	Resolves via Discogs search API
`maxSearchResults`	5	50	Only used with `searchQuery`
`maxPagesPerRelease`	4	20	25 listings per page
`maxListingsPerRelease`	100	500	Hard cap per release
`includeStatsRow`	`true`	—	One aggregate row per release
`useProxy`	`true`	—	Recommended for Apify cloud runs

Use cases 💡

Vinyl and CD reseller benchmarking. Took in 200 records at an estate sale? Pull the marketplace listings for each release, compute the median asking price by condition grade, and price your own copies in minutes instead of an afternoon — filtering on seller_country for the shipping-adjusted competitive picture.

Music-collectibles arbitrage. The same release often trades at materially different prices across seller countries. Query seller_country + asking_price + asking_currency across a handful of releases and spot regional underpricing before another buyer does.

Label and catalog market intel. For a label's back-catalog, track stats_num_for_sale and stats_lowest_price over time with Apify Schedules. A sustained drop in num_for_sale alongside a rising stats_lowest_price is a secondary-market appreciation signal.

Journalism and pricing studies. "What does a first pressing of Nevermind cost right now?" is a recurring music-publication paragraph. One Actor run on the relevant release IDs gives you a defensible, timestamped dataset instead of a manual spot-check.

Seller-quality screening. Filter listings by seller_rating_pct >= 99.0 and seller_rating_count >= 100 before buying, or sweep stats_blocked_from_sale to flag releases Discogs has quietly restricted. The Actor hands you the data; the filtering is one line of Pandas.

Pricing — exact numbers 💰

Pay-per-event. You pay only for rows that land in the dataset. No data, no charge (beyond the $0.05 run start fee).

Event	Price (USD)	When
`actor-start`	$0.05	Once per run, at boot
`result-row`	$0.005	Per listing OR per stats row written

What that looks like in practice:

Run	Rows	Cost
1 release × 25 listings + 1 stats row	26	$0.18
5 releases × 100 listings + 5 stats rows	505	$2.58
10 releases × 100 listings + 10 stats rows	1,010	$5.10
50 releases × 100 listings + 50 stats rows	5,050	$25.30

At scale the per-row charge dominates: approximately $5.05 per 1,000 rows, priced for a hand-parsed marketplace listing with full seller metadata rather than a JSON API field copy. Apify's $5 free trial credit covers your first ~990 listing rows, no credit card required.

The technically interesting part

The devil is in the data-attribute vs. display-text split. Discogs encodes the machine-readable price in an HTML attribute (data-pricevalue="0.50", data-currency="GBP") and a human-formatted string in the visible text ("£0.50"). The displayed text goes through server-side currency conversion for non-UK visitors, so text-scraping gives you a converted price in whatever currency the datacenter IP geolocates to. We always prefer data-pricevalue and data-currency — the canonical values the seller entered — and only fall back to text parsing (with a WARNING) when the attributes are missing. For any cross-country price comparison the text number is a moving target and the attribute is not.

Limitations 🚧

No closed-sale price history. Discogs hosts sold-price history at /sell/history/{release_id}, but that page is gated behind account login (Auth0) and is out of scope without user OAuth. What you get instead: every active asking price (the live offer side) plus the public lowest_price aggregate. For most reseller and arbitrage workflows the live asking-price distribution is the more actionable signal anyway.

One snapshot per run. The Actor captures marketplace state at a point in time. For time-series tracking, schedule recurring runs via Apify Schedules and concatenate datasets by release_id + scraped_at.

500-listing ceiling per release. Discogs serves 25 listings per page server-side; the maxPagesPerRelease cap of 20 gives a 500-listing hard ceiling per release per run. Releases with more active listings need multiple runs.

Currency is not normalised. asking_price comes in the seller's local currency, stats_lowest_price in the request IP's resolved currency. Join on asking_currency / stats_lowest_currency; no canonical USD conversion is applied.

Throughput is ~40 req/min, imposed by Discogs — a 10-release run with 4 pages each takes roughly 90 seconds plus warm-up. And note the 7-day default storage retention on the Apify FREE plan: export your dataset right after the run or upgrade for longer retention.

FAQ ❓

Is scraping Discogs marketplace listings legal?
The marketplace listings page is public — no login, no paywall. This Actor reads only what the anonymous public UI exposes, paces itself at ~40 req/min (below Discogs' documented 60 req/min ceiling), and authenticates against nothing. As always, verify against your own jurisdiction and use case before running at scale.

Does Discogs have an API for marketplace listings?
Not for per-listing data. The REST API returns only lowest_price and num_for_sale aggregates per release; individual listings (price, condition, seller) live only on the Cloudflare-protected HTML page, which this Actor scrapes.

Can I get the closed-sale / sold-price history?
No. The /sell/history/{release_id} page is gated behind Auth0 login. The "sold price" in the slug is a vestige of the original scope. What the Actor delivers is the ask-side snapshot: every active listing plus the public floor-price aggregate — for most reseller workflows the more actionable data.

Can I export the results to a spreadsheet or warehouse?
Yes — CSV, Excel, JSON, and XML exports from the Apify dataset viewer. You can also webhook the dataset on ACTOR.RUN.SUCCEEDED into Make, Zapier, or n8n, or pull it via the Apify API: GET /datasets/{id}/items?format=csv&clean=true.

Try it

The Actor is live: apify.com/DevilScrapes/discogs-sold-price.

Free $5 Apify credit, no credit card. Run it on release ID 249504 (the Rick Astley test classic) and you will have 25 typed listing rows in your dataset in under a minute. Found a field you wish it returned — median price, condition-grade distribution? Drop a comment or open a request on the Store page. We ship based on what builders actually need.

Built by Devil Scrapes — Apify Actors with attitude. Pay-per-event, transparent pricing, no junk fields. 😈

DEV.to Scraper: pull articles by tag, author, or feed into clean JSON

Devil Scrapes — Sun, 31 May 2026 11:57:44 +0000

Quick answer: DEV.to (built on the Forem platform) publishes a public v1 REST API at https://developers.forem.com/api/v1 — but it paginates 30 articles at a time and offers no bulk-by-tag export beyond the first 1,000 items. A DEV.to scraper fans out across those paginated endpoints, fetches each article's full Markdown body in parallel, and returns one clean typed row per article. The Apify Actor below does it for $0.002 per article ($2.00 per 1,000), with rate-limit pacing, retries, and Pydantic-validated rows handled for you.

DEV.to's feed is one of the richest free sources of developer-written technical content on the web. On any given day the python tag alone has thousands of articles — tutorials, opinions, walkthroughs, career posts — each with engagement signals (reactions, comments, reading time) attached. The platform's UI surfaces individual articles just fine. What it doesn't give you is a download button, a bulk-export endpoint, or a way to pull every article in a tag across the full history. The API hands back 30 at a time, then stops answering after 1,000 items per tag.

If you want this as a dataset — to seed a RAG corpus, run an engagement benchmark, or mirror an author's catalogue — you have to stitch it together yourself. Here's what that involves, and how I turned it into a one-call Actor.

What is DEV.to? 🔎

DEV.to is a community publishing platform for software developers, built on Forem — the open-source publishing engine that also powers CodeNewbie and several smaller communities. Launched in 2016, DEV.to hosts millions of articles across tags like python, webdev, typescript, beginners, and ai, written by everyone from student bloggers to senior engineers.

What makes DEV.to useful as a data source:

Every article carries structured engagement metrics: positive reactions, comments, and reading time
Articles are tagged with community-maintained taxonomy (lowercase tags like javascript, devops, aws)
The body of every article is available as raw Markdown — ready to embed in a vector store without stripping HTML
Authorship is consistent: every article has a username and display_name, making per-author analysis straightforward

What the platform does not give you: a bulk export, a search-by-keyword endpoint, or a way to get all articles in a tag older than the most recent thousand.

Does DEV.to have an API? 🔌

Yes — but it has meaningful limits. DEV.to's Forem v1 API is public for read access, requires no API key for GET /articles, and is reasonably well-documented. That's genuinely the good news.

The constraints that send people looking for a scraper:

30 articles per page, hard cap. You can request per_page=30 — that's the max the server will honor. Getting 10,000 articles means 334 sequential (or carefully paced parallel) requests.
Tag endpoint cuts off at 1,000 items. Past page 34, the API returns empty arrays. There's no cursor mechanism, no since timestamp, and no workaround documented by Forem.
Body Markdown requires a second request. GET /articles returns metadata. To get body_markdown you need a GET /articles/:id call per article — one extra round trip for every row you want full-text on.
Rate limits are real and undocumented. Hit them and you get 429s with no Retry-After header. Retry naively and you accumulate backoff penalties.

None of that is a dealbreaker on its own. Together, for a 10,000-article corpus pull, it's several hours of babysitting a script. That's the gap the Actor fills.

What the data looks like

Each article becomes one flat, typed row. Here's a real-shaped output record with all 16 fields from models.py:

{
  "id": 1893402,
  "slug": "build-a-rag-pipeline-with-python-and-chroma-3x7k",
  "title": "Build a RAG pipeline with Python and Chroma",
  "description": "A step-by-step guide to building a retrieval-augmented generation system using Python, ChromaDB, and the OpenAI API.",
  "url": "https://dev.to/pythonista/build-a-rag-pipeline-with-python-and-chroma-3x7k",
  "cover_image": "https://media.dev.to/cdn-cgi/image/quality=100/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xyz.jpg",
  "tags": ["python", "ai", "machinelearning", "beginners"],
  "author_username": "pythonista",
  "author_name": "Alex Chen",
  "reading_time_minutes": 8,
  "positive_reactions_count": 347,
  "comments_count": 19,
  "body_markdown": "## Introduction\n\nRetrieval-augmented generation (RAG)...",
  "published_at": "2026-04-12T09:14:00Z",
  "edited_at": "2026-04-14T11:02:00Z",
  "scraped_at": "2026-05-29T08:22:41+00:00"
}

Sixteen fields, consistent shape every time, Pydantic-validated before the row is written. It drops straight into Pandas, a vector store, or BigQuery with no field-wrangling on your side.

The naive approach (and why it falls apart) 🔧

The obvious script goes like this:

import httpx, json

articles = []
page = 1
while True:
    resp = httpx.get(
        "https://dev.to/api/articles",
        params={"tag": "python", "page": page, "per_page": 30}
    )
    batch = resp.json()
    if not batch:
        break
    articles.extend(batch)
    page += 1

This works until it doesn't. Three failure modes that matter at scale:

1. The 1,000-item ceiling. Around page 34 the API returns an empty list for any tag endpoint. There's no error, no header — just silence. A naive loop exits thinking it finished. You have 1,000 rows instead of the 12,000 that exist.

2. The body-fetch N+1 problem. Want Markdown? Every article needs a second GET /articles/:id. For 1,000 articles that's 1,000 extra requests. We handle this with a concurrency parameter — up to 16 parallel body fetches — so the Actor fans out rather than serializing. We pace those fetches and retry with exponential backoff on 408 / 429 / 5xx, up to 5 attempts per article before we surface a partial-success status rather than handing you a half-empty dataset.

3. Rate limits that arrive unannounced. DEV.to's rate-limit threshold isn't published; it varies by endpoint and time of day. We back off on rate-limit signals and reset the session rather than triggering a retry storm — and we surface a set_status_message so you know what happened, rather than silently returning fewer articles than you asked for.

We rotate browser fingerprints via curl-cffi so requests look like a real browser's TLS handshake, and we thread Apify residential proxies on every session rotation — fresh exit IP, fresh cookie jar — so a single blocked IP doesn't kill the run.

The Actor

I packaged this as an Apify Actor: DEV.to Articles Scraper.

Open the Apify Console and click Start, or run it programmatically with the Apify Python client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("DevilScrapes/dev-to-articles-scraper").call(
    run_input={
        "mode": "tag",
        "tag": "python",
        "includeBody": True,
        "maxResults": 500,
        "concurrency": 8,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["positive_reactions_count"])

The four mode values from the input schema:

Mode	What it fetches
`tag`	All articles for a given tag (e.g. `python`, `ai`, `webdev`)
`username`	All articles by a specific DEV.to author
`latest`	Global latest feed — newest articles across all tags
`top`	Top articles of all time across the platform

Set includeBody: false if you only need metadata — that halves the request count and the run time. maxResults caps the total rows; concurrency controls how many body fetches run in parallel (1–16, default 4).

Use cases

RAG corpus seeding. Pull mode=tag, tag=python, includeBody=true, maxResults=1000 to get 1,000 Python tutorials in raw Markdown. Each article is already chunked — the body_markdown field is the unit. Embed it directly into Chroma, Pinecone, or Weaviate. The url and author_username fields give you citation metadata for free.

Trending tag dashboards. Schedule a daily run on mode=tag, tag=ai with maxResults=50. Diff today's positive_reactions_count against yesterday's — any article that gained more than 50 reactions in 24 hours is trending. Wire it into a Slack webhook and you have a free daily briefing.

Author monitoring and portfolio analysis. Pull mode=username for a specific author to mirror their full catalogue. Useful for DevRel teams tracking competitor advocates, recruiters benchmarking engineering blog authors, or writers building a personal analytics dashboard outside DEV.to's own stats page.

Newsletter assembly. Pull mode=top or mode=latest with maxResults=10, sort by positive_reactions_count, and render the top 5 to Markdown for a weekly digest. The reading_time_minutes field tells readers upfront what they're committing to.

Engagement benchmarking. Pull 500 articles in a tag, group by author_username, and compute average reactions per post — a simple "who are the most impactful writers in this niche?" query for sponsorship research, guest-post pitching, or a contributor leaderboard.

Pricing — exact numbers 💰

Pay-per-event. You pay for articles written to your dataset, nothing for the ones you don't get.

$0.005 per run (covers the Actor warm-up)
$0.002 per article written to the dataset

Pull	Cost
30 articles (default)	$0.07
100 articles	$0.21
1,000 articles	$2.01
5,000 articles	$10.01
10,000 articles	$20.01

Apify's $5 free trial credit covers your first ~2,490 articles with no credit card required. No subscription, no minimum, no charge for runs that return zero results.

The technically interesting bit

DEV.to's API officially cuts off article listings at 1,000 per tag — but the per-article GET /articles/:id endpoint has no such limit. So a full corpus is achievable by combining the listing endpoint (for IDs) with the detail endpoint (for bodies): even when the listing only goes 34 pages deep, you can supplement IDs from the username endpoint, the latest feed, or a prior run's dataset. The Actor exposes this as a design choice — mode=tag is the fast lane for recent articles; mode=latest is the slow lane for full-history accumulation over scheduled runs. Both paths produce identical row shapes, so your downstream pipeline never needs to know which mode fed it.

Limitations 🚧

Tag endpoint hard cap at ~1,000 items. The DEV.to v1 API does not paginate beyond this for the tag feed. Full-history pulls require either the username mode (per-author) or multiple scheduled latest-mode runs.
Body Markdown is the API's version. If an author used DEV.to's rich editor with embedded Liquid tags (custom video/link cards), those render as raw Liquid syntax in the Markdown — not HTML. Post-processing is on you.
No comment bodies. comments_count is in the metadata, but fetching individual comment threads would multiply the request count significantly. Not in scope for v1.
No read-time filtering at the API level. You can filter post-scrape, but the API doesn't accept a min_reading_time param. Download the dataset and filter in Pandas.
Private/draft articles are inaccessible. The public API only surfaces published, non-hidden articles.

FAQ

Is scraping DEV.to legal?
This Actor calls DEV.to's own published public API (https://developers.forem.com/api/v1) — no authentication bypassed, no HTML scraped, no undocumented endpoint hit. The API is designed for programmatic access. Standard advice: read DEV.to's Terms of Service, stay within polite request rates, and don't republish article bodies wholesale without attribution.

Can I export the dataset to a spreadsheet or warehouse?
Yes — export CSV, JSON, or Excel directly from the Apify Console. Alternatively, webhook the dataset on ACTOR.RUN.SUCCEEDED into Make, Zapier, or n8n, or fetch it via the Apify API for direct warehouse ingestion.

Does DEV.to have an official bulk-export API?
No. The Forem v1 API paginates 30 articles at a time and caps tag listings at approximately 1,000 items. There is no official bulk download, no CSV export, and no GraphQL endpoint on the public surface.

Why are some body_markdown fields null?
Some DEV.to articles link out to a canonical URL hosted on the author's own blog — the metadata (title, tags, reactions) lives on DEV.to but the body lives elsewhere. In those cases the API returns an empty or very short body; the Actor surfaces that faithfully as null rather than silently dropping the row.

Try it

The Actor is live on the Apify Store: apify.com/DevilScrapes/dev-to-articles-scraper.

Free $5 trial credit, no credit card. Run it on tag=python with maxResults=30 and you'll have a full typed dataset in under a minute — Markdown bodies included if you leave includeBody: true. Need a field that isn't there (comment threads, co-authors, series metadata)? Drop a note in the comments. We read every one.

Built by Devil Scrapes — the devil's in the data, and we keep it clean. 😈

CoinGecko API Scraper: pull crypto price, market cap, ATH for $2/1K

Devil Scrapes — Sun, 31 May 2026 11:52:29 +0000

Quick answer: A CoinGecko API scraper feeds a list of coin IDs, rotates TLS fingerprints and proxy sessions to stay within CoinGecko's free-tier rate limits (~10 requests per minute), and returns one clean, typed JSON row per coin with 26 fields — price, market cap, ATH/ATL, supply, and sentiment. The Apify Actor below costs $0.002 per coin (~$2.00 per 1,000), handles the rate-limiting for you, and lands the data wherever your stack lives.

If you've built anything crypto-adjacent — a portfolio dashboard, an alert bot, a strategy backtest — you've run into CoinGecko's free tier. The data is excellent: prices, market caps, circulating vs total supply, all-time highs and lows, community sentiment, even GitHub activity. The problem isn't the data; it's getting it at scale without triggering a rate-limit cascade.

A for loop over your coin list, one requests.get per coin, works fine for five coins and falls apart at fifty. By the time you've added backoff, fingerprint rotation, proxy management, and typed output, you've spent half a day on plumbing you'd rather not maintain. Here's how I wrapped all of that into a single API call.

What is CoinGecko? 🔎

CoinGecko is an independent cryptocurrency data aggregator, founded in 2014, that tracks over 14,000 digital assets across 900+ exchanges. It publishes market data — prices, volumes, supply figures, ATH/ATL records, community sentiment — through a public REST API (v3) with a free "Demo" tier and paid Pro plans. Unlike exchange APIs, CoinGecko aggregates across venues, so you get one normalized view regardless of where a coin trades.

For portfolio tracking, research, alerting, or tax reporting, CoinGecko is the reference source: public dataset, deep coverage, documented API. The catch is throughput — the free Demo tier enforces a hard cap that makes bulk pulls slow and fragile without deliberate rate-limit management.

Does CoinGecko have an API? 🔌

Yes — CoinGecko publishes an official v3 REST API. This Actor uses the /coins/{id} endpoint to fetch the per-coin detail payload. The free Demo tier is rate-limited to roughly 10 requests per minute from a single IP; the Pro tier lifts that significantly. The API is documented and stable, but managing those rate limits across concurrent workers — when you're pulling hundreds of coins and can't afford to serialize them — is where most scripts break down.

This Actor adds the plumbing the raw API doesn't: fingerprint rotation, proxy sessions, backoff on 429s, and clean validated output. You bring the coin list; we bring the resilience.

What the data looks like 📤

Each coin comes back as one flat, typed row. Every field below is from models.py:

{
  "coin_id": "bitcoin",
  "symbol": "btc",
  "name": "Bitcoin",
  "image_url": "https://coin-images.coingecko.com/coins/images/1/large/bitcoin.png",
  "current_price_usd": 71250.32,
  "current_price_vs": 65120.10,
  "vs_currency": "eur",
  "price_change_24h_pct": 2.41,
  "price_change_7d_pct": -1.18,
  "price_change_30d_pct": 14.73,
  "market_cap_usd": 1408000000000,
  "market_cap_rank": 1,
  "total_volume_usd": 28540000000,
  "circulating_supply": 19700000,
  "total_supply": 21000000,
  "max_supply": 21000000,
  "ath_usd": 73738.00,
  "ath_date": "2024-03-14T07:10:36.635Z",
  "atl_usd": 67.81,
  "atl_date": "2013-07-06T00:00:00.000Z",
  "sentiment_up_pct": 74.29,
  "exchanges_count": 58,
  "github_stars": null,
  "github_forks": null,
  "coingecko_url": "https://www.coingecko.com/en/coins/bitcoin",
  "scraped_at": "2026-05-31T09:00:00+00:00"
}

Twenty-six fields per coin, Pydantic-validated before they reach your dataset. github_stars and github_forks populate when you enable includeDeveloperData. Everything else — prices, supply, ATH/ATL, sentiment, exchange count — comes through on the default configuration.

The naive approach (and why it falls apart) ⚠️

The immediate instinct for anyone who's glanced at the CoinGecko API docs:

import httpx, asyncio

async def fetch_coins(coin_ids):
    async with httpx.AsyncClient() as c:
        for coin_id in coin_ids:
            r = await c.get(f"https://api.coingecko.com/api/v3/coins/{coin_id}")
            yield r.json()

Works for three coins on your laptop. Here's where it breaks:

1. Rate limits shut you down without ceremony. The Demo tier enforces roughly 10 requests per minute. A list of 100 coins serialized naively takes 10 minutes minimum and still 429s if your timing slips. We sleep 30 seconds on a 429, retry once, and warn loudly if it persists, so partial success surfaces rather than silently producing an empty dataset.

2. TLS fingerprinting tightens under load. Hit the endpoint at volume from a datacenter IP with Python's default TLS stack and the response profile changes. We rotate through Chrome 131, Chrome 124, Firefox 147, and Safari 180 TLS fingerprints via curl-cffi, picked randomly per session, so the handshake looks like a real browser even under concurrent load.

3. Concurrency without a semaphore blows the limit. Even asyncio.gather() with ten concurrent tasks exceeds the free-tier quota instantly. We thread a Semaphore around the fetch so parallel lookups stay within the configured ceiling — default 2, configurable up to 32 for Pro-key holders.

4. Null fields for low-coverage coins. Smaller coins return incomplete payloads with missing keys or null market-data fields. A naïve data["market_data"]["current_price"]["usd"] raises KeyError or TypeError. Our _build_row guards every field with .get() chains and Pydantic absorbs the rest — nulls land as typed None values rather than crashing the run.

We rotate residential proxies through Apify Proxy on a block, retry with exponential backoff on 408/429/5xx, and hand back Pydantic-validated rows. No data, no charge — only the small actor-start warm-up fee fires if the run produces zero items.

The Actor ⚙️

Store listing: apify.com/DevilScrapes/coingecko-crypto-scraper

Paste a list of coin IDs in the Apify Console and click Start, or drive it from Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("DevilScrapes/coingecko-crypto-scraper").call(
    run_input={
        "coinIds": ["bitcoin", "ethereum", "solana", "cardano", "chainlink"],
        "vsCurrency": "eur",
        "includeDeveloperData": False,
        "concurrency": 2,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["name"], item["current_price_usd"], item["price_change_24h_pct"])

Key input fields:

Field	Default	Notes
`coinIds`	`["bitcoin", "ethereum", "solana"]`	CoinGecko slug from the URL — e.g. `uniswap`, `the-graph`
`vsCurrency`	`"eur"`	Secondary quote currency alongside USD
`includeDeveloperData`	`false`	Adds GitHub stars / forks for coins with linked repos
`apiKey`	—	CoinGecko Pro key — lifts the 10 req/min Demo ceiling
`concurrency`	`2`	Keep at 2 on Demo; raise to 8+ with a Pro key

Coin IDs are the lowercase slugs in the URL — bitcoin is /en/coins/bitcoin, the-graph is /en/coins/the-graph. The CoinGecko coin list endpoint returns every valid ID if you need to enumerate them first.

Use cases 💡

Portfolio tracker. Schedule a daily run with your watch list — 20 coins, vsCurrency: "eur", concurrency: 2. Each run produces one row per coin with price in USD and EUR, 24h/7d/30d change, and market cap. Pipe the dataset to a Google Sheet via the Apify API or a Make webhook. Cost: 20 coins × $0.002 = $0.04 per day.

Alert bot. Run every hour, diff price_change_24h_pct against the previous run, and fire a Slack notification when any coin moves more than 10% in either direction. The scraped_at ISO timestamp on each row makes the diff deterministic.

Trading-strategy backtest data. Pull 50 coins by market cap rank, store daily snapshots in a named Apify dataset, then read back the stored runs for strategy simulation. market_cap_rank, total_volume_usd, and the price-change fields give you the features most trend-following strategies need.

Tax reporting. Capital-gains math needs a daily closing price for every asset held. One run per day per asset, stored with scraped_at, gives you an auditable price history without a dedicated pricing-feed subscription. At $0.002 per coin, 10 coins × 365 days = $7.30/year.

Developer-activity screening. Enable includeDeveloperData, then filter coins where github_stars > 500 and price_change_30d_pct > 0 — a rough proxy for active development plus market momentum. CoinGecko sources the GitHub stats from each project's linked repository.

Pricing — exact numbers 💰

Pay-per-event. You pay for coins you receive, nothing for coins that error out.

Event	Price
`actor-start` (once per run)	$0.005
`result` (per coin in dataset)	$0.002

Pull	Cost
10 coins	$0.025
100 coins	$0.205
1,000 coins	$2.005
5,000 coins	$10.005

Apify's $5 free trial credit — no credit card — covers your first 2,497 coins. This Actor uses the free Demo tier by default, so you get well-structured, well-managed access to the same data without a subscription.

The technically interesting bit 🧠

CoinGecko's /coins/{id} endpoint packs market data into nested dicts with currency-keyed sub-objects — market_data.current_price.usd, market_data.ath.usd, and so on. Low-coverage coins return partial payloads: some fields have the outer key but null, some skip the key entirely, some return an empty dict {} for the inner object. A straightforward data["market_data"]["current_price"]["usd"] fails unpredictably across that variance.

The _build_row function in scraper.py guards every multi-level access with .get() chains — (market.get("ath") or {}).get("usd") — which safely returns None for any missing level without raising. Pydantic then validates the final ResultRow, coercing None to JSON null on model_dump(mode="json"). Your rows stay consistently typed regardless of which level of the CoinGecko response is missing, and the run never crashes on a partially-covered altcoin.

Limitations 🚧

Demo tier is ~10 req/min. Large coin lists (500+) are slow without a CoinGecko Pro key. Attach one via apiKey to lift the ceiling.
Some fields are null for low-coverage coins. Smaller, less-traded coins often have null market_cap_usd, total_volume_usd, or exchanges_count. The fields are present in the schema — just typed as number | null.
No historical OHLC. This Actor fetches current market state; historical price series use a different CoinGecko endpoint and are out of scope here.
CoinGecko updates every ~60 seconds. Not a real-time tick feed. For sub-second data, use an exchange WebSocket.
exchanges_count reflects CoinGecko's ticker window. It's the number of distinct exchanges in the tickers CoinGecko returns for the coin, which may not be every venue listing it globally. Treat it as a coverage signal, not an exhaustive count.

FAQ ❓

Is scraping CoinGecko legal?
This Actor calls CoinGecko's own documented public API — the same endpoint CoinGecko publishes in its docs. It doesn't bypass authentication, doesn't scrape rendered HTML, and respects rate limits by backing off on 429s. CoinGecko's Terms of Service permit API access for non-commercial and commercial use subject to rate limits. Check your own use case against their terms.

Can I export to Google Sheets or a data warehouse?
Yes. Export CSV/Excel/JSON directly from the Apify Console dataset view. Use a webhook on ACTOR.RUN.SUCCEEDED to push each run's results into Make, Zapier, or n8n. Pull the Apify dataset API directly from any pipeline that can make an HTTP call.

Is there a CoinGecko API?
Yes — this Actor wraps CoinGecko's v3 REST API. If you prefer to call it directly, the documentation is public. The Actor adds rate-limit management, fingerprint rotation, proxy routing, and Pydantic-typed output on top of the raw API surface.

Why does my run return fewer coins than I requested?
A coin ID that doesn't exist returns a 404, which the scraper logs and skips. The usual cause is a typo in the slug — the-graph not thegraph, uniswap not uni. Check the exact slug in the CoinGecko URL.

Try it

apify.com/DevilScrapes/coingecko-crypto-scraper

Free $5 trial credit, no credit card required. Drop in ["bitcoin", "ethereum", "solana"], click Start, and you'll have your first three rows in seconds. Need a larger coin universe, or a field that isn't here yet? Drop a comment — the roadmap is driven by what people build with this.

Built by Devil Scrapes — Apify Actors for builders who'd rather not debug rate-limit waterfalls at 2 a.m. Pay-per-event, transparent pricing, no junk fields. 😈

Bluesky Starter Pack Scraper: export any community list for $2.05/1K

Devil Scrapes — Sun, 31 May 2026 11:47:16 +0000

Quick answer: A Bluesky starter pack is a curated community list published on the AT Protocol. There is no official export button or bulk API endpoint for reading pack membership programmatically. A Bluesky starter pack scraper queries the public AT Protocol AppView (https://public.api.bsky.app/xrpc/) and returns every member's profile — handle, DID, follower count, post count — as clean, typed JSON. The Apify Actor below does it for $0.002 per member row (~$2.05 per 1,000 members), handling pagination, retries, and URL normalization for you.

Starter packs drove a reported 43% of all follows during Bluesky's 2024 growth surge (EurekAlert, 2024). That number matters: when someone publishes "ML Researchers on Bluesky" or "London Founders" as a starter pack, they are not just recommending accounts — they are shaping the social graph of an entire professional niche. For researchers, growth marketers, and social-graph analysts, those packs are some of the most signal-dense audience lists on any platform.

The problem: the Bluesky web UI shows a pack's members one scroll at a time. There is no CSV download, no endpoint labeled "give me all members", and no obvious way to turn a pack into a spreadsheet your CRM can ingest. Here is what it actually takes to get that data programmatically.

What is a Bluesky Starter Pack? 🔎

A Bluesky Starter Pack is a named, curator-published list of accounts, stored as a record in the AT Protocol — the open, federated protocol underpinning Bluesky. Any user can create one, and the platform surfaces popular packs on the onboarding screen and in discovery feeds. Each pack carries a title and optional description, a list of member DIDs (the protocol's decentralized account identifiers), the creator's handle (e.g. pfrazee.com), and a stable AT URI (at://did:plc:.../app.bsky.graph.starterpack/...) that identifies it across the federated network.

When a new user follows an entire pack, every member gains a follower at once. That viral follow-through is why the "43% of follows" stat exists — packs are follow-amplification engines, and the membership list is the core asset.

Does Bluesky have an API for starter packs?

Yes, but it does not expose keyword search or bulk export. The AT Protocol AppView offers three relevant lexicon methods:

app.bsky.graph.getStarterPack — fetch one pack by AT URI
app.bsky.graph.getActorStarterPacks — list all packs published by a creator handle or DID
app.bsky.graph.getList — retrieve member profiles from the list embedded in a pack

What you cannot do: search packs by keyword. The lexicon defines app.bsky.graph.searchStarterPacks but the public AppView returns XRPCNotSupported (HTTP 404). Discovery by plain-text query is not available on the public tier — to find packs in a topic area, identify a prominent curator and enumerate their packs via creatorHandle.

What the data looks like

Every member row comes back flat and typed — pack metadata is denormalized onto each row so a single CSV export is self-contained. The output shape from src/models.py:

{
  "pack_uri": "at://did:plc:abc123/app.bsky.graph.starterpack/xyz789",
  "pack_name": "AI Researchers on Bluesky",
  "pack_description": "Curated list of ML/AI researchers who migrated from Twitter.",
  "pack_creator_handle": "alice.bsky.social",
  "member_did": "did:plc:def456",
  "member_handle": "bob.bsky.social",
  "member_display_name": "Bob Smith",
  "member_followers_count": 1204,
  "member_following_count": 380,
  "member_posts_count": 841,
  "member_indexed_at": "2024-11-14T09:22:01.000Z",
  "scraped_at": "2026-05-16T12:00:00.000Z"
}

Twelve fields per row. pack_uri, pack_name, pack_creator_handle, member_did, member_handle, and scraped_at are always present. The count fields, member_display_name, and member_indexed_at are nullable — if the API omits them for a profile, the Actor emits the row with those fields set to null rather than dropping it.

The naive approach (and why it falls apart)

The AT Protocol is designed for open access, which is genuinely unusual. The https://public.api.bsky.app/xrpc/ base URL needs no session token. A first response is minutes away. The complexity lives elsewhere.

Cursor pagination across chained endpoints. Getting all members is not a single call. You call getStarterPack to learn the embedded list AT URI, then getList with that URI, which returns a page of members plus a cursor. You loop until the response omits the cursor — for large packs, a dozen sequential calls that all need to succeed and reassemble in order. We thread that loop cleanly, applying the maxMembersPerPack cap as a client-side guard to bound run cost.

Nested record structure. The starterPack view object puts uri and creator at the top level, but name and description live inside a nested record sub-object. A naive parser reading pack["name"] returns None every time; the correct path is pack["record"]["name"]. We pin the parser against this shape and validate output with Pydantic before it reaches your dataset.

Retry discipline on 429/503. The AT Protocol does not publish rate limits, but it enforces them. On 429/503 we retry with exponential backoff — base 2 seconds, doubling, capped at 30 seconds, up to 5 attempts — honouring Retry-After when present. We rotate the curl-cffi browser fingerprint (Chrome / Firefox / Safari TLS profiles) so the handshake looks like a real browser, not a Python script.

Pydantic-validated rows or nothing. Every row passes through ResultRow.model_validate(...) before Actor.push_data(...) writes it. If the API contract drifts and a required field disappears, the Actor fails loud with a clear status message rather than emitting garbage. No data, no charge.

None of this is insurmountable to build yourself. All of it is what you skip by using the Actor.

The Actor

The packaged result is on the Apify Store: Bluesky Starter Pack Scraper.

Two modes — single pack by URI or URL, or bulk export of every pack owned by a creator handle.

Run it via the Apify Python SDK:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

# Single-pack mode — paste a bsky.app URL directly; the Actor normalizes it
run = client.actor("DevilScrapes/bluesky-starter-pack").call(
    run_input={
        "starterPackUri": "https://bsky.app/starter-pack/pfrazee.com/3l2stmy4ote2b",
        "maxMembersPerPack": 500,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["member_handle"], item["member_followers_count"])

Or in creator mode — export every pack a curator has published:

run = client.actor("DevilScrapes/bluesky-starter-pack").call(
    run_input={
        "creatorHandle": "pfrazee.com",
        "maxPacks": 10,
        "maxMembersPerPack": 200,
    }
)

The starterPackUri field accepts both AT URIs and Bluesky web URLs; the Actor normalizes web URLs to AT URI form before the first network call — no manual conversion required.

Use cases

Five concrete scenarios where this data earns its keep:

Academic community-topology research. Export a pack, load it into NetworkX, and run community detection. The member_followers_count and member_following_count fields give you edge weights without a second API call.

B2B growth marketing. Find packs in your industry (e.g. "ML Researchers on Bluesky", "London Founders") via their curator handles, export membership, filter by member_followers_count >= 500, and you have a list of niche-verified, high-signal accounts — warm context for relevant replies, not cold outreach against scraped emails.

Competitive intelligence. Track which accounts influential curators are recommending. A startup appearing in three separate "AI tools" packs in one week is a signal worth noticing. Schedule a weekly run and diff the membership list.

Social graph analysis. Compare follower distributions and post activity across a curated peer group. Export "AI Researchers" and "AI Founders" packs and measure whether the founder community posts more or follows more — one flat CSV, no joins.

OSINT and journalism. Document community formation around an event. A pack published during breaking news captures a snapshot of who organized around it — a record the AT Protocol's mutable graph will not preserve indefinitely.

Pricing — exact numbers 💰

Pay-per-event. You pay for rows written to the dataset, not for idle compute or pages fetched. No data, no charge (beyond the small start fee).

Event	Price
Actor start (once per run)	$0.05
Member row emitted	$0.002

Members scraped	Total cost
100 members	$0.25
500 members	$1.05
1,000 members	$2.05
5,000 members (max per pack)	$10.05

Apify's $5 free trial credit covers your first ~2,470 member rows — enough to export several typical starter packs — with no credit card required.

The technically interesting part

The AT Protocol's URL-normalization step is worth understanding if you build anything else on it.

A Bluesky web URL looks like https://bsky.app/starter-pack/alice.bsky.social/abc123rkey. The canonical form is at://alice.bsky.social/app.bsky.graph.starterpack/abc123rkey. The Actor's ActorInput model handles this in a @field_validator that fires before any network call: strip the https://bsky.app/starter-pack/ prefix, split on / for handle and rkey, reassemble the AT URI. Pass a malformed URL — missing rkey, extra segments — and Pydantic raises a ValidationError before the run bills you for the start event.

The broader point: AT Protocol tools must be careful about this translation layer. The web UI uses handles; the protocol uses DIDs. Denormalizing pack_creator_handle (human-readable) onto every row while storing member_did (machine-canonical) is deliberate — the handle can change; the DID cannot.

Limitations (the honest list)

No keyword search across packs. Discovery is by creator handle, not by topic keyword. The app.bsky.graph.searchStarterPacks lexicon method returns XRPCNotSupported on the public AppView.
Public packs only. Private or invite-only packs are not visible to the unauthenticated API.
Current membership only. The AT Protocol does not expose member join or leave history. This Actor captures a snapshot, not a timeline.
No post content. Member posts are out of scope. Use the companion Bluesky Feed Posts Actor for post data.
Member cap per pack. maxMembersPerPack defaults to 500 (max 5,000). At 5,000 members the per-run cost is $10.05.
7-day dataset retention on Apify's free plan. Export your results or use a named dataset if you need longer retention.

FAQ

Is scraping Bluesky starter packs compliant with their Terms of Service?
Yes. The AT Protocol is an open protocol designed for interoperability and data portability. The public AppView (public.api.bsky.app) is the official mechanism for unauthenticated access. Bluesky has publicly proposed a scraping standard for AI training datasets (Slashdot, 2025). This Actor reads only what the public API exposes.

Is there an official Bluesky API I could use instead?
The three lexicon methods the Actor uses (getStarterPack, getActorStarterPacks, getList) are the official public API. The Actor saves you the pagination loop, URL normalization, retry handling, and Pydantic validation — not a workaround around a locked endpoint.

Can I export the data to Google Sheets or a data warehouse?
Yes. Export CSV, JSON, or Excel from the Apify Console, or webhook the dataset on ACTOR.RUN.SUCCEEDED into Make, Zapier, or n8n. The flat row schema loads into any spreadsheet or SQL table without a join.

Does this export a user's full follower list, or just pack members?
Pack members only. The Actor scopes to the membership of a specific pack (or all packs by a creator). It does not enumerate the general follower graph of any account.

Try it

The Actor is on the Apify Store: apify.com/DevilScrapes/bluesky-starter-pack.

Free $5 trial credit, no credit card. Paste a bsky.app URL, get a flat JSON dataset of every member in under a minute. Researching a niche, mapping a community, or building an outreach workflow? Start there.

If there is a field or an AT Protocol endpoint you wish this Actor covered, drop it in the comments.

Built by Devil Scrapes — Apify Actors for the data-hungry. Pay-per-event, honest pricing, typed rows. 😈

Bluesky Feed Scraper: export any custom or algorithm feed to clean JSON

Devil Scrapes — Sun, 31 May 2026 11:42:02 +0000

Quick answer: The Bluesky AT Protocol exposes a public, unauthenticated app.bsky.feed.getFeed endpoint that returns posts from any custom or algorithm feed. To get that data as a flat, analytics-ready CSV or JSON — with engagement counts and feed metadata on every row — you call that endpoint with cursor pagination, stitch in metadata from getFeedGenerator, normalise DIDs, and handle backoff on rate limits. The Bluesky Feed Posts Scraper does it for $0.002 per post (~$2.05 per 1,000), no Bluesky account required.

Bluesky ships a Feed Generator protocol that lets any developer publish an algorithm. The result is hundreds of community-curated feeds — topic feeds, language feeds, niche hobby feeds — each one a named curator's idea of what the most relevant posts look like. And unlike Twitter/X's opaque ranked feed, the curation logic and the post inventory are publicly queryable.

The catch is that "publicly queryable" and "easily queryable" are not the same sentence. The AT Protocol surfaces the data across three separate endpoints, each cursor-paginated, each returning nested structures that need denormalising before they're useful in a spreadsheet. This is the gap this Actor fills.

What is Bluesky? 🔎

Bluesky is a decentralised social platform built on the AT Protocol — an open federated standard for social data. Every post, like, and follow is a signed, addressable record stored in a Personal Data Server (PDS). A public AppView aggregates those records and serves them over a JSON RPC API at public.api.bsky.app/xrpc/ — no authentication required, no API key, no OAuth dance.

The feed system sits on top of this: a Feed Generator is a small service that responds to getFeed calls with a list of post URIs. Bluesky's own algorithms ("Discover", "What's Hot", "With Friends") are feed generators, and so are the thousands of community-built ones. Each has a stable AT URI in the form at://did:plc:.../app.bsky.feed.generator/<rkey>.

Does Bluesky have an API for feed posts? 🔌

Yes — and it is intentionally public. The AT Protocol AppView exposes app.bsky.feed.getFeed for cursor-paginated post retrieval, app.bsky.feed.getFeedGenerator for feed metadata, and app.bsky.feed.getActorFeeds to enumerate every feed a creator has published. All three are unauthenticated. Bluesky's open-protocol commitments mean this is not an accident: the data portability is by design.

What the API does not give you: a flat, analytics-ready row with feed metadata and engagement counts already joined. It gives you nested JSON, positional cursors, and a DID per author that you carry through the schema yourself. That joining and validation is the whole job.

What the data looks like 📋

Each post comes back as one typed, denormalised row. Feed metadata — display name, creator handle, description — lands on every row so a CSV export is self-contained. Here is a real record from the "Discover" feed:

{
  "feed_uri": "at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.feed.generator/whats-hot",
  "feed_display_name": "Discover",
  "feed_creator_handle": "bsky.app",
  "feed_description": "Trending content from your personal network",
  "post_uri": "at://did:plc:sj5wj7libgr7omqiotenxadx/app.bsky.feed.post/3mlxmr4jyfs2s",
  "post_cid": "bafyreidgimgd7v3g3pazsp5oq7ur6bvedpnwohul26mss7cbffg6bdqjkm",
  "post_indexed_at": "2026-05-16T10:20:40.467Z",
  "post_text": "If you never read the book or saw the movie, you missed one of the greatest Pulitzer Prize winning sagas ever written.",
  "post_lang": "en",
  "post_reply_count": 89,
  "post_repost_count": 414,
  "post_like_count": 1288,
  "post_quote_count": 27,
  "author_did": "did:plc:sj5wj7libgr7omqiotenxadx",
  "author_handle": "louiseplease.bsky.social",
  "author_display_name": "Louise",
  "scraped_at": "2026-05-16T12:00:00+00:00"
}

Seventeen fields, Pydantic-validated before they're written. Every optional field (feed_description, post_lang, author_display_name) is null when the API omits it — rows are never dropped for missing optional data.

The naive approach (and why it falls apart) 🛠️

The obvious path: hit https://public.api.bsky.app/xrpc/app.bsky.feed.getFeed?feed=at://...&limit=100, parse the JSON, paginate until the cursor runs out. It works for the first page. Then the edges appear.

1. DID resolution and AT URI construction. Bluesky web URLs look like bsky.app/profile/bsky.app/feed/whats-hot. The AT Protocol wants at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.feed.generator/whats-hot. Those are not the same string. The DID is what app.bsky.actor.getProfile returns when you look up the handle bsky.app, and a single typo in it returns {"error":"InvalidRequest","message":"could not find feed"}. We resolve web URLs to AT URIs via getProfile on every run so you never have to track which DID is current.

2. Feed metadata is on a different endpoint. getFeed returns post URIs and engagement counts. It does not return the feed's display name, description, or creator handle — those come from getFeedGenerator. For a self-contained dataset you need both calls stitched together. We make the getFeedGenerator call once per feed and denormalise the result onto every row.

3. Cursor pagination and the client-side cap. The public AppView paginates at up to 100 posts per page. A feed with 500 posts needs 5 round trips with cursor threading. A feed with an undefined number of posts needs a sensible client-side cap so the run does not accumulate unbounded cost. We thread cursors, respect the per-feed cap you set (maxPostsPerFeed, default 100, max 5,000), and stop cleanly when the cursor is exhausted or the cap is hit.

4. Rate limits at scale. We retry on 408, 429, and 503 with exponential backoff — base 2 seconds, doubling each attempt, capped at 30 seconds, up to 5 attempts per request — and honour Retry-After headers when the API sets them. We rotate browser fingerprints via curl-cffi so the TLS handshake looks like a real browser client, not a Python script. On a partial-success run we surface the count via Actor.set_status_message rather than returning a green status with a silently truncated dataset.

None of this is conceptually hard. It is just engineering tax that adds up to a weekend from scratch.

The Actor 🚀

I packaged the result as an Apify Actor: Bluesky Feed Posts Scraper.

Paste a feed URI or creator handle in the Apify Console and click Start, or run it programmatically via the Python SDK:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

# Single-feed mode — pull up to 200 posts from Bluesky's Discover feed
run = client.actor("DevilScrapes/bluesky-feed-posts").call(
    run_input={
        "feedUri": "at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.feed.generator/whats-hot",
        "maxPostsPerFeed": 200,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["author_handle"], item["post_like_count"], item["post_text"][:60])

Or in creator-discovery mode — enumerate every feed a creator publishes and pull posts from each:

run = client.actor("DevilScrapes/bluesky-feed-posts").call(
    run_input={
        "creatorHandle": "bsky.app",
        "maxPostsPerFeed": 100,
        "maxFeeds": 10,
    }
)

Two input modes, mutually exclusive. Setting both feedUri and creatorHandle causes the Actor to fail fast with a clear error before any network call — no silent half-runs. You can also paste a bsky.app web URL directly into feedUri — the Actor converts it to AT URI form automatically. The raw AT URI skips the resolution step and runs slightly faster.

Use cases 💡

Algorithm research. Sample what the "Discover" or "What's Hot" feeds surface across multiple days or weeks and track topic drift, amplification patterns, or the ratio of original posts to reposts. The AT Protocol's open data makes this the most researcher-accessible large social feed available right now.

Newsroom social listening. Subscribe to curated topic feeds and pipe new posts into Slack or a Google Sheet via Apify Webhooks. Because feed metadata is denormalised onto every row, the Slack message template needs no join.

NLP corpus building. Collect labelled training data from topic-curated feeds for sentiment models, topic classifiers, or RAG systems. A feed labelled "AI news" or "climate science" gives you weakly supervised labels without manual tagging of raw timelines.

Creator and feed analytics. Pull every post a niche feed generator surfaces and rank by like / repost / quote ratios. Benchmark your own Bluesky posts against what the feed amplifies, and see which content formats dominate the engagement distribution.

Competitive monitoring. Track community-curated feeds that aggregate competitor announcements, support complaints, or product mentions. Creator-discovery mode pulls a creator's full feed catalogue in a single run.

Pricing — exact numbers 💰

Pay-per-event. You pay for posts that land in your dataset. No data, no charge (beyond the $0.05 run warm-up).

Event	Price
Actor start	$0.05 per run
Post row written	$0.002 per row

Posts scraped	Cost
100	$0.25
500	$1.05
1,000	$2.05
5,000	$10.05

The maximum single-run input (50 feeds × 100 posts = 5,000 rows) comes out to around $10.05. Apify's $5 free trial credit covers roughly 2,475 posts — no credit card needed.

The technically interesting part

The AT Protocol uses Content Identifiers (CIDs) — IPLD content-addressed hashes — as the stable identifier for every post record. The post_cid field in each row is the cryptographic fingerprint of the exact post record at time of indexing. Two runs returning the same post_cid for a post_uri are guaranteed to be the same record; two different post_cid values mean the post was edited in between. This makes longitudinal feed studies possible — you can track not just which posts appeared in the feed, but whether their content changed over time.

Limitations 🚧

Private or access-restricted feeds are not exposed by the public AppView API. Only feeds visible at public.api.bsky.app can be scraped.
Global feed discovery by keyword is not supported. Bluesky's getPopularFeedGenerators endpoint returns MethodNotImplemented on the public AppView. Use creator-discovery mode (creatorHandle) to enumerate one creator's feeds.
Post images, embeds, and quoted-post bodies are not extracted. Only the plain-text post_text is captured. Image ALT text and quoted-post content are outside the current schema.
Reply thread expansion is out of scope. Only the top-level post row is emitted. Thread context (parent/root posts) would require additional getPostThread calls.
The maxPostsPerFeed cap is client-side. If a feed has fewer posts than the cap, fewer rows are returned — expected behaviour, not a failure.
Storage retention on Apify's FREE plan is 7 days. Export your dataset immediately after the run, or use a named dataset for longer retention.

FAQ ❓

Is scraping public Bluesky feeds legal?
The AT Protocol is an open, federated standard. public.api.bsky.app is explicitly unauthenticated and publicly accessible without login. The Bluesky Terms of Service permit accessing public data programmatically, as long as you do not impersonate users or violate AT Protocol data-portability principles. Always verify the current Terms of Service and your local jurisdiction's data-protection rules before using scraped data commercially.

Is this a replacement for the Twitter/X API?
No. Bluesky's AT Protocol is a different architecture: the open feed-generator system, content-addressed records, and unauthenticated public AppView are native to its design, not workarounds. If you need Twitter/X data, use a Twitter scraper. If you want Bluesky's unique feed-curation graph, this Actor is built for that.

Can I export to Google Sheets or a data warehouse?
Yes. Export CSV/Excel/JSON from the Apify Console after the run, webhook the dataset on ACTOR.RUN.SUCCEEDED into Make, Zapier, or n8n, or pull via the Apify REST API: GET /datasets/{id}/items?format=csv&clean=true. Because feed metadata is already denormalised onto every row, no pivot or VLOOKUP is needed.

What is a feed URI and how do I find one?
An AT URI looks like at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.feed.generator/whats-hot. Every Bluesky feed also has a bsky.app web URL in the form https://bsky.app/profile/<creator>/feed/<rkey>. You can paste either format into the feedUri field — the Actor converts web URLs automatically. To get all feeds from a creator, use creatorHandle and set maxFeeds.

Try it

The Actor is live on the Apify Store: apify.com/DevilScrapes/bluesky-feed-posts.

Free $5 trial credit, no credit card. Run it against the whats-hot feed URI above with maxPostsPerFeed: 100 and you will have a clean dataset of today's Discover feed posts in under 30 seconds. The AT Protocol documentation and Apify Python SDK docs are the two reference links you will reach for most.

Have a feed analysis use case I haven't covered, or a field you wish was in the output? Drop it in the comments — I ship based on what people actually need.

Built by Devil Scrapes — Apify Actors with attitude. Pay-per-event, transparent pricing, no junk fields. 😈

Bing Search API Replacement: scrape SERP results for $1.05/1K

Devil Scrapes — Sun, 31 May 2026 11:36:47 +0000

Quick answer: Microsoft retired the Bing Search API on August 11, 2025. There is no longer an official endpoint. A Bing search scraper hits the same www.bing.com/search HTML endpoint any browser hits, decodes the bing.com/ck/a redirect wrappers to canonical URLs, and returns clean typed rows — query, position, title, URL, snippet, country, language, timestamp. The Apify Actor below costs $0.001 per row (~$1.05 per 1,000), with TLS fingerprint rotation, residential proxy routing, and Bing's pagination handled for you.

Microsoft's retirement notice landed on August 11, 2025, and thousands of teams who had wired Bing SERP data into rank trackers, brand-monitoring dashboards, and RAG pipelines found themselves holding a dead api.bing.microsoft.com call. The official replacement — "Grounding with Bing Search" inside Azure AI — is gated behind an Azure subscription and priced for LLM workflows, not raw SERP pulls. SerpApi and DataForSEO fill the gap at $75–225/month. DIY means debugging redirect wrappers at 11pm.

This post covers where a naive scraper breaks and how I packaged the fix as a cheap Apify Actor.

What is Bing Search? 🔎

Bing is Microsoft's web search engine. For SEO practitioners it matters because its ranking signals diverge from Google's — it weights on-page keyword density and anchor text differently, surfaces more forum content, and runs its own crawler (bingbot) with independent crawl frequency. For AI and RAG applications, Bing's index includes different long-tail pages than Google's; an LLM agent that only queries one engine has blind spots the other would fill.

The SERP itself — www.bing.com/search?q=<query> — is a server-rendered HTML page. It returns 10 organic <li class="b_algo"> blocks per page, paginated via &first=0,10,20,.... Everything the old API gave you (titles, URLs, snippets) is in that HTML.

Does Bing have an API for search results?

No — as of August 11, 2025, Microsoft has no official API for Bing organic search results. The Bing Web Search API v7 was retired on that date. The Azure AI "Grounding with Bing Search" product is a different surface, gated to the Azure OpenAI workflow and not available as a standalone SERP endpoint for arbitrary queries.

The only programmatic path to Bing SERP data is the public HTML endpoint the browser hits — which is what this Actor uses.

What the data looks like

Each organic result becomes one flat, Pydantic-validated row. Every field is verified against the ResultRow model in src/models.py before it reaches your dataset — no positional-array guessing, no silent null promotion:

{
  "query": "bing search api replacement",
  "position": 1,
  "title": "Bing Search Scraper — SERP organic results to JSON",
  "url": "https://apify.com/DevilScrapes/bing-search-scraper",
  "displayed_url": "https://apify.com › DevilScrapes › bing-search-scraper",
  "snippet": "Drop-in replacement for the retired Bing Search API. Returns title, URL, snippet, position for any query and locale.",
  "country": "US",
  "language": "en",
  "scraped_at": "2026-05-16T13:40:00.000Z"
}

Nine fields. The url is always the canonical destination — never a bing.com/ck/a redirect wrapper. The displayed_url is the breadcrumb Bing renders on screen. position is 1-indexed across all paginated pages, so position 11 is the first result on page 2 — no off-by-one math on your side.

The naive approach (and why it falls apart) 🔧

The SERP is just HTML, so the obvious path looks like:

import requests
from parsel import Selector

resp = requests.get("https://www.bing.com/search", params={"q": "bing api replacement"})
sel = Selector(text=resp.text)
for li in sel.css("li.b_algo"):
    print(li.css("h2 a::text").get())

This will fail in production. Three reasons, and they are why a hosted Actor earns its keep:

1. TLS fingerprint inspection. Bing inspects the JA3/JA4 signature of your TLS handshake. Python's stdlib ssl, requests, and httpx emit fingerprints no real browser produces — the server returns 403 before reading the query string. We route every request through curl-cffi's AsyncSession, impersonating Chrome 131, Chrome 124, or Firefox 147 TLS + HTTP/2 SETTINGS frames at the socket level, rotating profiles per page to reduce burst correlation.

2. The bing.com/ck/a redirect wrapper. Every organic href is a click-tracking redirect of the form https://www.bing.com/ck/a?...&u=a1<base64url>.... A naive parser hands you that wrapper, which joins to nothing downstream. We decode the u=a1<base64url> on every href — the url field in your dataset is always the canonical destination.

3. Proxy continuity across pagination. Bing serves 10 results per page. Rotating proxies mid-query degrades results: fewer organic blocks, more ads, eventually empty pages. We thread Apify residential proxies with a stable session_id per query's page-burst, then rotate on the next query. On 429/503 we retry with exponential backoff (2s → 4s → ... capped at 30s, max 5 attempts, honouring Retry-After). A query that exhausts retries doesn't abort the run — partial success surfaces via status message.

The Actor ⚙️

The result is packaged as an Apify Actor: Bing Search Scraper.

Run it from the Apify Console by pasting your queries into the input form, or call it programmatically:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("DevilScrapes/bing-search-scraper").call(
    run_input={
        "queries": [
            "bing search api replacement",
            "bing serp scraper",
            "scrape bing results python",
        ],
        "country": "US",
        "language": "en",
        "maxResultsPerQuery": 30,
        "useProxy": True,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["position"], item["url"], item["snippet"])

Input parameters:

Field	Type	Default	Notes
`queries`	string[]	required	One row emitted per (query × result)
`country`	string	`US`	ISO-3166-1 alpha-2, passed as `cc=`
`language`	string	`en`	ISO-639-1, passed as `setlang=`
`maxResultsPerQuery`	int 1–100	`30`	Capped at 100; Bing quality degrades past ~page 5
`useProxy`	bool	`true`	Apify residential proxy; strongly recommended

The output streams to the default dataset as the scrape progresses — you can read partial results before the run finishes.

What you'd actually use this for 📊

Bing Search API migration. If you have any Python code calling api.bing.microsoft.com, it is dead. The Actor's input and output map to the same logical shape: queries in, ranked organic results out. The migration work is replacing the client call, not re-architecting the downstream.

SEO rank tracking across locales. Run the same query list with country=US, country=GB, country=DE on a weekly schedule and chart position drift over time. Locale-specific Bing rank tracking is hard to buy — most tools are Google-first.

Brand monitoring. Pull the top 30 results for branded queries and alert when a new domain enters the top 10. At 100 queries × 30 results that is 3,000 rows per run — $3.05 total including the actor-start fee.

AI agent retrieval / RAG diversification. LangChain's BingSearchAPIWrapper and LlamaIndex's BingToolSpec both called the now-dead endpoint. An Apify Actor call is the drop-in. Bing's index skews toward forum content and long-tail pages where Google's top results are SEO-dominated — the diversification is real.

Pricing — exact numbers 💰

Pay-per-event. You pay only for rows that land in your dataset. No data, no charge (beyond the warm-up fee).

Event	Rate (USD)	When
`actor-start`	$0.05	Once per run
`result-row`	$0.001	Per organic SERP row pushed

Pull	Cost
100 rows	$0.15
300 rows (10 queries × 30)	$0.35
1,000 rows	$1.05
3,000 rows (100 queries × 30)	$3.05
10,000 rows	$10.05

Apify's free $5 trial credit covers roughly 4,950 rows — no credit card required. For comparison, SerpApi's entry plan runs $75/month, and DataForSEO requires a separate account plus API key management. This Actor undercuts on simplicity and price by going direct to Bing's HTML endpoint.

The part worth understanding technically

The redirect decoder is the technically non-obvious piece. Bing wraps every organic href in bing.com/ck/a?!&&p=<hash>&u=a1<base64url>.... The u= parameter carries the destination URL prefixed with a1 and encoded as URL-safe base64:

import base64, re

def decode_bing_href(href: str) -> str:
    match = re.search(r"[?&]u=a1([A-Za-z0-9_-]+)", href)
    if not match:
        return href  # already canonical
    return base64.urlsafe_b64decode(match.group(1) + "==").decode("utf-8", errors="replace")

Bing has kept this format stable since at least 2023, but it's the detail that causes a naive scraper to silently fill your dataset with bing.com/ck/a junk that joins to nothing. Every URL is decoded before it reaches the dataset.

Limitations 🚧

Organic web results only. People-Also-Ask, news, images, videos, and local packs are out of scope for v1.
100-result ceiling per query. Bing's HTML becomes unreliable past page 10. The Actor stops paginating rather than returning garbage.
No is_featured flag. Featured snippets that render inside b_algo blocks appear as regular rows; v1 has no way to distinguish them.
No JavaScript rendering. The current path hits server-rendered HTML (verified working 2026-05-16). A Bing move to client-rendering would require a Camoufox upgrade.
Snapshots only. Every run is a fresh snapshot. For time-series tracking, schedule runs and export to named Apify datasets or your own storage. Free-plan default storage purges after 7 days.
Country/language passed through, not validated. Unrecognized ISO codes silently return default results.

FAQ ❓

Is scraping Bing search results legal?
The Actor sends well-formed HTTP requests to the same public, unauthenticated page any browser accesses — no authentication bypass, no paywall, no personal data. Comply with Bing's Terms of Service and your jurisdiction's applicable law.

Can I export results to Google Sheets or a data warehouse?
Yes — download CSV, Excel, JSON, or XML from the Apify Console Export button, or use the ACTOR.RUN.SUCCEEDED webhook to push the dataset into Make, Zapier, or n8n. Raw API access via the Apify Dataset API.

Is there still an official Bing search API?
No. The Bing Web Search API v7 was retired August 11, 2025. Azure AI "Grounding with Bing Search" is a different surface — an LLM-workflow integration, not a raw SERP endpoint for arbitrary queries.

Why are the redirect URLs always decoded?
The bing.com/ck/a?...&u=a1<base64url> wrapper carries no information your workflow can use. Canonical URLs join cleanly with any URL-keyed dataset. The decode is always on; there is no option to keep the raw wrapper.

Try it

The Actor is live on the Apify Store: apify.com/DevilScrapes/bing-search-scraper.

Free $5 trial credit, no credit card. Paste bing search api replacement as your first query and you'll have 30 ranked results in your dataset in under a minute. Building something on top — a rank tracker, a brand-monitoring webhook, a RAG pipeline? Drop the use case in the comments. If there's a feature the README doesn't cover (PAA boxes, news vertical, location targeting), that's how v2 gets scoped.

Built by Devil Scrapes — Apify Actors with attitude. Pay-per-event, transparent pricing, no junk fields. 😈

ATS Tech Stack Detector: pull company back-end stacks from jobs for $5.05/1K

Devil Scrapes — Sun, 31 May 2026 11:31:32 +0000

Quick answer: Greenhouse, Lever, and Ashby each publish a public job-board API that any job aggregator can hit — no auth required. An ATS tech stack detector calls those APIs, strips the HTML from each job description, then runs a curated vocabulary of ~110 canonical tech names against the text to produce a deduplicated detected_techs list per job row. The Apify Actor below does it for $0.005 per row ($5.05 per 1,000 jobs, including the one-time $0.05 start fee), with exponential backoff, per-company fault isolation, and Pydantic-validated output ready to drop into any CRM pipeline.

Every B2B sales motion eventually hits the same wall: you need to know what your prospect actually runs in production, and front-end sniffers like BuiltWith and Wappalyzer can't tell you. They read Cloudflare headers, tag-manager IDs, and JS bundle manifests — which surfaces HubSpot and Segment beautifully but says nothing about whether the company runs Postgres or Snowflake, Kafka or RabbitMQ, Kubernetes or Nomad. Engineering teams declare their real stack in job descriptions. A "Senior Backend Engineer" posting that lists Django, Postgres, Kafka, and Kubernetes tells you more about their infrastructure than any front-end scan ever will.

The hard part is extracting that signal at scale: hit the APIs, unpick the HTML, normalize "node.js" vs "Node.js" vs "NodeJS", deduplicate, then keep the vocabulary current as new tools ship. Or call one Actor and get a typed dataset back. Here's how the second option works.

What is an ATS? 🔎

An Applicant Tracking System (ATS) is the software companies use to post jobs, accept applications, and manage candidates. The three platforms this Actor covers — Greenhouse, Lever, and Ashby — collectively handle a large share of Series A+ tech-company hiring, and all three publish their job-board data via public, unauthenticated REST APIs so that aggregators and "Powered by Greenhouse" listings can pull postings without a login.

That public-API design is the foundation this Actor builds on. We call the same endpoint a job-board aggregator would, then add what they don't: tech-stack extraction from the description text, normalization to a canonical vocabulary, and a flat, validated output schema.

Does Greenhouse have an API for job postings?

Yes. Greenhouse's Job Board API (boards-api.greenhouse.io/v1/boards/{token}/jobs?content=true) returns every active posting for any company with a public board, including the full description HTML. Lever exposes api.lever.co/v0/postings/{token} with the same shape, and Ashby uses api.ashbyhq.com/posting-api/job-board/{token} — case-sensitive on the token (Ramp works; ramp returns zero). None require an API key or OAuth.

What the data looks like

Each job posting produces one flat, typed row. Every field comes directly from the ATS response or from our detection pass — nothing is inferred or synthesised:

{
  "ats": "greenhouse",
  "company_token": "airtable",
  "job_id": "4567890",
  "title": "Senior Backend Engineer",
  "location": "Remote, US",
  "department": "Engineering",
  "url": "https://boards.greenhouse.io/airtable/jobs/4567890",
  "description_text": "We're looking for a backend engineer with deep experience in Python, Django, Postgres and Redis. Our data pipeline runs on AWS with Kubernetes...",
  "detected_techs": ["AWS", "Django", "Kubernetes", "Postgres", "Python", "Redis"],
  "posted_at": "2026-05-01T09:00:00+00:00",
  "scraped_at": "2026-05-31T14:22:11+00:00"
}

Eleven fields, the same shape every time. detected_techs is sorted (case-insensitive) and deduplicated. Every row is Pydantic-validated before it is written, so you never receive a missing required field or an ats value outside ["greenhouse", "lever", "ashby"].

The naive approach (and why it falls apart)

The first attempt looks straightforward: hit the API, grab the description, split on whitespace, look for known tech names. It breaks almost immediately, and the breakage is instructive.

1. HTML encoding layers. Greenhouse double-encodes its content field — the HTML you receive contains <div> where the original had <div>. A single html.unescape() leaves literal <div> tags in the text that your regex then trips over. The parser unescapes twice, then strips tags, before the vocabulary scan runs.

2. Lever's split description. Lever's descriptionPlain often omits everything inside "Requirements" bullets — that content lives in lists[].content chunks in a separate array. Read only descriptionPlain and you silently miss half the tech signals in engineering roles. We concatenate every lists[].content chunk before scanning.

3. Case-sensitivity on Ashby tokens. Write ramp instead of Ramp and Ashby's API returns an empty list — not a 404, just zero jobs. Without a guard, you produce an empty dataset and think you scraped it. The Actor fails loud when every company returns zero rows rather than reporting a hollow success.

4. Rate-limiting. All three APIs throttle undeclared bursts. We retry on 429 and 503 with exponential backoff (2 seconds, doubling, capped at 30 seconds, up to 5 attempts), and we honour Retry-After when the server sends it. On multi-company runs, one company isolating a rate limit doesn't abort the rest — each company's fetch loop is isolated, so partial success is partial success, not total failure.

5. Vocabulary normalization. Job descriptions use "Node.js", "NodeJS", "Node JS", and "Node" to mean the same runtime, and "postgres", "Postgres", and "PostgreSQL" for the same database. A naive substring match either misses casing variants or fires on the wrong word ("Java" inside "JavaScript"). The vocabulary uses case-insensitive word-boundary regex (\bPostgres\b), longest-match-first, so the canonical name is emitted regardless of how the author capitalised it and substrings never false-fire. Underneath, every request goes out through curl-cffi impersonating a real Chrome 131 TLS + HTTP/2 fingerprint, we thread Apify residential proxies when you flip the useProxy flag on, and we hand back Pydantic-validated typed rows — no data means no charge.

None of that is complicated in hindsight. All of it is the difference between a script that worked on three companies in a notebook and a pipeline that survives 500 companies overnight.

The Actor ⚙️

I packaged this as an Apify Actor: ATS Tech Stack Scraper.

You can run it from the Apify Console by filling in the input form, or call it programmatically:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("DevilScrapes/ats-tech-stack-scraper").call(
    run_input={
        "companies": [
            {"companyToken": "airtable", "atsType": "greenhouse"},
            {"companyToken": "palantir", "atsType": "lever"},
            {"companyToken": "Ramp", "atsType": "ashby"},
        ],
        "maxJobsPerCompany": 100,
        "minTechsDetected": 2,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["company_token"], item["title"], item["detected_techs"])

minTechsDetected: 2 is a useful floor that drops sales, marketing, and ops roles mentioning no engineering tools — you keep only rows where the vocabulary actually fired. maxJobsPerCompany caps spend on large companies with hundreds of open roles.

Finding a company's ATS token — it's the board slug in the careers URL: boards.greenhouse.io/{token} (e.g. airtable), jobs.lever.co/{token} (e.g. palantir), or jobs.ashbyhq.com/{Token} (e.g. Ramp — case-sensitive; lowercase returns zero jobs).

Use cases 💡

Three buyer types, five concrete scenarios:

1. B2B sales qualification. You sell a Postgres performance tool. Filter detected_techs for "Postgres" and sort by posted_at recency — any company actively hiring "Senior Backend Engineer" with Postgres in the JD is pre-qualified, and you know they're staffing that stack now, not three years ago.

2. Recruiter sourcing. Filling a Rust role? Pull every engineering posting from your target-company list, filter detected_techs for "Rust", and you have a live shortlist of companies already running it in production. Pair with minTechsDetected: 3 to exclude postings that mention Rust only in passing.

3. Competitive intelligence. Track which competitors are posting Kubernetes, Snowflake, or dbt roles. A cluster of new infra-heavy postings often signals a platform migration before it becomes a press release.

4. CRM enrichment. Feed the output into Clay, HubSpot Operations Hub, or Attio as a custom enrichment step — replacing a BuiltWith seat for the subset of back-end and data-platform signals engineering-buying teams actually need.

5. Investment research. Map a pipeline target's tech footprint across quarters. A company that moves from "MySQL" to "Postgres" and "Snowflake" postings in eighteen months is scaling its data stack — a signal front-end sniffers miss entirely.

Pricing — exact numbers 💰

Pay-per-event. You pay for rows you get; rows that don't land cost nothing.

Event	Price	When
`actor-start`	$0.05	Once per run
`result-row`	$0.005	Per job row emitted

Rows pulled	Cost
50 rows	$0.30
200 rows	$1.05
1,000 rows	$5.05
10,000 rows	$50.05

For a 200-company qualification pass (roughly 2,000–4,000 rows), this Actor costs $10–$20 — no subscription, no seat limit, no contract. Apify's $5 free trial credit covers your first ~990 rows with no credit card.

The technically interesting bit

The vocabulary is the product. ~110 canonical tech names sounds small, but precision matters more than recall here — a false positive ("Java" matching "JavaScript" without word boundaries) poisons the very signal buyers are paying for. Every entry uses case-insensitive word-boundary matching (\bPostgres\b), and the alternation is sorted longest-first so "PostgreSQL" wins over "Postgres" on a shared prefix while "Javanese" and "Djangocon" never fire. Buyers in this category have been burned by LLM-based detection inventing plausible-sounding stacks; a deterministic, auditable regex is the antidote. Miss a tool? Open a feedback ticket and it ships in the next version — no model fine-tuning required.

Limitations 🚧

Regex detection only. Tools outside the curated ~110-name vocabulary won't appear in detected_techs. Missing one? Submit a feedback request on the Store page and it ships in the next release.
You supply the token. The Actor requires the ATS board token — it does not discover which ATS a company uses or look it up by company name. No LinkedIn or Workday support.
Greenhouse double-encoding edge cases. The parser handles the common double-encode; exotic Greenhouse configs may leave extra HTML fragments in description_text.
Ashby case-sensitivity is on you. Pass ramp when the token is Ramp and you get zero rows — find the correct casing in the Ashby job-board URL.
7-day default storage. On the FREE plan, default datasets expire after 7 days. For persistent pipelines, open a named dataset: Actor.open_dataset(name="my-stack-data").

FAQ

Is it legal to scrape job boards?
These endpoints are public APIs published intentionally by Greenhouse, Lever, and Ashby so that job aggregators can distribute postings. The Actor requests only publicly visible job post data at a rate the APIs can handle, collects only company-aggregated tech signals (no personal applicant data), and bypasses no authentication. As always, review your own jurisdiction and intended use case before running in production.

Can I export the results to a spreadsheet or data warehouse?
Yes. The Apify Console exports CSV, Excel, and JSON directly from the dataset view. You can also webhook the dataset on ACTOR.RUN.SUCCEEDED into Make, Zapier, or n8n, or pull it programmatically via the Apify API.

Is there an official API for this?
The Actor itself is the programmatic interface — call it via the Apify Python client or REST API. The underlying Greenhouse, Lever, and Ashby job-board APIs are official and publicly documented.

How do I find a company's ATS if I don't know it?
Look for the "Powered by Greenhouse / Lever / Ashby" badge on their careers page, or inspect the job-listing URL — the token is the path segment between the domain and /jobs/.

Try it

The Actor is live on the Apify Store: apify.com/DevilScrapes/ats-tech-stack-scraper.

Free $5 trial credit, no credit card. Drop in Airtable on Greenhouse and Ramp on Ashby, set minTechsDetected: 2, and you'll have a cross-company tech-stack dataset in under a minute. Building a Clay recipe or a qualification workflow and want a specific output field? Leave a comment — I build based on what people actually need.

Further reading:

Built by Devil Scrapes — Apify Actors that do the dirty work so your dataset stays clean. 😈

Google AI Overview Tracker: 8-selector battery + citation drift telemetry

Devil Scrapes — Sun, 31 May 2026 11:26:18 +0000

Quick answer: Google publishes no API for AI Overview citations. The only way to get the data programmatically is to render Google SERPs in a real browser and parse the citation carousel client-side. The Google AI Overview Citation Tracker does exactly that — one Pydantic-validated row per (query × cited source) at $5.50 per 1,000 rows, with selector-drift telemetry so you know when Google rotates its markup before your dashboard goes dark.

Answer Engine Optimization has a measurement problem no major SEO platform has solved. Ahrefs, Semrush, and Sistrix track your domain's SERP rank, but AI Overview appears above position 1 for roughly 30% of informational queries in 2026, and its citations are drawn from a different pool than your normal rankings. You can rank position 1 and still be invisible in AI Overview while a competitor's 2022 blog post gets cited six times — with no backlink your SEO tools can detect. That gap, structured per-query citation data you can query against a competitor list, is what this Actor closes.

What is Google AI Overview? 🔎

Google AI Overview is the AI-generated summary block at the top of Google's search results for informational queries. It rolled out broadly in the US in May 2024 and expanded globally through 2025 — Google's generative-AI answer inside the SERP, its response to Perplexity and ChatGPT Search. For a query like "what causes inflation", it renders a 3-5 sentence synthesis with a carousel of 4-8 cited sources below it.

The citations are the commercially interesting part. Those cited domains get free brand impressions, click-throughs, and authority signals that traditional SEO tools never surface. The shift is large enough that some publishers have watched informational-query traffic fall 20-40% even while their SERP rank held steady.

Does Google AI Overview have an API? 📡

No. As of 2026, Google publishes no official API, export endpoint, or structured feed for AI Overview citations. The only programmatic surface is what the browser renders client-side. Google's Search Central documentation covers AEO best practices but provides no access to citation data. To collect it at scale you render real Google SERPs in a real browser and parse the output yourself — the entire reason this Actor exists instead of a three-line API call.

What the data looks like

Every citation in an AI Overview carousel produces one flat, typed row:

{
  "query": "what causes inflation",
  "country": "us",
  "language": "en",
  "ai_overview_appeared": true,
  "ai_overview_text_excerpt": "Inflation is caused by a combination of demand-pull factors, cost-push factors...",
  "citation_position": 1,
  "source_domain": "imf.org",
  "source_url": "https://www.imf.org/en/Publications/fandd/issues/Series/Back-to-Basics/Inflation",
  "source_title": "Inflation: Prices on the Rise",
  "selector_used": "div[aria-label=\"AI Overview\"]",
  "scraped_at": "2026-05-16T20:50:00.000Z"
}

When AI Overview did not appear for a query, the Actor still emits a row — ai_overview_appeared: false, all citation fields null. That absence is itself a valid AEO signal: you need to know which queries don't trigger AI Overview today, because that changes.

Eleven fields total, validated through Pydantic v2 ResultRow.model_validate before writing. Drop it straight into BigQuery, Sheets, or a pandas pivot — no positional-array wrangling on your side.

The naive approach (and why it falls apart) 🔩

The mental model most people start with: open DevTools, find whatever request the SERP makes, replay it in Python. Three failure modes kill that before the first result lands.

1. Google hard-blocks datacenter IPs. Our recon showed the sorry/index reCAPTCHA interstitial appearing within one second for direct-IP requests, regardless of fingerprint quality. Proxy is load-bearing, not optional. We thread Apify residential proxies, rotate the session ID per query (Apify's session_id regex requires ^[\w._~]+$ — no hyphens), and fall back to BUYPROXIES94952 when residential is unavailable on your plan.

2. AI Overview lazy-renders client-side. The carousel appears 5-7 seconds after domcontentloaded via a separate async render pass — a tool that scrapes the raw HTML response gets nothing, because the container does not exist in the initial DOM. We render with Camoufox (the Firefox fork with anti-detection patches our org mandates per ADR-0002) and wait a configurable 4-15 seconds for the overlay to settle before probing.

3. Google rotates the AI Overview markup. This kills scrapers quietly. Since launch in May 2024, the container's identifying attributes have changed at least three times. A scraper that hardcodes div[aria-label="AI Overview"] works until Google A/B-tests a new attribute, then silently returns zero citations.

We absorb all three. We rotate browser fingerprints through Camoufox's Firefox TLS and navigator stack, and on 408 / 429 / 5xx or a CAPTCHA intercept we rotate the proxy session and retry once before emitting a marker row. We back off when Google rate-limits, and surface partial success with a clear Actor.set_status_message — we never silently return an empty dataset. The selector_used field makes drift detection a single SQL query, which brings us to the most interesting part of this build.

The Actor: 8-selector fall-through battery 🎯

I packaged this as an Apify Actor: Google AI Overview Citation Tracker. The selector battery is the load-bearing decision — eight selectors probed in priority order, first hit wins:

Priority	Selector	Origin
1	`div[aria-label="AI Overview"]`	Current canonical (2026)
2	`div[data-attrid="AI Overview"]`	2025 rotation
3	`div[data-attrid="wa:/description"]`	Historical knowledge-panel reuse
4	`div[jsname][data-rl="ai_overview"]`	2025 rotation
5	`div[data-async-context*="ai_overview"]`	Async-loaded variant
6	`div#m-x-content`	Mobile SGE legacy id
7	Any `h1/h2/h3` whose text starts `AI Overview`	Last-resort text fallback
8	`div` containing `h2` with text `AI Overview`	Last-resort structural fallback

Every row records which selector fired. When selector 1 stops winning and selector 4 starts, open an issue — we'll add the new attribute to the battery.

Run it from the Apify Console or programmatically:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("DevilScrapes/ai-overview-citations").call(
    run_input={
        "queries": [
            "best CRM for startups 2026",
            "what causes inflation",
            "how to reduce churn rate",
        ],
        "country": "us",
        "language": "en",
        "maxQueries": 25,
        "waitMsAfterLoad": 8000,
        "proxyConfiguration": {
            "useApifyProxy": True,
            "apifyProxyGroups": ["RESIDENTIAL"],
        },
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["ai_overview_appeared"]:
        print(item["source_domain"], item["citation_position"], item["query"])

Input accepts 1-50 queries per run, plus country (ISO-3166 alpha-2 → gl=) and language (ISO-639-1 → hl=) for locale targeting. waitMsAfterLoad (default 8000ms) controls how long the Actor waits after domcontentloaded before probing — raise it to 12000-15000ms for slow proxy exits.

Use cases

AEO dashboard. Schedule a weekly run for your 50 highest-priority informational queries. Chart source_domain share-of-citation over time alongside ai_overview_appeared rate, and catch when a competitor first appears in the carousel for a query where you rank position 1. A 50-query run yields roughly 95 rows — about $0.53 per run.

Competitive citation gap analysis. Run the 20 queries you want to rank for and map which domains Google currently cites for them. That list is your outreach shortlist — a mention from a site Google already trusts beats generic link-building.

Brand monitoring. Run your core product-category queries weekly and alert when your domain drops out of the citation set — or when a direct competitor appears. Most brands have no instrumentation here.

Localized AEO comparison. Run identical query lists with country=us vs country=gb. Citations for "best mortgage rates" differ sharply between US and UK — different markets entirely.

Pricing — exact numbers 💰

Pay-per-event: actor-start is $0.05 once per run, result-row is $0.005 per row written (citation hit or no-AI-Overview marker). You pay only for rows that land in your dataset.

Scenario	Rows	Cost
10-query spot check (~30% hit rate, ~4 citations/hit)	~19	~$0.15
50-query weekly AEO audit	~95	~$0.53
500-query category sweep	~950	~$4.80
1,000-row dataset (effective rate)	1,000	~$5.50

The $5.50/1,000 effective rate sits above commodity SERP scrapers because this citation data is essentially unavailable elsewhere at this granularity. Ahrefs and Semrush are beginning to ship AEO modules at $300-1,500/month — and they only track your own domain. Apify's $5 free trial credit covers roughly 900-950 rows, no credit card needed.

The technically interesting part: why we record which selector fired

The selector_used field is deliberately shipped operational telemetry. Google has rotated the AI Overview container's attributes multiple times since launch in 2024, and each rotation silently kills scrapers that hardcode one selector — the parser falls through to empty, the dataset looks fine, until someone notices the citation count dropped to zero. Recording which of the 8 selectors matched on every row turns that into a dead-simple query against your own dataset:

SELECT selector_used, COUNT(*) as hits, DATE(scraped_at) as day
FROM your_aeo_dataset
GROUP BY 1, 3
ORDER BY 3 DESC, 2 DESC;

When the distribution shifts — selector 1 dropping, selector 4 climbing — you get a 24-48 hour warning before coverage degrades. The alternative is waking up to a week of empty citation data and no idea why.

Limitations (the honest list) 🚧

AI Overview triggers on roughly 30% of queries. Transactional, navigational, and trademark-heavy queries mostly produce ai_overview_appeared=false marker rows; informational queries (what is, how to, best X 2026) have the best trigger rate. Marker rows are charged at the same per-row rate — the absence is a valid data point.
v0.1 is English-tuned. The text-based fallback selectors (7 and 8) match the literal string AI Overview, so non-English locales may produce false negatives on that path. The CSS battery (selectors 1-6) is locale-agnostic.
Apify Proxy is required — not optional. Google hard-blocks Apify datacenter IPs; without proxy enrichment the Actor fails fast at startup with a clear status message. FREE tier gets BUYPROXIES94952 (5 datacenter IPs, higher CAPTCHA rate); paid plans with RESIDENTIAL get substantially cleaner runs.
Mobile SERP is out of scope for v0.1. Mobile AI Overview has a different DOM structure; a mobile-variant Actor is planned.
The Actor records citation URLs but does not follow them. For destination page content, pair it with a downstream HTTP scraper.

FAQ

Is scraping Google Search results legal?
The data returned is public — the same content anyone sees in a browser. This Actor reads only what Google renders in the public SERP, at a paced rate with per-query session isolation, and collects no personal data. hiQ Labs v. LinkedIn (9th Circuit, 2022) affirmed that scraping publicly accessible data is not a CFAA violation. Legality still varies by jurisdiction and use case — review Google's Terms of Service and your local regulations for your situation.

Can I export the data to Sheets, CSV, or a data warehouse?
Yes. The Apify Console downloads CSV / Excel / JSON directly from the dataset view. You can also webhook the dataset on ACTOR.RUN.SUCCEEDED into Make, Zapier, or n8n, or pull it via the Apify API using the datasetId from the run response.

Is there an official Google API for AI Overview citations?
No. As of 2026, Google provides no API or structured export for AI Overview citation data. Google Search Central documents general AEO guidance but no programmatic citation access. This Actor is the practical alternative.

Why emit a row even when AI Overview didn't appear?
Because the absence is meaningful AEO data. Run the same query set weekly and you want the ai_overview_appeared rate over time — when a query transitions from non-triggering to triggering, that's the moment a citation opportunity opens. Marker rows make the transition visible, charged at the same $0.005 per-row rate.

Try it

The Actor is on the Apify Store: apify.com/DevilScrapes/ai-overview-citations.

Free $5 trial credit, no credit card. Run it on your 10 most important informational queries and you'll have the citation breakdown in your dataset within minutes. Find a selector miss, a locale that doesn't work, or a field you wish it returned? Drop it in the comments — real reported drift is exactly what I build the next selector battery from.

Built by Devil Scrapes — Apify Actors with attitude. Pay-per-event, transparent pricing, no junk fields. 😈

arXiv Scraper: build a RAG-ready paper corpus for $1.50/1K

Devil Scrapes — Sun, 31 May 2026 11:21:04 +0000

Quick answer: arXiv publishes an Atom feed API at export.arxiv.org/api/query that any program can call — but it rate-limits hard, paginates awkwardly, and ships XML you still have to wrangle into rows. An arXiv scraper wraps that feed with proper pagination, retry logic, and rate-limit pacing, then returns every matching paper as structured JSON. The Apify Actor below does it for $0.0015 per paper (~$1.50 per 1,000), handling the failure modes so your embedding pipeline sees clean rows instead of dropped requests.

There is a moment every ML engineer knows: you have picked your retrieval architecture, your vector store is standing by, and you just need a clean slice of arXiv papers to run benchmarks on. So you open the arXiv export API docs, write twenty lines of Python, and an hour later you are still debugging timeouts, malformed Unicode in author names, and a cursor that resets when you change the start offset without the right headers.

The papers are not the problem. Getting them out is.

What is arXiv? 📄

arXiv (pronounced "archive") is a free, open-access preprint server operated by Cornell University since 1991. Researchers in physics, mathematics, computer science, quantitative biology, statistics, and economics deposit pre-peer-review papers there before — and sometimes instead of — formal journal submission.

For machine learning, it is effectively the primary literature. Every major model, benchmark, and technique shows up on arXiv before it shows up anywhere else. The cs.AI, cs.LG, and cs.CL subcategories receive hundreds of new papers every weekday. As of 2026, the total corpus sits above 2.3 million papers.

For researchers and engineers building retrieval systems, evaluation corpora, or trend-monitoring pipelines, arXiv is an obvious source of truth — if you can get the data out reliably.

Does arXiv have an API?

Yes — but it is more limited than it looks. arXiv runs a public Atom feed API at export.arxiv.org/api/query. You can query by title, author, category, abstract, or combinations. It supports sorting by submission date or last-update date. Pagination works through start and max_results parameters.

What the documentation underplays: arXiv's own guidelines request no more than one request every three seconds from a single client. On a 30,000-paper category sweep at 50 papers per page, that is 600 pages, 1,800 seconds of wait time, and dozens of transient network failures to handle across a 30-minute run. The API also returns Atom XML, not JSON — which is fine for a one-off, and tedious for a pipeline.

This is the gap the Actor fills.

What the data looks like

Each paper comes back as one flat, typed row. Here is a real example with every field the Actor returns:

{
  "arxiv_id": "2401.12345v2",
  "url": "https://arxiv.org/abs/2401.12345v2",
  "pdf_url": "https://arxiv.org/pdf/2401.12345v2",
  "title": "Scaling Laws for Sparse Mixture-of-Experts Language Models",
  "summary": "We investigate scaling laws for sparse mixture-of-experts (MoE) language models...",
  "authors": ["Alex Doe", "Jamie Smith", "Morgan Lee"],
  "primary_category": "cs.CL",
  "categories": ["cs.CL", "cs.LG", "cs.AI"],
  "doi": "10.18653/v1/2024.acl-long.001",
  "journal_ref": "Proceedings of ACL 2024",
  "comment": "18 pages, 7 figures. Accepted at ACL 2024.",
  "published": "2024-01-22T16:00:00+00:00",
  "updated": "2024-03-15T09:30:00+00:00",
  "scraped_at": "2026-05-31T11:00:00+00:00"
}

Fourteen fields, Pydantic-validated before they reach your dataset. arxiv_id includes the version suffix; published and updated are ISO-8601 UTC; doi and journal_ref are null for preprints that have not been formally published. That is the full schema — no hidden fields, no surprise null rows.

The naive approach (and why it falls apart) 🔧

The first script most engineers write looks something like this:

import requests, time
base = "https://export.arxiv.org/api/query"
for start in range(0, 3000, 50):
    r = requests.get(base, params={"search_query": "cat:cs.AI", "start": start, "max_results": 50})
    # parse Atom XML, extract entries
    time.sleep(3)

That script fails in production for three reasons that are not obvious until you have already burned a Sunday on them.

1. The XML response is finicky. arXiv's Atom feed can return an empty <feed> when pagination goes past the result set, a 503 under load, or a malformed Unicode character in an author name that breaks naive XML parsers. A parse failure stops your sweep silently at row 2,400.

2. The rate-limit pushes back mid-sweep. arXiv enforces the three-second guideline inconsistently. During peak hours you can hit a 503 that a simple time.sleep(3) loop will not recover from gracefully. We retry with exponential backoff on 408 / 429 / 5xx, honour Retry-After headers, and surface partial success rather than returning an empty dataset.

3. The page cursor is fragile. If you restart a failed run with different start offsets, arXiv treats it as a fresh query and results shift between pages as new papers are indexed mid-sweep. We log the precise offset for each page so you can audit exactly what was fetched.

We absorb all three failure modes. We also rotate browser fingerprints via curl-cffi so requests look like a browser hitting the feed, not a raw Python requests call — which matters less for arXiv than for more aggressive targets, but the defence is already there.

The Actor ⚙️

The arXiv Papers Scraper is on the Apify Store. You can run it from the Apify Console UI, or call it programmatically:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("DevilScrapes/arxiv-papers-scraper").call(
    run_input={
        "searchQuery": "cat:cs.AI AND ti:transformer",
        "sortBy": "submittedDate",
        "sortOrder": "descending",
        "maxResults": 1000,
        "pageSize": 50,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["arxiv_id"], item["title"])

The key input parameters:

Parameter	Default	Notes
`searchQuery`	`cat:cs.AI`	arXiv query string. Supports `ti:`, `au:`, `cat:`, `abs:` prefixes and Boolean operators.
`sortBy`	`submittedDate`	`relevance`, `lastUpdatedDate`, or `submittedDate`.
`sortOrder`	`descending`	`ascending` or `descending`.
`maxResults`	50	Cap at 5,000 per run. arXiv recommends ≤30,000 per query.
`pageSize`	50	Papers per API call; arXiv caps at 2,000.

The output streams live into the run's dataset — export as JSON, CSV, or Excel from the Apify Console, or fetch via the Apify dataset API.

Use cases

RAG corpus ingestion. Build a domain-specific retrieval pipeline — run cat:cs.CL for 12 months, drop arxiv_id, title, summary, and published into your vector store, and you have a searchable knowledge base in an afternoon. The Actor handles pagination; you handle embeddings.

Literature review seeding. Query ti:scaling laws AND cat:cs.LG to seed a systematic review. Every result comes back with DOI and journal reference where available — enough to drop straight into a citation manager.

Trend monitoring. Schedule a daily run on cat:cs.AI sorted by submittedDate descending with maxResults: 50. Pipe into a Slack webhook and get yesterday's new papers in your inbox every morning.

Research attribution tracking. Run au:lecun weekly, diff arxiv_id against last week's run, and alert on new papers. Useful for tracking lab output or advisor submissions.

Pricing — exact numbers 💰

Pay-per-event. You pay for papers that land in your dataset; nothing for the setup or failed retries.

Event	Price
Actor start (one-off per run)	$0.005
Result (per paper written)	$0.0015

Volume	Cost
100 papers	$0.16
1,000 papers	$1.51
10,000 papers	$15.01
30,000 papers (full category sweep)	$45.01

Apify's $5 free trial credit covers your first ~3,300 papers with no credit card required. For context: building and maintaining the pagination, retry, and parsing layer yourself costs engineering time you could spend on the embedding pipeline.

The part worth knowing: arXiv's rate-limit is real

Most arXiv scrapers ignore the official bulk-access guidance. arXiv asks for a 3-second delay between requests and automated clients to identify themselves in the User-Agent header. We follow both: the Actor sets a User-Agent that identifies it as a DevilScrapes Apify Actor, paces requests at the recommended interval, and backs off further when the server signals overload.

This is not altruism — it is the reason arXiv's API is still open. If you hammer arXiv with an aggressive client, you hurt every researcher who relies on programmatic access. We take the slower throughput and keep the access clean.

Limitations

Metadata only. The Actor uses the Atom feed API. Full-text search over PDF content is not supported — queries operate on metadata fields (title, abstract, author, category). For PDF content extraction, you would need a second-stage pipeline.
No canonical author IDs. arXiv does not assign stable author identifiers in the public API. Author-disambiguation across name collisions ("J. Smith" being three different people) is on you.
Version pinning. arxiv_id includes the version suffix (e.g. 2401.12345v2). If you want only the latest version of each paper, filter on the highest v suffix yourself — the API can return multiple versions of the same paper in a sweep.
Result cap. maxResults is capped at 5,000 per run in the Actor input. For larger sweeps (a full cs.AI category, 30k+ papers), run multiple queries partitioned by date range and merge the datasets.
DOI availability. Most preprints have no DOI until the paper is journal-published. doi is null for the majority of recent papers.

FAQ

Is scraping arXiv legal?
arXiv publishes the Atom API specifically for programmatic access and has explicit bulk-access guidance. The Actor follows that guidance: polite request pacing, identified User-Agent, metadata only, no auth bypass. As always, review arXiv's terms and your use case.

Does arXiv have a proper JSON API?
No. The public programmatic surface is the Atom XML feed at export.arxiv.org/api/query — no official REST/JSON API exists for bulk metadata export. The Actor converts Atom responses to clean JSON rows.

Can I download the actual PDFs?
Not through this Actor. We surface pdf_url for each paper; pulling the PDFs is a separate step. Use the URL as input to a follow-up pipeline.

How do I build a RAG pipeline on top of this?
Run the Actor, export as JSON, load arxiv_id + title + summary into your vector store (ChromaDB, Pinecone, Qdrant — any accept a list of dicts), embed summary, and query semantically.

Try it

The Actor is on the Apify Store: apify.com/DevilScrapes/arxiv-papers-scraper.

Free $5 credit, no credit card. Run cat:cs.AI with maxResults: 100 and you will have a hundred recent AI papers as clean JSON in under two minutes. If you build something interesting on top of the corpus — a paper recommender, a retrieval benchmark, a trend chart — drop it in the comments. The dataset is the interesting part; we just make it easier to get.

Built by Devil Scrapes — Apify Actors with attitude. Pay-per-event, transparent pricing, no junk fields. 😈

Google Ads Transparency Scraper: pull any competitor's ads for $1.20/1K

Devil Scrapes — Sun, 31 May 2026 09:52:50 +0000

Quick answer: The Google Ads Transparency Center is a public registry of every ad Google runs — but it ships no API and no bulk export. To get the data programmatically you scrape it. A Google Ads Transparency scraper sends the same RPC call the website uses and returns every ad creative for an advertiser as structured JSON. The Apify Actor below does it for $0.0012 per ad (~$1.20 per 1,000), with the TLS fingerprinting, proxy rotation, and pagination handled for you.

Google's Ads Transparency Center is one of the most underused datasets in marketing. Launched in 2023 under the EU Digital Services Act and parallel US pressure, it indexes every ad campaign currently running on Search, YouTube, Display, Shopping, Maps, and Play — keyed by advertiser. Google's own counter lists 300,000+ active creatives for a brand like Nike. For your nearest competitor, it's usually 50–500.

The catch: there's no download button. Just an interactive UI that paginates 40 creatives at a time. If you want this as a CSV — for a competitor sweep, a trademark audit, or a RAG corpus — you have to extract it yourself. Here's what that actually takes, and how I shortened it to one API call.

What is the Google Ads Transparency Center? 🔎

The Google Ads Transparency Center is a public, Google-operated registry that shows the ad creatives any verified advertiser is running, the date range each ad was shown, and roughly where. Google built it to comply with ad-disclosure regulation, so the data is public by design — you're reading the same registry a regulator would.

What it gives you per advertiser:

Every ad creative currently or recently live (text, image, video)
The landing domain each ad clicks through to
First-shown / last-shown timestamps and a rough impression count
A deep link to each creative inside the Transparency Center

What it does not give you: a search-by-keyword mode, region-filtered results from the server, or — crucially — an API.

Does the Google Ads Transparency Center have an API?

No. As of 2026 Google publishes no official API or bulk export for the Ads Transparency Center. The only programmatic surface is the internal SearchService/SearchCreatives RPC that the website itself calls. That endpoint is undocumented, returns a positional protobuf-style array (not labeled JSON), and inspects your TLS fingerprint before it answers. Scraping it reliably is the whole job — which is why a hosted Actor exists instead of a three-line snippet.

What the data looks like

Each ad creative comes back as one flat, typed row. Concrete beats abstract, so here's a real one:

{
  "advertiser_id": "AR18378488041124659201",
  "advertiser_name": "Nike Retail BV",
  "creative_id": "CR15771942603307614209",
  "creative_url": "https://adstransparency.google.com/advertiser/AR18378488041124659201/creative/CR15771942603307614209?region=anywhere",
  "landing_domain": "nike.com",
  "format_type": 1,
  "first_shown_ts": 1761145807,
  "last_shown_ts": 1778871417,
  "impressions": 205,
  "preview_image_url": "https://tpc.googlesyndication.com/archive/simgad/12774179880874022668",
  "preview_content_js_url": null,
  "region": "anywhere",
  "scraped_at": "2026-05-15T19:17:59+00:00"
}

Thirteen fields, the same shape every time, validated with Pydantic before it's written. It drops straight into Pandas, BigQuery, or a vector store — no positional-array wrangling on your side.

The naive approach (and why it falls apart)

The first thing every scraper-aware person tries:

Open Chrome DevTools, find the XHR call to SearchCreatives
Replay it with requests.post()
Parse the JSON, paginate, done

It breaks on the first request. Three reasons, and they're the reasons a hosted Actor earns its keep:

1. TLS fingerprinting. Google's endpoint inspects the JA3/JA4 signature of your TLS handshake. Python's stdlib SSL doesn't match any real browser, so the server returns 403 before it even reads your payload. We get around it by impersonating a real Firefox 147 TLS + HTTP/2 fingerprint via curl-cffi — so the handshake looks like a browser, because functionally it is one.

2. Cookie continuity across pagination. The pagination cursor is bound to a session cookie. Rotate IPs naively between pages and the server invalidates your cursor mid-scrape. We thread Apify residential proxies with sticky sessions so each advertiser's pagination keeps one stable exit IP and cookie jar, and we pace requests at ~1/sec to stay polite.

3. A positional, protobuf-flavored response. The reply isn't keyed JSON — it's nested arrays where meaning depends on position. One Google A/B rotation and a naive parser silently emits garbage. We pin the parser against four captured creative shapes (still image, rich video, minimal, malformed) and run live wire-validation to catch contract drift before it reaches your dataset. On 408/429/5xx we retry with exponential backoff and fail loud on partial success rather than handing you a half-empty file.

None of that is glamorous. All of it is the difference between a script that worked once on your laptop and a feed that survives Google's quarterly cipher rotation.

The Actor

I packaged the result as an Apify Actor: Google Ads Transparency Scraper.

Paste a domain in the Apify Console and click Start, or run it programmatically:

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("DevilScrapes/google-ads-transparency").call(
    run_input={
        "searchDomains": ["nike.com", "adidas.com"],
        "maxResults": 5000,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

You search by landing domain (returns every ad pointing at that domain — including ones bought by resellers and affiliates) or by advertiser ID when you already know the exact advertiser. Multiple targets per run, deduplicated automatically.

What you'd actually use this for

Four concrete patterns, not generic "competitive intelligence":

Weekly competitor sweep. Schedule a run on your top 5 competitors, diff this week's creative IDs against last week's, and alert when a new product line launches. Five competitors × ~200 ads each = roughly $1.20/week of data.
Trademark enforcement. Sweep your own domain and you'll see ads other people bought against your brand keyword — resellers, affiliates, competitors. Cross-reference advertiser IDs against your trademark portfolio and flag the unlicensed ones.
Affiliate-fraud detection. Pull every advertiser whose landing_domain doesn't match the advertiser_name. Mismatches are common in crypto, nutra, and supplement verticals: [c for c in creatives if c["landing_domain"] not in c["advertiser_name"].lower()].
AI / RAG ingestion. Feed creative metadata plus image URLs into a vector store for image-grounded competitive analysis.

Pricing — exact numbers 💰

Pay-per-event. You pay for ads you get, nothing for ads you ask for. No data, no charge.

$0.005 per run (covers warm-up + cookie handshake)
$0.0012 per ad written to the dataset

Pull	Cost
100 ads	$0.13
1,000 ads	$1.21
10,000 ads	$12.01
100,000 ads (monthly sweep)	$120.05

Apify's $5 free trial credit covers your first ~4,000 ads with no credit card. For comparison, the nearest SaaS substitutes (Adbeat, SpyFu) start around $249/month for a slice of the same Google data.

The part other scrapers won't tell you

Region filtering doesn't work — and we say so. The region parameter on Google's SearchCreatives RPC is server-ignored. We tested every plausible request-body shape and none of them returned a region-narrowed result set; the browser UI shows a region selector, but the server hands back the same creative set regardless of what you pass.

So we expose region only as a metadata tag — useful for labeling exports by intended market when you run parallel campaigns, useless as a filter. No public Actor offers real region-narrowed scraping, because Google's endpoint doesn't support it. We'd rather under-promise than ship a filter that silently does nothing.

Limitations (the honest list)

No keyword search. You search by advertiser/domain, not by ad copy — Google's RPC exposes no keyword mode.
Video creatives return a JS bundle, not an MP4. You get a preview_content_js_url; rendering the actual frame needs a headless browser and is out of scope for v1.
~12 months of history. Google purges older creatives, so a wider date range just clips to what they retain.
Big brands hit a cap. Google stops paginating past ~1,000 ads per query, so full-history pulls on a Nike-sized advertiser need maxPages raised deliberately.

FAQ

Is scraping the Google Ads Transparency Center legal?
The Center is a public registry Google operates under regulatory pressure. This Actor reads only what the public UI exposes, at ~1 request/second per session, collects only advertiser-level metadata (no personal data), and bypasses no authentication. As always, check your own jurisdiction and use case.

How is this different from the Facebook Ad Library?
Different platform, different endpoint. This covers Google's network (Search, YouTube, Display, Shopping, Maps, Play). For Meta, use a dedicated Facebook Ad Library scraper.

Can I export to Google Sheets or a warehouse?
Yes — export CSV/Excel/JSON from the Apify Console, webhook the dataset on ACTOR.RUN.SUCCEEDED into Make/Zapier/n8n, or pull it via the Apify API.

Why are some preview_image_url values null?
Those are rich/video/animated creatives — Google renders them with JavaScript, so you get a content.js URL instead of a static image.

Try it

The Actor is on the Apify Store: apify.com/DevilScrapes/google-ads-transparency.

Free $5 trial credit, no credit card. Run it on nike.com and you'll have ~1,000 creatives in your dataset in under a minute. Find a use case I missed, or a field you wish it returned? Drop it in the comments — I ship based on what people actually need.

Built by Devil Scrapes — Apify Actors with attitude. Pay-per-event, transparent pricing, no junk fields. 😈