DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Meta's Ad Library 403 is 481 bytes of instructions, not a block

#python #api #webscraping #debugging

Quick answer

Meta's public Ad Library returns HTTP 403 to your first request from any IP, on any proxy tier, with any TLS fingerprint — and that 403 is not a block. Its body is 481 bytes containing a single fetch() call to a per-session verification path. POST that path on the same session, re-GET, and you get 810 KB of real markup. Then a second trap waits at page 2: the GraphQL pagination endpoint answers a datacenter IP with HTTP 200 carrying a rate-limit error envelope, which parses as "no more results" if you only check for the happy path. One of those mistakes costs you a target. The other one ships a green run that scraped one page.

Five probes, three proxy tiers, one wrong conclusion 🎭

Here is what a plain GET of the Ad Library search URL returned during recon:

Tier Profile Result
direct (local IP) chrome131 403
direct (local IP) firefox133 403
Apify residential, US chrome131 403
Apify residential, DE chrome131 403
Datacenter pack, US chrome131 403

Five data points, three tiers, two fingerprints, all identical. That is a textbook "hard IP/fingerprint block, file it as unreachable" evidence set — and every one of those data points is real. It is also completely wrong, and the only thing separating the two readings is whether anyone opened the response body:

<script>
  fetch('/__rd_verify_<opaque>?challenge=3', { method: 'POST' })
  .finally(() => window.location.reload());
</script>
Enter fullscreen mode Exit fullscreen mode

Four hundred and eighty-one bytes. A block page is a page — branding, an explanation, a support ID, usually several KB of it. A sub-1 KB 403 is far more often a machine talking to a machine, and the whole exchange is spelled out right there in the body. The rule we wrote down afterwards: read any 403 under about a kilobyte before you attribute it to the IP.

The unblock is three requests on one session, no browser engine, no captcha solver, no login:

  1. GET the search URL → 403, 481 bytes. Extract the __rd_verify_... path from the inline fetch(...).
  2. POST that path — no body, no special headers → 200, empty, sets cookie rd_challenge.
  3. GET the same URL again → 200, 810,794 bytes, with the search payload embedded as JSON.

The path is opaque and per-session. Hardcoding the one you saw in your terminal is a fix with a shelf life measured in minutes; it has to be read out of each session's own 403.

The 200 OK that means "stop" 🚧

Page 1 is server-rendered HTML. Page 2 is not: it comes from POST /api/graphql/ running AdLibrarySearchPaginationQuery, threading page_info.end_cursor forward. There is no doc_id anywhere in the page HTML — zero matches — so the query id has to come from the site's own JS bundles. (The legacy /ads/library/async/search_ads/ endpoint is a 404 now, on both GET and POST, with or without the lsd token. It's gone.)

Once wired, that endpoint paginated cleanly to 60 pages and 581 unique ads on a single query. But only on a residential-class exit. On a datacenter IP, the very first pagination request comes back like this:

{"errors": [{"message": "Rate limit exceeded", "code": 1675004}]}
Enter fullscreen mode Exit fullscreen mode

Status line: HTTP/2 200. A parser that looks for the connection object and shrugs when it's absent will read that as an empty page, conclude the results ended, mark the run SUCCEEDED and hand the customer 30 ads out of a possible 29,620. That is exactly how version 0.0.1 of this Actor behaved in the cloud, and every dashboard scored it 100% healthy.

So the failure is now explicit, and it distinguishes when it happened:

class MetaPaginationError(RuntimeError):
    """The FIRST attempt to fetch page 2 failed outright."""
    def __init__(self, detail: str, *, ads: list[AdRecord]) -> None:
        self.ads = ads
Enter fullscreen mode Exit fullscreen mode

If pagination fails on its first attempt, pagination never worked — the run fails loudly and says why. If page 37 fails after 36 pages proved the mechanism works, that's a legitimate partial result and the rows still ship. Either way the exception carries the ads already collected, so real data that landed still gets pushed and billed rather than thrown away with the error. And because of all this, the Actor's shipped default requests residential exits — not as a reflex, but because a specific endpoint was measured refusing anything else.

Output

One row per ad, JSON/CSV/Excel:

ad_archive_id, page_id, page_name, is_active, collation_count, collation_id,
publisher_platforms[], link_url, cta_text, cta_type, body_text,
link_description, image_urls[], display_format, start_date_unix,
end_date_unix, ad_library_url, query, country, scraped_at
Enter fullscreen mode Exit fullscreen mode 
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("DevilScrapes/meta-ad-library-scraper").call(
    run_input={
        "queries": ["nike", "adidas"],
        "country": "US",
        "activeStatus": "active",
        "maxResults": 300,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["page_name"], "|", item["cta_text"], "|", item["link_url"])
Enter fullscreen mode Exit fullscreen mode

Search by keyword or by advertiser Page ID — both axes are verified. No Meta developer account, no access token, no App Review. Pricing is $0.20 to start a run plus $0.003 per ad — $3.20 per 1,000 ads landed.

→ Meta Ad Library Scraper on Apify

Built by Devil Scrapes. We replay the per-session challenge, run the proxy tier this target's pagination endpoint actually tolerates, and treat a 200 with an error envelope inside it as the failure it is — never as a quiet end of results.

Top comments (0)