Pinterest's ad library endpoint was hiding behind one missing curl flag

#python #webscraping #marketing #api

Quick answer

Three earlier attempts at this concluded "Pinterest has no public ad library endpoint" — and all three were wrong for the same reason: they ran curl without --compressed against Pinterest's own route bundle and grepped gzip bytes, which reads exactly like an empty search. The real endpoint, https://api.pinterest.com/ads/v4/ads_repository/ad_library, is named in plain text inside that bundle and answers 200, application/json, 40,983 bytes, with zero headers and zero cookies required. It also repeats roughly a third of its records across pages and puts its pagination cursor at the top level of the response instead of inside data — two more ways to conclude "this doesn't work" without actually being blocked at all.

Why did this look unreachable? 🔍

The two obvious URLs are decoys. https://www.pinterest.com/ad-library/ returns 200 with about 1 MB of HTML — but it's a catch-all app shell: ad_library, adLibrary, and resourceResponses each occur zero times in that payload, and its __PWS_INITIAL_PROPS__ blob is identical to what an unrelated Pinterest path returns. A 200 status code told nobody anything here. https://ads.pinterest.com/adlibrary/ is more honest about it — a flat 404.

The real endpoint is sitting in plain text inside Pinterest's own compiled JS:

curl -s --compressed https://s.pinimg.com/webapp/sterling/ads-repository-cf78c5da96a88f88.mjs \
  | grep -oE '"/ads/v4/ads_repository/[a-z_]+"'

The --compressed flag is load-bearing. Drop it and you grep raw gzip bytes, get nothing back, and the result reads exactly like "there are no ad-library endpoints referenced in this bundle." That single missing flag is the entire reason this looked like dead ground for three prior passes — the endpoint was never hidden, it was just never decompressed.

What does the real endpoint actually require? 🎯

GET https://api.pinterest.com/ads/v4/ads_repository/ad_library?start_date=2026-08-01&country=DE comes back 200, application/json, 40,983 bytes, byte-identical on both TLS profiles we tested — and it needs neither a header nor a cookie to answer. start_date and country are both required; omit either and you get an error before any real data comes back. Records land 25 to a page under data.pin_ids_with_metadata.

Pagination is the second gotcha: the bookmark cursor for the next page lives at the top level of the response, not inside data. Reasonable code that walks straight into data looking for a next-page token won't find one and will assume page 1 is the whole answer — an easy miss dressed up as "page 2 doesn't exist."

Why can't you trust the raw page count? 🧮

Because Pinterest repeats itself. We measured this twice: 20 pages of raw fetches returned 350 unique pin_id values out of 500 raw records, and our own cloud run at 4 pages returned 90 unique ads out of 100 raw rows. Roughly a third of every batch is a repeat of something you already have. A row count of pages × 25 is never the true count, and any downstream billing or reporting built on raw page math will overstate its own dataset by about 30%. We dedupe incrementally by pin_id as pages come in, so what lands in your dataset is always the true unique count.

What happens when you paginate too fast? ⏱️

The rate limit is real, and it's fast: 20 pages in about 5 seconds and page 21 comes back 429 {"code": 8}. But it's a window, not a hard cap — after roughly 45 seconds, eight further pages fetched at 5-second intervals all returned clean 200s, and the bookmark cursor was still valid from where it left off. Treating that 429 as a dead end throws away a cursor that's still good; we pause on the limit and resume from the exact same bookmark instead of failing the run or starting over from page 1.

One more thing worth saying plainly before you pay for a run: coverage is EU27, Brazil, and Turkey — this is the DSA transparency registry, not a global or US ad library. If you're expecting US ad data, this isn't that dataset.

What we handle for you

🛡️ We rotate browser fingerprints (curl-cffi impersonation, Chrome and Firefox) as a hedge, even though this endpoint's limiter is currently request-based rather than fingerprint-based.
🧱 We back off on the rate-limit window — pause roughly 45 seconds, then resume from the same bookmark, never losing your place or failing the run over a temporary 429.
🧊 We deduplicate by pin_id incrementally so your dataset count is always the true unique total, never pages × page size.
🔁 We retry with exponential backoff on server errors and network hiccups, honouring Retry-After, up to 5 attempts per page.
💰 You pay only for results that land. No data → no charge beyond the small actor-start fee.

How much does it cost?

$0.20 to start a run, plus $3.00 per 1,000 unique ads landed.

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("DevilScrapes/pinterest-ad-library-scraper").call(
    run_input={
        "start_date": "2026-08-01",
        "country": "DE",
        "maxResults": 200,
        "maxPages": 10,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["pin_id"], "|", item["title"], "|", item["start_date"], "-", item["end_date"])

→ Pinterest Ad Library Scraper on Apify

Built by Devil Scrapes. We decompress the bundle, read the cursor where it actually lives, dedupe the repeats, and pace against the window instead of quitting at the first 429.

DEV Community