DEV Community

Jamel elhadj hamlaoui
Jamel elhadj hamlaoui

Posted on

Reverse-engineering the Google Ads Transparency Center API

#python #webscraping #api

The Google Ads Transparency Center shows every ad an advertiser runs. There is no public API for it, but the site itself talks to an internal RPC endpoint, and that endpoint answers anonymous requests.

I spent a day mapping it. Posting the field map here because I could not find it documented anywhere.

The endpoint 

POST https://adstransparency.google.com/anji/_/rpc/SearchService/SearchCreatives?authuser=
Content-Type: application/x-www-form-urlencoded

f.req=<url-encoded JSON>
Enter fullscreen mode Exit fullscreen mode

The payload is protobuf-JSON, so every field is a number rather than a name:

{
  "2": 40,
  "3": {
    "4": 3,
    "8": [2250],
    "12": { "1": "nike.com", "2": true },
    "13": { "1": ["AR01614014350098432001"] },
    "14": [5]
  },
  "7": { "1": 1, "2": 0, "3": 2250 }
}
Enter fullscreen mode Exit fullscreen mode
Field Meaning
2 page size (40 works)
3.4 ad format: 1 = IMAGE, 2 = TEXT, 3 = VIDEO
3.8 array of region codes
3.12.1 free-text query: brand name or domain
3.13.1 array of advertiser IDs
3.14 array of platform codes
4 pagination cursor

The response is {"1": [creatives...], "2": nextCursor}. Feed field 2 back as request field 4 to page.

Region codes are just ISO numeric + 2000

This one is tidy. France is ISO 3166-1 numeric 250, and its code is 2250. The United States is 840, and its code is 2840.

region_code = 2000 + iso_3166_1_numeric
Enter fullscreen mode Exit fullscreen mode

So you can accept ordinary country codes from users and convert.

The free-text field is the useful one

Field 3.12.1 is the one worth knowing about. It accepts a brand name or a bare domain:

{"2": 10, "3": {"12": {"1": "nike.com", "2": true}}, "7": {"1": 1, "2": 0, "3": 2250}}
Enter fullscreen mode Exit fullscreen mode

That returns ads for Nike, Inc. without ever touching an advertiser ID. Most tooling in this space makes you go hunt the AR... ID by hand first.

Advertiser lookup, no auth required 

POST /anji/_/rpc/SearchService/SearchSuggestions?authuser=
f.req={"1": "decathlon", "2": 10, "3": 10, "5": {"1": 1}}
Enter fullscreen mode Exit fullscreen mode

Each result is either an advertiser (field 1) or a domain (field 2):

{"1": {"1": "DECATHLON FRANCE", "2": "AR12956355609636110337", "3": "FR",
       "4": {"2": {"1": "6", "2": "6"}}, "5": true}}
Enter fullscreen mode Exit fullscreen mode

Name, advertiser ID, country, an approximate ad-count range, and a verified flag. No token needed.

The endpoint you cannot use

LookupService/GetCreativeById returns per-ad country data and creative variations. It also requires an x-framework-xsrf-token header.

That token is embedded in the page HTML as xsrfToken: '...' — but only when you are signed in to Google. Fetch the same page anonymously and the value is simply absent. An anonymous scraper cannot get one.

Workaround: if you want to know which countries served an ad, search each country separately and record which ones returned it. Same answer, public endpoint only.

Ad copy is hidden in plain sight

This was the surprise. Creatives point at a content.js bundle on displayads-formats.googleusercontent.com. It looks like a 150 KB renderer blob, and my first assumption was that extracting ad text would need a headless browser.

It does not. The ad document is embedded inside that bundle as a JavaScript string full of escaped hex. Un-escape it and you get real HTML back:

import re

HEX = re.compile(r"\\x([0-9a-fA-F]{2})")
UNI = re.compile(r"\\u([0-9a-fA-F]{4})")

def unescape(js: str) -> str:
    out = HEX.sub(lambda m: chr(int(m.group(1), 16)), js)
    out = UNI.sub(lambda m: chr(int(m.group(1), 16)), out)
    return out.replace("\\/", "/")
Enter fullscreen mode Exit fullscreen mode

From the result you can pull the landing page, the display domain, the creative dimensions and the on-creative text.

Two traps I hit:

  1. Shopping ads are assembled by feed vendors (Criteo, Yteo and friends). The vendor host appears in the bundle before the advertiser link, so a naive "first non-Google URL" grabs the wrong company. Prefer a URL whose host matches the creative display domain.
  2. Feed templates leave placeholders behind like [Frais de livraison] and bare price fragments. They are not ad copy.

The bot wall

Google blocks a single IP after roughly 60 to 80 requests and redirects to /sorry/.

The important part: backing off does not clear it. I retried with exponential backoff for 21 seconds and stayed blocked. Only a different exit IP clears it. If you are paginating a large advertiser, rotate proxies mid-run and retry the same cursor, otherwise you lose everything collected so far.

Fill rates, honestly

Enrichment depends entirely on what kind of ads the brand runs. Two real measurements, 120 ads each:

Brand Creative size Display domain Creative text
decathlon.fr (feed/shopping ads) 83% 81% 37%
nike.com (mostly plain images) 20% 12% 20%

Plain image ads ship no data bundle at all. Nothing to extract, for anyone.

Disclosure

I packaged this into an Apify actor: Google Ads Transparency Scraper. Search by brand or domain, filter by country, format and date.

But the field map above is the useful part and it is yours whether you use the actor or not. Happy to answer questions about the format in the comments.

Top comments (0)