DEV Community: Alom Dev

Getting a competitor's Google ad copy out of the Ads Transparency Center

Alom Dev — Mon, 05 Oct 2026 12:52:55 +0000

Google's Ads Transparency Center is one of the most useful free things for
anyone doing PPC. Type in a competitor's domain and you see every ad they've run: Search, YouTube, Shopping, Display,
with dates and regions.

The annoying part is that you can look but you can't really use it. There's no export, and if you try to copy the
text of a Search ad, you can't, because it isn't text. Each ad preview is rendered as an image.

So if you want something like "all the headlines HubSpot ran in Germany last quarter" in a spreadsheet, you're
screenshotting, or you're scraping.

I went the scraping route. These are my notes, plus the tool I ended up with.

What's in there

For each ad you can get:

the advertiser (the legal entity that paid, which is often not the brand name you'd expect)
first and last shown dates, so how long it's been running
which countries it ran in
format (text, image, video) and platform (Search, YouTube, Shopping, Maps, Play)
how many variations of the creative exist

Long-running ads are the interesting ones. If a competitor has been paying for the same ad for 200+ days, it's
probably converting.

The hard parts

The search goes through advertisers, not ads. You don't search ads directly. A domain or brand name resolves to
one or more advertiser IDs first, and big brands often have several (regional entities, agencies). Miss one and you
miss half their ads.

The text is in images. For Search ads the preview is an image of the ad. To get headline, description, display
URL and sitelinks you need OCR. Shopping ads are easier, the product title is in the data. YouTube ads give you the
video ID.

Languages. I assumed the region filter would tell me the language. It doesn't. Filtering an ad set to Germany I
still got ads in Turkish, because advertisers target people, not just places. So the language has to be detected from
the image itself. In my tests across 11 languages (Japanese, Korean, Chinese, Arabic, Russian, Ukrainian, German,
French, Spanish, Portuguese, Italian) the OCR read 54 of 56 real ads. The two misses were a Japanese ad with a big
photo over the text and an odd Maps format.

Rate limits. Google starts refusing an IP after roughly 20 to 40 requests in a row. You need rotating proxies,
and the rotation logic matters more than the scraping code.

The tool

Disclosure: I turned this into a Google Ads Transparency scraper on Apify.
It does the advertiser lookup, the OCR and the proxy rotation, and gives you a table.

Every ad a domain runs, with the actual ad copy:

{
  "searchQueries": ["hubspot.com"],
  "maxResults": 200,
  "extractAdCopy": true,
  "region": "DE"
}

Some other things it can do:

"resultType": "advertisers" gives one row per advertiser with their estimated ad count and format mix, which is a quick way to size up a market before pulling every ad
"includeLandingPage": true follows each ad to its landing page and grabs the title, H1 and main call to action
"politicalAds": true adds spend ranges, impressions and targeting for political ads where Google publishes them
"deltaMode": true on a schedule only returns ads it hasn't seen before, so you get a feed of a competitor's new ads

Price is $2 per 1,000 ads on Apify's free plan, $1 on the bigger plans. Ad copy is an extra $2 to $3 per 1,000
because OCR is the expensive part. Landing pages are extra too, and only charged when the page actually returned
something.

Ways to use it

Pull a competitor's Search ads, sort by days running, and read the top 20 headlines. Those are the ones working.
Run the same domain monthly with deltaMode and see what they're testing.
Compare headlines across countries. Companies often test messaging in one market before rolling it out.

If you'd rather build it yourself, the short version: resolve advertisers first, OCR the Search previews, detect the
language from the image, and spend most of your effort on proxy rotation.

Scraping Bilibili comments: the 3-comment limit was my own fault

Alom Dev — Mon, 05 Oct 2026 12:31:28 +0000

If you've tried to pull comments from Bilibili (B站) without logging in, you've probably hit the same wall I did:
every video gives you 3 comments. Not 20, not "the first page". Three.

I believed this for a while. Guests get the top 3, log in for the rest. It sounded plausible, Chinese platforms are
strict about logged-out access, so I wrote it into my README as a known limitation and moved on.

Then, while working on something else, a request went out without my usual cookies and came back with 20 comments.

What was actually going on

To look like a normal browser, I was sending the visitor cookies Bilibili hands out on the first page load (buvid3
and friends). Most scraping guides tell you to do this, and for search and video details it's fine.

For the comment API, those cookies mark you as a guest with a session, and guests with a session get a preview: 3
top comments, then a prompt to log in. A request with no cookies at all is treated like an anonymous API call and
gets normal pages of 20.

I tested it properly to make sure it wasn't a fluke. Without cookies the paging just kept going: about 1,200 unique
comments across the videos I tried, no login anywhere.

The fix was literally to stop sending something. That stung a bit.

The other things that tripped me up

WBI signing. Most Bilibili endpoints want a w_rid signature built from two keys in the nav response, mixed
with a fixed shuffle table and a timestamp. Skip it and you get error codes instead of data. The keys rotate, so don't
hardcode them.

The 1,000-result search cap. Search stops at about 50 pages no matter what. If you need more videos for a keyword,
sort by publish date and walk backwards in date windows. Each window gets its own 1,000. Slow, but it works.

Danmaku (弹幕) pools are capped. The bullet comments you see flying over the video are a separate XML or protobuf
feed. For popular videos Bilibili only keeps a slice of them. In one test the video had 128,575 danmaku and the pool
held 3,600. If you're doing "danmaku per minute" analysis, keep that in mind, the counts are a sample, not the whole
thing.

Reply threads. The main comment list shows only a few replies under each comment. To get the full thread you page
a second endpoint per root comment. Easy to miss, and it's where the interesting arguments live.

If you just want the data

Disclosure: after all of this I packaged it as a Bilibili scraper on Apify,
so if you don't want to maintain the signing and paging yourself, this does it. No login, no cookies, and the field
names match the most common Bilibili scraper, so switching is easy.

Comments with full reply threads for a few videos:

{
  "mode": "video_comments",
  "videoUrls": ["https://www.bilibili.com/video/BV1LraH6qEr5", "av170001"],
  "includeReplies": true,
  "maxComments": 200,
  "sortComments": "hot"
}

Search plus comments under each result:

{
  "mode": "search",
  "searchQuery": "人工智能",
  "maxResults": 100,
  "includeComments": true,
  "maxComments": 50
}

b23.tv short links work as input too. Pricing is $4 per 1,000 rows on Apify's free plan and goes down to $2 on the
bigger plans. A danmaku profile (the per-minute reaction summary for a video) is an optional extra at $0.02 per video.

If you'd rather build it yourself

Totally doable. The short version of everything above:

Don't send visitor cookies to the comment API.
Sign requests with WBI and refresh the keys from nav regularly.
Use date windows to get past the search cap.
Page reply threads separately.
Treat danmaku counts as a sample on popular videos.

That's most of the pain. Took me longer than I'd like to admit, mostly because of step 1.

Threads search shows logged-out visitors about 20 posts. Here's how I get 60 to 85.

Alom Dev — Mon, 05 Oct 2026 12:31:18 +0000

If you open a Threads search without being logged in, you get one page of results, roughly 20 posts, and that's it.
No "load more", no cursor, nothing. Every no-login Threads scraper I looked at returns about that many per keyword, and
most of them say so in their README.

I build scrapers, and I wanted more than 20, so I spent some time poking at it. A few notes, in case you're doing the
same.

Search: the 20 posts aren't always the same 20

What I noticed first: if you load the same search twice from two different IPs, you don't get identical results.
There's overlap, but the ranking shifts. And there are actually three different pages that answer a keyword query:
the normal search, the tag search (serp_type=tags), and the /tag/<name> page.

So instead of trying to paginate (you can't, logged out), you sample. Load each of the three surfaces from fresh IPs,
merge by post ID, and stop when three loads in a row bring nothing new. In my tests that lands on 60 to 85 unique
posts per keyword. It's not unlimited, and I'd be lying if I said it was, but it's 3 to 4 times what one page gives
you.

If you need it sorted newest first, sort what you collected. That's not the same as Threads' real "Recent" tab, which
is only for logged-in users, and it's worth being honest about that difference if you sell data to someone.

The Replies and Reposts tabs work logged out

My notes said these needed a login. They don't. The profile page loads them through GraphQL queries with a flag that
the logged-out page just doesn't send by default. Send it and you get the account's replies (with the post they were
replying to) and their reposts (with who reposted and when). That's handy for influencer research: what someone says in
replies is often more revealing than their posts.

The "user not found" false alarm

This one cost me a while. Under load, some real accounts came back as "not found". Threads answers an unknown username
by redirecting to the login page. It also redirects a perfectly fine username to the login page when your IP is being
throttled. Same response, different meaning.

The fix I ended up with: when a profile redirects to login, load a profile that definitely exists from the same IP. If
that one is walled too, the IP is the problem, so rotate and retry. Only if the control profile loads fine is the
username really unknown.

Funny side note: the accounts that kept "failing" in my test list (bbcnews, vogue, samsung, theverge) turned out to
genuinely not exist on Threads under those handles. The scraper was right and my test data was wrong.

E-mails in bios

A lot of business accounts put a contact e-mail in their bio. If you search accounts by keyword and open each profile,
you can pull those out, plus phone numbers and bio links. It only works for people who chose to publish one, which is
also why it's fine to use, but it's surprisingly effective for finding contacts in a niche.

The tool

Disclosure: I packaged all of this as a Threads scraper on Apify. Profiles,
posts, replies, the Replies and Reposts tabs, keyword and hashtag search, account search with bio contacts, and a
monitoring mode that only returns new posts on a schedule. No login. It's $2.50 per 1,000 results on Apify's free plan
and $1.50 on the bigger plans.

Example, hashtag plus keyword search:

{
  "searchQueries": ["#buildinpublic", "indie hacker"],
  "maxPosts": 80,
  "requireKeywordMatch": true
}

If you already use another Threads scraper, the main input fields (usernames, searchQueries, maxPosts,
postedAfter) are named the same, so trying it is just swapping the Actor ID.

There's also an optional mode where you paste the cookie of your own logged-in account. Then search uses the real
Recent tab and pages through it like the app does (300 recent posts for "news" in my test, versus 155 without). It's at
your own risk, use a spare account if you try it.

Google Trends in Python after pytrends died

Alom Dev — Mon, 05 Oct 2026 12:10:35 +0000

If you search for this error you'll find a lot of unhappy people:

pytrends.exceptions.TooManyRequestsError: The request failed: Google returned a response with code 429

The usual advice is to add sleeps, then retries, then proxies. None of it really fixes it, because the repo was
archived in April 2025 and nobody is coming to fix it. Google changed how the Trends site loads its data, and the
library's pattern (lots of quick requests from one IP) gets rate limited almost right away now.

I needed Trends data for a project anyway, so I went down the rabbit hole. Here's what I found, in case you're in the
same spot.

Quick disclosure before anything else: I ended up building a hosted scraper for this and I use it below. I'll try to
be fair about the other options, because for some people they're the better choice.

The options, honestly

Google's official API. Google announced an official Trends API in 2025. It's in alpha and you have to apply.
If you get access, use it. Last time I checked it was still invite-only.

Write it yourself. The site talks to internal JSON endpoints: /trends/api/explore gives you widget tokens,
then you call /trends/api/widgetdata/... with each token. I did this, and it works, but expect to deal with:

the first request getting a 429 on purpose before Google sets the NID cookie
tokens that only work for the exact parameters they were issued for
IP blocks. From a laptop you hit them in minutes. Datacenter IPs get blocked in ranges, so you need rotating proxies and some fallback logic
the )]}' prefix on every response, and parameters that change without notice

It took me a few days to get it stable, and it still needs fixing every couple of months.

Use something hosted. That's what the rest of this post does. I use my Google Trends Scraper on Apify
from Python with the official client. It's $1.50 per 1,000 search terms on the free plan and goes down to $0.50 on
the bigger plans. Proxies are included, and Apify's free plan gives you some monthly credit, which is enough to play with.

Setup

pip install "apify-client>=3" pandas
export APIFY_TOKEN=...   # Apify Console > Settings > API & Integrations

This helper runs the scraper and loads the flat CSV it saves (one row per data point, which is what I usually want
for pandas):

import io
import os

import pandas as pd
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

def google_trends(**run_input) -> pd.DataFrame:
    """Run the Google Trends scraper and return one row per data point."""
    run = client.actor("alom/google-trends-scraper").call(run_input=run_input)
    record = client.key_value_store(run.default_key_value_store_id).get_record("RESULTS_FLAT.csv")
    value = record["value"]
    text = value.decode("utf-8") if isinstance(value, (bytes, bytearray)) else value
    return pd.read_csv(io.StringIO(text))

Interest over time

This is the one most people used pytrends for:

df = google_trends(searchTerms=["web scraping"], geo="US", timeRange="today 12-m")

timeline = df[df.section == "timeline"][["date", "value"]]
timeline["date"] = pd.to_datetime(timeline["date"])
print(timeline.tail())

Time ranges are the same strings the website uses: now 1-H, now 7-d, today 3-m, today 5-y, all. For exact
dates use customTimeRange="2024-01-01 2024-12-31".

Comparing terms

Trends values are 0 to 100 inside one request, so two separate requests aren't comparable. Put the terms in one
comparison instead:

df = google_trends(searchTerms=["tea, coffee, matcha"], isMultiple=True, geo="GB", timeRange="today 3-m")

wide = (df[df.section == "timeline"]
        .pivot_table(index="date", columns="term", values="value"))
print(wide.tail())

Google caps this at 5 terms. If you need more, there's a normalizeAcrossQueries option that chains several 5-term
comparisons through one shared anchor term and rescales them onto one scale. Pick your most searched term as the
anchor. With a tiny anchor the rounding gets ugly, and the output flags that with lowPrecision. It costs a bit
extra per comparison because it really is several requests under the hood.

States, cities and US metro areas

pytrends gave you one resolution per call. Here you get states, cities and metro areas (DMA) in the same result:

df = google_trends(searchTerms=["coffee"], geo="US")

metros = (df[df.section == "metro"]
          .sort_values("value", ascending=False)[["label", "code", "value"]])
print(metros.head(10))

The section column is one of timeline, country, subregion, city, metro, relatedQueryTop,
relatedQueryRising.

Related queries

df = google_trends(searchTerms=["web scraping"], geo="US")

rising = df[df.section == "relatedQueryRising"][["label", "formattedValue"]]
print(rising)

One thing that confused me: rising queries are sometimes weird. For "coffee" I once got "pet care tips". I assumed it
was a parsing bug and spent an evening on it. It wasn't. The Trends website shows the exact same list. Google's data
is just like that sometimes.

Trending Now

The newer page at trends.google.com/trending never had a pytrends equivalent. This gets the list per country (or US
state) with search volume:

df = google_trends(mode="trendingNow", trendingGeos=["US", "GB"], trendingHours="24",
                   trendingSort="volume", maxTrendsPerGeo=25)

print(df[["geo", "rank", "title", "searchVolume", "increasePercentage", "categories"]].head(10))

Schedule it hourly with onlyNewSinceLastRun=True and each run only returns trends you haven't seen yet. It's
basically a cheap alert feed.

If you'd rather have JSON

run = client.actor("alom/google-trends-scraper").call(run_input={"searchTerms": ["web scraping"], "geo": "US"})
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["searchTerm"], len(item["interestOverTime_timelineData"]), "points")

One limitation

Related topics come back empty. Google only shows them to signed-in users now, so pytrends and every logged-out
tool lose them too. Related queries still work fine.

So which one?

If you need a couple of charts, just download the CSV from the Trends website, honestly. If you get into the official
API alpha, use that. If you have something scheduled or bulk, or it sits inside a pipeline, paying someone else to
deal with the 429s is cheaper than doing it yourself, at least it was for me.