DEV Community: Alom Dev

How to scrape Threads posts without logging in (Python and JavaScript)

Alom Dev — Thu, 15 Oct 2026 11:54:20 +0000

Disclosure first: I built the scraper this post uses (Threads Scraper on Apify),
so take my enthusiasm with a grain of salt. The code below is real and I ran every snippet before posting it. The
outputs are what I got on October 15.

Why not just use the official API?

Meta does have a Threads API, but it needs a Meta developer app and an access token, and it's built mostly around your
own account. If you want "the last 50 posts of some public account" or "what people posted about espresso this week",
you end up scraping the website. A logged-out visitor can see quite a lot on threads.com: profiles, posts, the replies
under a post, a keyword search. That's what the scraper collects, without any account, so there's nothing to get banned.

You need an Apify account (the free plan includes $5 of usage a month, which at $2.50 per 1,000 results is about 2,000
posts) and your API token from Settings → API & Integrations.

Python: an account's posts with view counts

pip install apify-client
export APIFY_TOKEN=your_token

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("alom/threads-scraper").call(run_input={
    "usernames": ["mosseri"],
    "maxPosts": 20,
    "includeViews": True,
})

for item in client.dataset(run.default_dataset_id).iterate_items():
    if item["type"] == "post":
        print(item["likeCount"], item["viewCount"], item["text"][:60])

Output (likes, views, text):

1018 320506 Big day for the Muse team with the introduction of Muse Wond
372 42105 As is often the case with features like this one, this post
216 31669 Once iOS27 reaches a majority of our users though I think we
156 14703 This is rad.
340 19718 Three DM settings worth knowing about. Let me know if there

One gotcha if you copy older tutorials: apify-client 3.x returns a Run object, so it's run.default_dataset_id, not
run["defaultDatasetId"]. The dictionary version throws 'Run' object is not subscriptable. I hit that myself while
writing this.

About includeViews: Threads shows a view count ("320K views") on the post page, not in the feed, so the scraper has
to open each post. That adds roughly 1.5 seconds per post, which is why it's off by default. For a post you pass by URL
you get the view count anyway.

The first row of the dataset is the profile itself (type: "profile") with followers, bio, bio links and verification.
Set "mode": "profile" if that's all you want.

Python: keyword search, newest first, with filters

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("alom/threads-scraper").call(run_input={
    "searchQueries": ["espresso", "#coffee"],
    "searchSort": "recent",
    "maxPosts": 50,
    "minLikes": 5,
    "languages": ["en"],
})

rows = list(client.dataset(run.default_dataset_id).iterate_items())
print(len(rows), "posts")
for row in rows[:5]:
    print(row["date"], row["username"], row["likeCount"], row["text"][:60])

56 posts
2026-10-15T00:41:07.000Z riim__79 33 Espresso ☕️.
2026-10-14T22:55:40.000Z andi.vdb 10 Depresso in need of espresso
2026-10-14T17:25:44.000Z knowyourworthfl 295 Yeah a local coffee shop said they could not make me a doubl
2026-10-09T20:59:18.000Z j_x84 205 Espresso Alfredo❣️🪄.
2026-09-30T03:54:21.000Z alanchunggg 13 Espresso ☕️

A few honest notes about search:

Logged out, Threads shows one page of about 20 results per search. The scraper samples it several times and merges, which usually gets 60 to 85 posts per keyword. It is not every post ever written about espresso.
searchSort: "recent" sorts what it found newest first. It isn't Threads' real "Recent" tab, which needs a login.
languages uses the language Threads reports when it has one, otherwise a guess from the text. A post like "Espresso ☕️." is too short to tell, and those are kept rather than dropped. That's why one slipped in above.
minLikes and minReplies are applied before the results are saved, and you only pay for the rows you get.

JavaScript: the replies under a post

npm install apify-client

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('alom/threads-scraper').call({
    postUrls: ['https://www.threads.com/@mosseri/post/DdUI1VfG_42'],
    maxReplies: 100,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const replies = items.filter((item) => item.type === 'reply');
console.log(`${replies.length} replies`);
for (const r of replies.slice(0, 5)) console.log(r.likeCount, r.username, r.text.slice(0, 60));

That printed 100 replies, sorted the way Threads sorts them ("top"). Set repliesSort: "recent" for newest first.
Each reply has parentPostId, so you can rebuild the reply chains, and isLikedByAuthor tells you which replies the
original poster liked. If replies are all you need, there's a separate
Threads Replies Scraper with a simpler input. Logged out, Threads
shows roughly the first 1,000 replies of a post, so that's the ceiling.

Using it from Claude, ChatGPT or Cursor (MCP)

You don't have to write code at all if an AI assistant should do the asking. Apify runs an MCP server, and you can
limit it to one Actor with the tools parameter:

https://mcp.apify.com/?tools=alom/threads-scraper

Claude (claude.ai or Desktop): Settings → Connectors → Add custom connector, paste the URL, then sign in to Apify when it asks.
Claude Code: claude mcp add --transport http threads "https://mcp.apify.com/?tools=alom/threads-scraper", then run /mcp to sign in.
Cursor and other clients with an mcpServers config:

{
  "mcpServers": {
    "threads": {
      "url": "https://mcp.apify.com/?tools=alom/threads-scraper"
    }
  }
}

The agent gets a tool called alom--threads-scraper with the same inputs as above. Then you just ask, for example:
"Get the last 20 posts of @mosseri with view counts and tell me which topics got the most views." Two tips: keep
maxPosts small (10 to 30) so the answer fits the model's context, and use minLikes and languages so it isn't
reading noise. Every call is a normal run on your Apify account, at the normal price.

Cost and limits

$2.50 per 1,000 results on Apify's free plan, $1.50 on the bigger plans, plus half a cent per run. Proxies and compute
are included in that. What it can't do logged out: follower and following lists, and post counts. A deleted post or an
unknown username fails only that input, not the whole run.

If something doesn't work for your case, open an issue on the Actor page. I read all of them.

I checked how many Threads accounts list an email in their bio. It's about 4%.

Alom Dev — Mon, 12 Oct 2026 10:44:04 +0000

I built a small tool that finds Threads accounts by keyword: type "coffee shop" or "marketing agency" and get a table
of accounts with followers, bio and links. My plan was to call it an email finder, because that's what people ask
for. Then I looked at the results and renamed it.

The number

Threads lets people write whatever they want in their bio, and some business accounts put a contact email there.
"Some" turned out to be fewer than I expected:

"marketing" and "coffee shop" together returned 59 accounts. 2 had an email in the bio and 4 had a phone number.
Across 111 different accounts from several keywords, about 4% had an email.
In a run that kept only accounts with an email, it opened 277 profiles to find 5.

So between 2% and 4%, depending on the niche. Agencies and local businesses list one more often than personal
accounts. If someone sells you "emails of every Threads account in your niche", that's not what's on the page.

What is on the page, and is more useful

For every account you do get:

Followers and verification. Enough to sort a list and decide who is worth a closer look.
The bio text. People describe what they do in one line, which is the fastest filter there is.
Bio links. Most business accounts link their website, a booking page or a link-in-bio page. That is where the contact form and the real email usually are.

In practice the bio link is the lead, not the email. A list of 300 accounts sorted by followers, with their websites,
is a morning of work done in a few minutes.

How the search behaves

A few things I learned about Threads' account search without logging in:

It matches the keyword against usernames, display names and bios.
It returns up to about 100 accounts per keyword. For a bigger list, use several specific keywords ("coffee shop berlin", "coffee roaster", "cafe owner") instead of one broad one.
The follower count and bio links are not in the search result. Each account has to be opened once to get them.

A simple way to use it

Write down 5 to 10 keywords your customers would use to describe themselves.
Collect the accounts for all of them. Duplicates across keywords are removed.
Sort by followers, read the bios, delete what doesn't fit.
For the ones that fit, open the bio link and use the contact route they offer there.

Only contact people through details they chose to publish, and check the rules for cold email where they live. In
the EU that matters.

The tool

Disclosure: this is my Threads Account Finder on Apify. No login is
involved, it reads what a logged-out visitor sees. It costs $2.50 per 1,000 accounts on Apify's free plan and $1.50
on the bigger plans, plus $0.01 for each row that has an email, because finding one means opening about 50 profiles.
On the free plan, 500 accounts with 20 emails come to about $1.45.

{ "searchQueries": ["coffee shop", "specialty coffee"], "maxPosts": 100 }

There is a ready-made example
you can run as it is. If you also need the posts, replies or a keyword search, the
Threads Scraper does those with the same input names.

Google Hotels showed me two different prices 12 seconds apart

Alom Dev — Thu, 08 Oct 2026 09:22:23 +0000

I built a hotel price scraper on top of Google Hotels. It collects the price every booking site shows for a hotel
(Booking.com, Expedia, Agoda, the hotel's own site and so on) for each night in a date range. The obvious use is
monitoring: run it every day, alert when a price changes.

Then my own monitoring test kept failing, and the reason is worth knowing if you work with hotel prices at all.

The test

The test is simple. Run the monitor twice in a row. The second run should return almost nothing, because prices
don't change in a few seconds.

The second run came back with 36 of 69 rows "changed". I assumed a bug in my comparison code. It wasn't. When I
looked at the rows, every one of them really had a different price. Same hotel (a Novotel in Paris), same night,
same booking site, 12 seconds apart: 347 the first time, 401 the second. Another site went from 270 to 315.

Google Hotels doesn't show one price. It shows a price for that request: which connection it came from, which
session, possibly which test bucket. Ask again from somewhere else and you can get a noticeably different number.

What this means in practice

If you monitor prices, some of the "changes" you see are this variation, not a real price move. A few ways to deal
with it:

Compare against a threshold, not equality. A 2% wobble is noise; a 15% jump on one site while the others stay put is worth a look.
Look at several sites together. If only one site moved, it's probably the request. If the hotel's own site and two OTAs moved the same way, it's real.
Sample more than once before you act on a single reading, especially for rate parity complaints.

The first date that had no prices

Another surprise: when you price one hotel over a range of nights, the very first night sometimes came back with no
prices at all, while the next night had 35 booking sites. The first lookup goes through a different request than the
rest of the range, and that request occasionally answers without its price section.

The scraper used to report that night as "no offers", which is wrong and also invisible, because the run still said
everything was fine. Now it asks the prices call once more before deciding a date really has no offers. Small thing,
but a monitor that silently drops a night is worse than one that fails loudly.

The tool

Disclosure: this is my Google Hotels Scraper on Apify. It searches
hotels in a city with prices for your dates, or tracks specific hotels across a date range with every booking site's
price, room rates, free-cancellation deadlines and guest reviews. A price-calendar row costs $1.00 per 1,000 on
Apify's free plan and $0.60 on the bigger plans.

{
  "mode": "hotelPrices",
  "hotels": ["Novotel Paris Les Halles"],
  "checkInDate": "+14",
  "lastCheckInDate": "+43",
  "providers": ["booking.com", "expedia", "official"]
}

I also just published a Google Flights scraper built on the same
Google Travel data, if you need flights next to hotels.

Scraping YouTube comments without the API quota (and the bug where 300 comments meant one thread)

Alom Dev — Thu, 08 Oct 2026 09:16:17 +0000

The YouTube Data API gives you 10,000 quota units a day by default. Reading comments costs 1 unit per page of up to
100 comments, which sounds generous until you need replies too, or more than a handful of videos, or a second
project running on the same key. People who analyse comments at any scale hit that wall pretty quickly.

I built a comments scraper that reads what youtube.com shows a logged-out visitor instead, so there is no key and no
quota. A few things I learned on the way, in case you are building something similar.

"300 comments" should mean 300 comments

My first version had one limit, maxComments, and replies counted toward it. Seemed reasonable. Then a test asked
for 300 comments with replies on a popular music video and got back 1 comment and its 299 replies.

Of course it did. Replies are delivered right after their parent, and the most liked comment on a big video has
thousands of them. The limit was used up by one thread.

Nobody who types "300" means that. The fix was two limits:

maxComments counts top-level comments only
maxRepliesPerComment caps the replies under each one (default 5)

The second one matters for cost too. If replies were unlimited, 200 comments could turn into 4,000 rows and a bill
nobody expected. With the default, 100 comments return at most 600 rows, and the input form says so.

Top comments vs newest

YouTube has two orders, Top and Newest. Top is what you want for "what do people think". Newest is what you want
for monitoring, for example comments on a launch video since yesterday. A date filter on top of Newest ("only the
last 30 days") lets the scraper stop early instead of paging through years of comments.

The timeouts that weren't network problems

This one took me a while. Runs with a short time limit kept ending as TIMED-OUT instead of stopping cleanly and
saving what they had. The code had a deadline check that should stop work 25 seconds before the limit. It didn't
fire in time.

The cause: on Apify, 256 MB of memory also means about 1/16 of a CPU core. Parsing a lot of YouTube pages at once
kept that tiny CPU so busy that the timer meant to stop the run fired late, sometimes by 20+ seconds. Raising the
minimum to 512 MB (twice the CPU) fixed it completely, and the extra cost is still a fraction of a cent per thousand
rows. If your scraper misses its own deadlines, check the CPU before you blame the network.

The tool

Disclosure: this is the YouTube Comments Scraper I published on
Apify. Paste video or Shorts URLs, pick Top or Newest, optionally replies and a date window. It costs $1.00 per
1,000 comments on Apify's free plan and $0.45 on the bigger plans.

{
  "startUrls": [{ "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ" }],
  "maxComments": 300,
  "sortCommentsBy": "TOP_COMMENTS",
  "includeReplies": true,
  "maxRepliesPerComment": 5
}

There are also two siblings built on the same code: a channel scraper
that exports every video of a channel with its stats, and a Shorts scraper
for Shorts by channel or keyword.

If you'd rather stay on the official API, that's a perfectly good choice for small jobs. Just budget your quota for
replies, they add up faster than you think.

Getting Google Flights prices into a spreadsheet

Alom Dev — Thu, 08 Oct 2026 09:10:16 +0000

Google Flights is the best flight search there is, and it has no public API. Google shut down the old QPX Express
API years ago, so if you want flight prices in a spreadsheet or a script, you either pay for a travel data provider
or you read what the Google Flights page shows.

I built a scraper for the second option, using the same Google Travel data I already used for a hotel price tool.
Here's what it does and a few things I ran into.

Three things people actually want

When I looked at what people ask for, it boiled down to three jobs:

Every flight for a route and date, with price, airline, times, stops and layovers. Basically the results page as a table.
The cheapest day to fly. Google's date grid shows a price for each departure date. That's the most useful view for anyone flexible on dates, and it's one request for about two months of prices.
Alerts. Run it every day and only hear about flights whose price changed.

Things I didn't expect

City names and multi-airport cities. People type "New York", "NYC" or "JFK,EWR", not airport codes. Google's own
search rejects some of those codes, so the scraper resolves names and city codes to airports first.

Prices were the same everywhere I tested. I expected the country and IP to change fares, like hotel prices do.
Across US, Ukrainian, German and Indian connections the prices came back identical; only the currency and the
language of names changed. That may not hold for every airline, but it held in all my tests.

Google refuses a fair share of requests. Roughly a quarter to a third of connections get refused for flight
searches. The fix that worked was boring: switch to a new connection right away, and after a few refusals fall back
to the normal results page, which has the same data. In the end no route failed in my test runs.

Layovers are where the detail is. The interesting fields aren't the price, they're the layovers: which airport,
how long, and whether you have to change airports. A 50-minute connection with an airport change is a very
different trip from a 2-hour one in the same terminal.

Using it

Disclosure: this is my Google Flights Scraper on Apify. You can run
it from the Apify website without code and download CSV or Excel, or call it from Python.

Every flight from New York to London, cheapest first:

{ "origin": "JFK,EWR", "destination": "LHR", "departureDate": "+45", "sortBy": "price", "maxResults": 50 }

Cheapest day to fly over the next two months, round trip with a 7-day stay:

{ "mode": "calendar", "origin": "BER", "destination": "BCN", "departureDate": "+7", "returnDate": "+14", "calendarDays": 60 }

Price alerts on a schedule:

{ "origin": "WAW", "destination": "LIS", "departureDate": "+40", "returnDate": "+47", "onlyNewSinceLastRun": true, "monitorName": "waw-lis" }

It costs $0.50 per 1,000 flights on Apify's free plan ($0.25 on bigger plans) and $0.20 per 1,000 calendar dates. A
whole route usually comes to a fraction of a cent.

If you need hotels for the same trip, there's a Google Hotels scraper
built on the same data.

Getting a competitor's Google ad copy out of the Ads Transparency Center

Alom Dev — Mon, 05 Oct 2026 12:52:55 +0000

Google's Ads Transparency Center is one of the most useful free things for
anyone doing PPC. Type in a competitor's domain and you see every ad they've run: Search, YouTube, Shopping, Display,
with dates and regions.

The annoying part is that you can look but you can't really use it. There's no export, and if you try to copy the
text of a Search ad, you can't, because it isn't text. Each ad preview is rendered as an image.

So if you want something like "all the headlines HubSpot ran in Germany last quarter" in a spreadsheet, you're
screenshotting, or you're scraping.

I went the scraping route. These are my notes, plus the tool I ended up with.

What's in there

For each ad you can get:

the advertiser (the legal entity that paid, which is often not the brand name you'd expect)
first and last shown dates, so how long it's been running
which countries it ran in
format (text, image, video) and platform (Search, YouTube, Shopping, Maps, Play)
how many variations of the creative exist

Long-running ads are the interesting ones. If a competitor has been paying for the same ad for 200+ days, it's
probably converting.

The hard parts

The search goes through advertisers, not ads. You don't search ads directly. A domain or brand name resolves to
one or more advertiser IDs first, and big brands often have several (regional entities, agencies). Miss one and you
miss half their ads.

The text is in images. For Search ads the preview is an image of the ad. To get headline, description, display
URL and sitelinks you need OCR. Shopping ads are easier, the product title is in the data. YouTube ads give you the
video ID.

Languages. I assumed the region filter would tell me the language. It doesn't. Filtering an ad set to Germany I
still got ads in Turkish, because advertisers target people, not just places. So the language has to be detected from
the image itself. In my tests across 11 languages (Japanese, Korean, Chinese, Arabic, Russian, Ukrainian, German,
French, Spanish, Portuguese, Italian) the OCR read 54 of 56 real ads. The two misses were a Japanese ad with a big
photo over the text and an odd Maps format.

Rate limits. Google starts refusing an IP after roughly 20 to 40 requests in a row. You need rotating proxies,
and the rotation logic matters more than the scraping code.

The tool

Disclosure: I turned this into a Google Ads Transparency scraper on Apify.
It does the advertiser lookup, the OCR and the proxy rotation, and gives you a table.

Every ad a domain runs, with the actual ad copy:

{
  "searchQueries": ["hubspot.com"],
  "maxResults": 200,
  "extractAdCopy": true,
  "region": "DE"
}

Some other things it can do:

"resultType": "advertisers" gives one row per advertiser with their estimated ad count and format mix, which is a quick way to size up a market before pulling every ad
"includeLandingPage": true follows each ad to its landing page and grabs the title, H1 and main call to action
"politicalAds": true adds spend ranges, impressions and targeting for political ads where Google publishes them
"deltaMode": true on a schedule only returns ads it hasn't seen before, so you get a feed of a competitor's new ads

Price is $2 per 1,000 ads on Apify's free plan, $1 on the bigger plans. Ad copy is an extra $2 to $3 per 1,000
because OCR is the expensive part. Landing pages are extra too, and only charged when the page actually returned
something.

Ways to use it

Pull a competitor's Search ads, sort by days running, and read the top 20 headlines. Those are the ones working.
Run the same domain monthly with deltaMode and see what they're testing.
Compare headlines across countries. Companies often test messaging in one market before rolling it out.

If you'd rather build it yourself, the short version: resolve advertisers first, OCR the Search previews, detect the
language from the image, and spend most of your effort on proxy rotation.

Scraping Bilibili comments: the 3-comment limit was my own fault

Alom Dev — Mon, 05 Oct 2026 12:31:28 +0000

If you've tried to pull comments from Bilibili (B站) without logging in, you've probably hit the same wall I did:
every video gives you 3 comments. Not 20, not "the first page". Three.

I believed this for a while. Guests get the top 3, log in for the rest. It sounded plausible, Chinese platforms are
strict about logged-out access, so I wrote it into my README as a known limitation and moved on.

Then, while working on something else, a request went out without my usual cookies and came back with 20 comments.

What was actually going on

To look like a normal browser, I was sending the visitor cookies Bilibili hands out on the first page load (buvid3
and friends). Most scraping guides tell you to do this, and for search and video details it's fine.

For the comment API, those cookies mark you as a guest with a session, and guests with a session get a preview: 3
top comments, then a prompt to log in. A request with no cookies at all is treated like an anonymous API call and
gets normal pages of 20.

I tested it properly to make sure it wasn't a fluke. Without cookies the paging just kept going: about 1,200 unique
comments across the videos I tried, no login anywhere.

The fix was literally to stop sending something. That stung a bit.

The other things that tripped me up

WBI signing. Most Bilibili endpoints want a w_rid signature built from two keys in the nav response, mixed
with a fixed shuffle table and a timestamp. Skip it and you get error codes instead of data. The keys rotate, so don't
hardcode them.

The 1,000-result search cap. Search stops at about 50 pages no matter what. If you need more videos for a keyword,
sort by publish date and walk backwards in date windows. Each window gets its own 1,000. Slow, but it works.

Danmaku (弹幕) pools are capped. The bullet comments you see flying over the video are a separate XML or protobuf
feed. For popular videos Bilibili only keeps a slice of them. In one test the video had 128,575 danmaku and the pool
held 3,600. If you're doing "danmaku per minute" analysis, keep that in mind, the counts are a sample, not the whole
thing.

Reply threads. The main comment list shows only a few replies under each comment. To get the full thread you page
a second endpoint per root comment. Easy to miss, and it's where the interesting arguments live.

If you just want the data

Disclosure: after all of this I packaged it as a Bilibili scraper on Apify,
so if you don't want to maintain the signing and paging yourself, this does it. No login, no cookies, and the field
names match the most common Bilibili scraper, so switching is easy.

Comments with full reply threads for a few videos:

{
  "mode": "video_comments",
  "videoUrls": ["https://www.bilibili.com/video/BV1LraH6qEr5", "av170001"],
  "includeReplies": true,
  "maxComments": 200,
  "sortComments": "hot"
}

Search plus comments under each result:

{
  "mode": "search",
  "searchQuery": "人工智能",
  "maxResults": 100,
  "includeComments": true,
  "maxComments": 50
}

b23.tv short links work as input too. Pricing is $4 per 1,000 rows on Apify's free plan and goes down to $2 on the
bigger plans. A danmaku profile (the per-minute reaction summary for a video) is an optional extra at $0.02 per video.

If you'd rather build it yourself

Totally doable. The short version of everything above:

Don't send visitor cookies to the comment API.
Sign requests with WBI and refresh the keys from nav regularly.
Use date windows to get past the search cap.
Page reply threads separately.
Treat danmaku counts as a sample on popular videos.

That's most of the pain. Took me longer than I'd like to admit, mostly because of step 1.

Threads search shows logged-out visitors about 20 posts. Here's how I get 60 to 85.

Alom Dev — Mon, 05 Oct 2026 12:31:18 +0000

If you open a Threads search without being logged in, you get one page of results, roughly 20 posts, and that's it.
No "load more", no cursor, nothing. Every no-login Threads scraper I looked at returns about that many per keyword, and
most of them say so in their README.

I build scrapers, and I wanted more than 20, so I spent some time poking at it. A few notes, in case you're doing the
same.

Search: the 20 posts aren't always the same 20

What I noticed first: if you load the same search twice from two different IPs, you don't get identical results.
There's overlap, but the ranking shifts. And there are actually three different pages that answer a keyword query:
the normal search, the tag search (serp_type=tags), and the /tag/<name> page.

So instead of trying to paginate (you can't, logged out), you sample. Load each of the three surfaces from fresh IPs,
merge by post ID, and stop when three loads in a row bring nothing new. In my tests that lands on 60 to 85 unique
posts per keyword. It's not unlimited, and I'd be lying if I said it was, but it's 3 to 4 times what one page gives
you.

If you need it sorted newest first, sort what you collected. That's not the same as Threads' real "Recent" tab, which
is only for logged-in users, and it's worth being honest about that difference if you sell data to someone.

The Replies and Reposts tabs work logged out

My notes said these needed a login. They don't. The profile page loads them through GraphQL queries with a flag that
the logged-out page just doesn't send by default. Send it and you get the account's replies (with the post they were
replying to) and their reposts (with who reposted and when). That's handy for influencer research: what someone says in
replies is often more revealing than their posts.

The "user not found" false alarm

This one cost me a while. Under load, some real accounts came back as "not found". Threads answers an unknown username
by redirecting to the login page. It also redirects a perfectly fine username to the login page when your IP is being
throttled. Same response, different meaning.

The fix I ended up with: when a profile redirects to login, load a profile that definitely exists from the same IP. If
that one is walled too, the IP is the problem, so rotate and retry. Only if the control profile loads fine is the
username really unknown.

Funny side note: the accounts that kept "failing" in my test list (bbcnews, vogue, samsung, theverge) turned out to
genuinely not exist on Threads under those handles. The scraper was right and my test data was wrong.

E-mails in bios

A lot of business accounts put a contact e-mail in their bio. If you search accounts by keyword and open each profile,
you can pull those out, plus phone numbers and bio links. It only works for people who chose to publish one, which is
also why it's fine to use, but it's surprisingly effective for finding contacts in a niche.

The tool

Disclosure: I packaged all of this as a Threads scraper on Apify. Profiles,
posts, replies, the Replies and Reposts tabs, keyword and hashtag search, account search with bio contacts, and a
monitoring mode that only returns new posts on a schedule. No login. It's $2.50 per 1,000 results on Apify's free plan
and $1.50 on the bigger plans.

Example, hashtag plus keyword search:

{
  "searchQueries": ["#buildinpublic", "indie hacker"],
  "maxPosts": 80,
  "requireKeywordMatch": true
}

If you already use another Threads scraper, the main input fields (usernames, searchQueries, maxPosts,
postedAfter) are named the same, so trying it is just swapping the Actor ID.

There's also an optional mode where you paste the cookie of your own logged-in account. Then search uses the real
Recent tab and pages through it like the app does (300 recent posts for "news" in my test, versus 155 without). It's at
your own risk, use a spare account if you try it.

Google Trends in Python after pytrends died

Alom Dev — Mon, 05 Oct 2026 12:10:35 +0000

If you search for this error you'll find a lot of unhappy people:

pytrends.exceptions.TooManyRequestsError: The request failed: Google returned a response with code 429

The usual advice is to add sleeps, then retries, then proxies. None of it really fixes it, because the repo was
archived in April 2025 and nobody is coming to fix it. Google changed how the Trends site loads its data, and the
library's pattern (lots of quick requests from one IP) gets rate limited almost right away now.

I needed Trends data for a project anyway, so I went down the rabbit hole. Here's what I found, in case you're in the
same spot.

Quick disclosure before anything else: I ended up building a hosted scraper for this and I use it below. I'll try to
be fair about the other options, because for some people they're the better choice.

The options, honestly

Google's official API. Google announced an official Trends API in 2025. It's in alpha and you have to apply.
If you get access, use it. Last time I checked it was still invite-only.

Write it yourself. The site talks to internal JSON endpoints: /trends/api/explore gives you widget tokens,
then you call /trends/api/widgetdata/... with each token. I did this, and it works, but expect to deal with:

the first request getting a 429 on purpose before Google sets the NID cookie
tokens that only work for the exact parameters they were issued for
IP blocks. From a laptop you hit them in minutes. Datacenter IPs get blocked in ranges, so you need rotating proxies and some fallback logic
the )]}' prefix on every response, and parameters that change without notice

It took me a few days to get it stable, and it still needs fixing every couple of months.

Use something hosted. That's what the rest of this post does. I use my Google Trends Scraper on Apify
from Python with the official client. It's $1.50 per 1,000 search terms on the free plan and goes down to $0.50 on
the bigger plans. Proxies are included, and Apify's free plan gives you some monthly credit, which is enough to play with.

Setup

pip install "apify-client>=3" pandas
export APIFY_TOKEN=...   # Apify Console > Settings > API & Integrations

This helper runs the scraper and loads the flat CSV it saves (one row per data point, which is what I usually want
for pandas):

import io
import os

import pandas as pd
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

def google_trends(**run_input) -> pd.DataFrame:
    """Run the Google Trends scraper and return one row per data point."""
    run = client.actor("alom/google-trends-scraper").call(run_input=run_input)
    record = client.key_value_store(run.default_key_value_store_id).get_record("RESULTS_FLAT.csv")
    value = record["value"]
    text = value.decode("utf-8") if isinstance(value, (bytes, bytearray)) else value
    return pd.read_csv(io.StringIO(text))

Interest over time

This is the one most people used pytrends for:

df = google_trends(searchTerms=["web scraping"], geo="US", timeRange="today 12-m")

timeline = df[df.section == "timeline"][["date", "value"]]
timeline["date"] = pd.to_datetime(timeline["date"])
print(timeline.tail())

Time ranges are the same strings the website uses: now 1-H, now 7-d, today 3-m, today 5-y, all. For exact
dates use customTimeRange="2024-01-01 2024-12-31".

Comparing terms

Trends values are 0 to 100 inside one request, so two separate requests aren't comparable. Put the terms in one
comparison instead:

df = google_trends(searchTerms=["tea, coffee, matcha"], isMultiple=True, geo="GB", timeRange="today 3-m")

wide = (df[df.section == "timeline"]
        .pivot_table(index="date", columns="term", values="value"))
print(wide.tail())

Google caps this at 5 terms. If you need more, there's a normalizeAcrossQueries option that chains several 5-term
comparisons through one shared anchor term and rescales them onto one scale. Pick your most searched term as the
anchor. With a tiny anchor the rounding gets ugly, and the output flags that with lowPrecision. It costs a bit
extra per comparison because it really is several requests under the hood.

States, cities and US metro areas

pytrends gave you one resolution per call. Here you get states, cities and metro areas (DMA) in the same result:

df = google_trends(searchTerms=["coffee"], geo="US")

metros = (df[df.section == "metro"]
          .sort_values("value", ascending=False)[["label", "code", "value"]])
print(metros.head(10))

The section column is one of timeline, country, subregion, city, metro, relatedQueryTop,
relatedQueryRising.

Related queries

df = google_trends(searchTerms=["web scraping"], geo="US")

rising = df[df.section == "relatedQueryRising"][["label", "formattedValue"]]
print(rising)

One thing that confused me: rising queries are sometimes weird. For "coffee" I once got "pet care tips". I assumed it
was a parsing bug and spent an evening on it. It wasn't. The Trends website shows the exact same list. Google's data
is just like that sometimes.

Trending Now

The newer page at trends.google.com/trending never had a pytrends equivalent. This gets the list per country (or US
state) with search volume:

df = google_trends(mode="trendingNow", trendingGeos=["US", "GB"], trendingHours="24",
                   trendingSort="volume", maxTrendsPerGeo=25)

print(df[["geo", "rank", "title", "searchVolume", "increasePercentage", "categories"]].head(10))

Schedule it hourly with onlyNewSinceLastRun=True and each run only returns trends you haven't seen yet. It's
basically a cheap alert feed.

If you'd rather have JSON

run = client.actor("alom/google-trends-scraper").call(run_input={"searchTerms": ["web scraping"], "geo": "US"})
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["searchTerm"], len(item["interestOverTime_timelineData"]), "points")

One limitation

Related topics come back empty. Google only shows them to signed-in users now, so pytrends and every logged-out
tool lose them too. Related queries still work fine.

So which one?

If you need a couple of charts, just download the CSV from the Trends website, honestly. If you get into the official
API alpha, use that. If you have something scheduled or bulk, or it sits inside a pipeline, paying someone else to
deal with the 429s is cheaper than doing it yourself, at least it was for me.