DEV Community: Gautham Krishna

youtube-transcript-api RequestBlocked and IpBlocked on cloud servers, and what actually fixes it

Gautham Krishna — Sat, 24 Oct 2026 19:12:24 +0000

If you've used youtube-transcript-api you've probably seen this. It works on your laptop. You deploy it to AWS or a VPS or some serverless thing, and every call dies with RequestBlocked or IpBlocked.

I've hit this a lot while building transcript scrapers. This week I reran it from scratch to get real numbers instead of forum guesses.

Why it works locally and fails on a server

YouTube decides per IP. The library's own README says it plainly: "YouTube has started blocking most IPs that are known to belong to cloud providers (like AWS, Google Cloud Platform, Azure, etc.)". A home connection looks like a person watching videos. A datacenter range doesn't.

The two errors mean slightly different things. I read the source of version 1.2.4 to check:

Error	What YouTube did
`RequestBlocked`	The player API said `LOGIN_REQUIRED` with its bot-detection reason, the "Sign in to confirm you're not a bot" wall
`IpBlocked`	YouTube answered HTTP 429, or sent a reCAPTCHA page instead of the video page

IpBlocked is a subclass of RequestBlocked, so except RequestBlocked catches both.

Reproducing it

Here's the smallest script that shows the problem:

from youtube_transcript_api import YouTubeTranscriptApi

ytt_api = YouTubeTranscriptApi()
transcript = ytt_api.fetch("jNQXAC9IVRw")
for snippet in transcript[:3]:
    print(snippet.start, snippet.text)

From my home connection:

1.2 All right, so here we are, in front of the
elephants
5.318 the cool thing about these guys is that they
have really...
7.974 really really long trunks

I didn't spin up an EC2 box for this. I sent the same script through datacenter proxy IPs instead. They aren't AWS, but they're datacenter ranges, same as AWS. I also tried residential proxy IPs, and my home connection again. 10 tries each, on 5 different videos, with a new IP per try on the proxies:

Where the request came from	Worked
My home connection	10 of 10
10 datacenter IPs	6 of 10, the other 4 raised `RequestBlocked`
10 residential IPs	9 of 10, 1 raised `IpBlocked`

Honestly I expected the datacenter IPs to do worse. But failing 4 times out of 10 is useless for a job nobody is watching. And even residential IPs aren't perfect, which matters for the fix below.

Fix 1: run it from your own machine

If you have a few hundred videos and no deadline, just run it at home. Go slowly. The same README warns that a home IP gets blocked too if you send too many requests.

Fix 2: a rotating residential proxy

That's the README's own answer. The library ships WebshareProxyConfig for Webshare, and GenericProxyConfig for any other provider. Heads up, the Webshare links in that README are referral links, which the author says openly.

Here's the snag I hit. GenericProxyConfig doesn't retry. Its retries_when_blocked is 0, and it keeps the connection alive, so with a rotating proxy you can keep landing on the same blocked IP. Only WebshareProxyConfig retries, 10 times by default. With any other provider, retry yourself and build a new client each time:

import os
from youtube_transcript_api import YouTubeTranscriptApi, RequestBlocked
from youtube_transcript_api.proxies import GenericProxyConfig

PROXY_URL = os.environ["PROXY_URL"]  # rotating residential proxy: http://user:pass@host:port


def fetch(video_id, tries=3):
    for attempt in range(tries):
        # a new client opens a new connection, so a rotating proxy gives it a new IP
        ytt_api = YouTubeTranscriptApi(
            proxy_config=GenericProxyConfig(http_url=PROXY_URL, https_url=PROXY_URL)
        )
        try:
            return ytt_api.fetch(video_id)
        except RequestBlocked:  # IpBlocked is a subclass, so this catches both
            if attempt == tries - 1:
                raise


print(fetch("jNQXAC9IVRw")[0].text)

All right, so here we are, in front of the
elephants

Residential proxies bill by traffic, so check what each video costs you. The library loads the full watch page before it asks for captions. For the three videos I checked, that page was 310 to 370 KB compressed.

Fix 3: yt-dlp

yt-dlp can grab just the subtitles, without the video:

yt-dlp --skip-download --write-subs --write-auto-subs --sub-langs en --sub-format vtt -o "%(id)s.%(ext)s" "https://www.youtube.com/watch?v=jNQXAC9IVRw"

WEBVTT
Kind: captions
Language: en

00:00:01.200 --> 00:00:03.360
All right, so here we are, in front of the
elephants

From home it took 3 seconds with version 2026.08.19. Through 5 datacenter IPs, 3 failed with this:

ERROR: [youtube] jNQXAC9IVRw: Sign in to confirm you’re not a bot. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp  for how to manually pass cookies. Also see  https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies  for tips on effectively exporting YouTube cookies

Same wall, so switching to yt-dlp doesn't get you around the IP problem. It needs a residential proxy on a server too.

The error tells you to pass cookies. I wouldn't do that with a real account. yt-dlp's own wiki says "By using your account with yt-dlp, you run the risk of it being banned (temporarily or permanently)."

This article was created with the help of AI. An AI agent (Claude) ran every test above on my machine on 25 Oct 2026 and drafted the text from those runs, and every output block is copied from them.

pytrends is archived and throws 429s. What still works for Google Trends data in 2026

Gautham Krishna — Sat, 24 Oct 2026 19:11:27 +0000

I build scrapers, so I spend more time than I'd like poking at Google Trends from code. This week I sat down and retested every free route I know, from my laptop and from a few datacenter IPs, to see what actually works in October 2026.

I also got my home IP blocked halfway through, which turned out to be the most useful part.

Is pytrends dead?

Sort of. The GeneralMills/pytrends repo is archived, and the last PyPI release is 4.9.2 from April 2023. In February 2025 the maintainer posted a goodbye issue telling people to stop using it. His reason wasn't only lack of time. He wrote that when Google thinks you're a bot, it gives you "slightly altered data different from the one you can see in their website".

The library still runs, though. I installed 4.9.2 today and this worked first try:

from pytrends.request import TrendReq

pytrends = TrendReq(hl="en-US", tz=0)
pytrends.build_payload(["chatgpt", "claude"], timeframe="today 3-m", geo="US")
df = pytrends.interest_over_time()
print(df.tail(3))

            chatgpt  claude  isPartial
date                                  
2026-10-22       81      26      False
2026-10-23       75      23      False
2026-10-24       51      17       True

Here's what each method did on 25 Oct 2026:

pytrends method	What happened
`interest_over_time()`	Works
`interest_by_region()`	Works
`related_queries()`	Works
`related_topics()`	Crashes with `IndexError`, because Google sends back an empty list
`trending_searches()`	Google answers 404
`realtime_trending_searches()`	Google answers 404

So if you only need timelines, regions and related queries, pytrends still does the job. The trending calls are gone and nobody is going to fix them.

The 429s

The thing people actually search for is this one:

pytrends.exceptions.TooManyRequestsError: The request failed: Google returned a response with code 429

I tried to trigger it on purpose and mostly couldn't. 20 keywords in a row from my home connection, about one every 2 seconds, all fine. Then 60 keywords on 20 threads through a single datacenter IP, each one asking for the timeline, the regions and the related queries. All 60 came back in 36 seconds.

Then my home IP got blocked anyway. After roughly 70 requests in 6 minutes, a couple of them full headless-browser page loads from a library I'll get to below, every request started bouncing to google.com/sorry, Google's "unusual traffic" page. pytrends reports that as the 429 above. The block was still there more than 15 minutes later, when I stopped checking.

So I can't give you a safe number of requests per minute. The block is per IP, it clearly isn't a plain request count, and once you're in it you stay in it for a while. What I'd do:

Build one TrendReq and reuse it, instead of a new one per keyword.
Go one keyword at a time with a pause. 2 seconds a call was fine for me.
Cache everything. Weekly data for the last 12 months doesn't change between Monday and Tuesday.
When you land on google.com/sorry, stop. Retrying in a loop just keeps you there.

Skip the library and call the endpoints yourself

pytrends is a thin wrapper around the JSON endpoints the Trends website calls. If you don't want an archived dependency, you can call them with requests in about 30 lines. /trends/api/explore gives you a signed token for each chart on the page, and /trends/api/widgetdata/multiline gives you the timeline for that token.

import json
import requests

BASE = "https://trends.google.com/trends/api"
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0"


def get_json(path, params):
    r = s.get(f"{BASE}/{path}", params=params, timeout=30)
    if r.status_code == 429 and "NID" in r.cookies:
        # the first call from a new IP often gets a 429 that sets the NID cookie
        r = s.get(f"{BASE}/{path}", params=params, timeout=30)
    r.raise_for_status()
    return json.loads(r.text[r.text.find("{"):])  # body starts with )]}' junk


req = {
    "comparisonItem": [{"keyword": k, "geo": "US", "time": "today 3-m"} for k in ["chatgpt", "claude"]],
    "category": 0,
    "property": "",
}
explore = get_json("explore", {"hl": "en-US", "tz": "0", "req": json.dumps(req)})
ts = next(w for w in explore["widgets"] if w["id"] == "TIMESERIES")
print("userType:", ts["request"]["userConfig"]["userType"])

data = get_json("widgetdata/multiline", {
    "hl": "en-US", "tz": "0",
    "req": json.dumps(ts["request"], separators=(",", ":")),
    "token": ts["token"],
})
for p in data["default"]["timelineData"][-3:]:
    print(p["formattedTime"], p["value"])

userType: USER_TYPE_SCRAPER
Oct 22, 2026 [81, 26]
Oct 23, 2026 [75, 23]
Oct 24, 2026 [52, 17]

The NID retry is there because Google often answers the very first request from a new IP with a 429 that sets a cookie, and the same request works right after.

Now look at userType. Google labels the request USER_TYPE_SCRAPER. I opened the same comparison on trends.google.com in Chrome and captured the website's own call, and there it says USER_TYPE_LEGIT_USER. So Google knows, which is what the pytrends maintainer was getting at.

I compared the numbers anyway. For the full days the script and the website matched exactly, 81/26 and 75/23. For the last, partial day the script said 52/17 and the website said 50/16, and pytrends had said 51/17 a few minutes earlier. That's one comparison on one day, so it doesn't prove Google never alters scraper data. For ordinary keywords I wouldn't lose sleep over it.

Where the label clearly matters is related topics. Google now sends them only to requests that come with the website's reCAPTCHA check, so every script gets an empty list. That's the IndexError in the table above.

One more snag. The top rising query for "claude" in the US was "codex vs claude code abraham quiros villalba", marked Breakout, and two more rising queries had the same name tacked on. The website shows them too. Rising queries come straight from Google with no cleanup, so filter them before they go in a report.

Maintained libraries

trendspyg is the most active one I found, version 1.9.0 from 1 Oct 2026. It reads trending searches from that RSS feed and from a CSV export. For keyword data it drives a headless Chrome through the real website. My first explore call took 27 seconds and returned data. The second one raised RateLimitError, "Google Trends is blocking Explore requests from this IP". Its error message says Google allows roughly 8 to 10 fresh browser sessions per hour. To be fair, I'd already sent a lot of requests from that IP by then, so I can't say it was trendspyg that tipped it over.

trendspy, which the pytrends maintainer pointed people to in 2025, hasn't had a release since December 2024.

The official Google Trends API

Google announced one in July 2025. As of today the page still says Google is "now accepting applications for alpha testers", with a form to apply. I don't have access, so I can't tell you how it behaves. On paper it has 5 years of rolling data, daily to yearly steps, regions, and values that aren't squashed to 0 to 100, so you can merge separate requests. If you need Trends data for something that has to keep running for years, apply. It's the only route here that isn't scraping.

BigQuery

Google also publishes a public BigQuery dataset, bigquery-public-data.google_trends. It holds the top 25 and top 25 rising searches for each US metro area, refreshed daily, plus tables for other countries. It's good for "what's hot in Denver this week". It can't look up your keyword, and I didn't run it for this post.

When I'd stop doing it myself

For a notebook, pytrends or the raw calls above are fine. Keep the volume low and cache what you get.

It gets annoying when it's a job that runs every day on a server. Then you're handling blocked IPs, retries, and endpoints that change without notice, like the trending calls that broke in pytrends.

That's when paying for a Trends API starts to make sense, SerpApi has one for example, because then keeping up with Google's changes is someone else's job. Ask any provider how they handle related topics before you pay, since the free routes above can't get them.