One of my extension's features imports a whole YouTube playlist into your notebook in one click — paste a playlist link, it pulls every video in as...
For further actions, you may consider blocking this person and/or reporting abuse
Dealing with YouTube API pagination and rate limits can be incredibly frustrating when building batch importers. I remember hitting a wall with a similar scraper where the API would silently drop requests if you exceeded the quota, forcing me to implement exponential backoff and chunked processing. Did you end up using the standard YouTube Data API v3 with page tokens, or did you have to scrape the initial page payload directly to bypass those strict quota limits? Handling the state for partial imports is a great edge case to solve for user experience.
Yeah, this is exactly where it hurt. Short answer: I skipped Data API v3 entirely.
The dealbreaker was the API key. It's a client-side extension, so any key I ship gets shared across every single user, and playlistItems.list eats quota fast. One power user importing a few big playlists and everyone else is locked out for the rest of the day. Dead on arrival.
So I went the scrape route. Pull ytInitialData off the playlist page for the first batch, then for pagination I hit YouTube's own internal endpoint (youtubei/v1/browse) with the continuation token from the bottom of each page — same InnerTube API their web player uses. No key, no quota to blow through. The tradeoff is the obvious one: it's undocumented and it drifts. They literally moved the videoId field on me halfway through the project.
On the silent-drop thing you mentioned — InnerTube doesn't have a hard published quota, but it'll soft-throttle if you hammer it, so I space the continuation calls out instead of firing them all at once. Haven't needed full exponential backoff yet, but I can see it coming the day someone points a 2,000-video playlist at it.
And 100% agreed on partial imports — that was the part I cared most about. It dedupes on re-run, so if it dies at video 80 you just run it again and it fills in the rest instead of starting from scratch. Failed items get surfaced with a count, never silently dropped. Silent drops are the cardinal UX sin here.
Bypassing the Data API entirely makes total sense for a client-side extension, since a shared quota is basically a ticking time bomb. Switching to scraping solves the immediate rate limit issue, but I'm curious how you handle the inevitable DOM changes or IP throttling from YouTube. Are you routing the scrape requests through a proxy network, or keeping it strictly local to the user's machine?
Strictly local, and honestly that's the part I'd defend hardest — no proxy, no relay, nothing of mine sits in the path. The requests fire from the user's own browser with their own YouTube cookies and their own IP. From YouTube's side it's indistinguishable from that person scrolling the playlist page themselves, because functionally that's what it is. The only sleight of hand is a declarativeNetRequest rule that rewrites the Origin/Referer header back to youtube.com — a request coming out of a chrome-extension:// origin gets a 403 otherwise. That's it. No server of mine ever touches YouTube.
Which also means there's no shared IP to throttle. This was the whole reason the API key was a non-starter — one shared key, one shared quota, one power user nukes it for everybody. Going local flips that completely: every user brings their own rate budget. Ten thousand users don't stack up against one bucket, they're ten thousand separate buckets. A proxy network would have quietly re-introduced the exact single-choke-point I was running away from, plus a hosting bill I don't want and a privacy story I couldn't stand behind. Hard pass.
The DOM drift you're right about — that's the standing tax and there's no clever way out, only two dampers. First, I don't scrape rendered HTML, I read ytInitialData and the InnerTube JSON, which drifts way slower than the visual DOM — though it still drifts, they moved the videoId field on me mid-project. Second, when the primary path returns nothing, it falls back to the official playlist RSS feed. RSS caps around 15 videos so it's a worse result, but "you got 15 instead of 124" degrades a lot more gracefully than a hard zero while I ship a fix. And the fix, when a field moves, is a one-line selector patch, not a re-architecture. It's a parasite product by nature — living on someone else's platform means budgeting for the day they redecorate. I priced that in from day one.
Relying entirely on the user's own session cookies and IP is a smart way to sidestep API quotas while keeping the architecture completely trustless. Using declarativeNetRequest to mimic natural scrolling is a clever workaround that keeps bot-detection happy without needing a backend. Out of curiosity, how does the extension handle the progressive loading chunks, like the initial 15 videos versus the final 124?
Ha, good question — but the 15 and the 124 aren't two stages of the same load. They're two completely different code paths, and which one you get depends on where the data comes from.
The 15 is the fallback. If the main path can't get a clean read on the page — playlist's not fully public, layout's doing something weird, whatever — I drop down to YouTube's RSS feed for that playlist. The feed is dead simple and never breaks, but it's hard-capped at 15 items on YouTube's end. So if you ever see exactly 15 come back, that's the tell that it fell back to RSS and you're only getting a slice.
The 124 is the real path. First chunk (~100) comes straight out of ytInitialData that's already embedded in the playlist page — no request needed, it's just sitting there in the HTML. Then for anything past that, YouTube itself paginates with a continuation token tucked at the bottom of each response. So I grab that token and POST it to youtubei/v1/browse (their own InnerTube endpoint), get the next ~100 plus the next token, and just loop until there's no token left. That's it — no page numbers, no "give me page 2," you follow the breadcrumb until it runs out.
The one gotcha that bit me there: the continuation token isn't always a single clean field. My first tree-walker did "last write wins" and happily overwrote a good token with a null further down the tree, so pagination just... stopped early. Fix was to collect every token I find into an array and take the first valid one instead of trusting the last. That's the difference between capping at 100 and actually getting to 124.
And each chunk gets appended + deduped as it lands, not at the end — so if it dies mid-loop, the resume picks up from what's already in instead of refetching. The progress you see counting up is literally each continuation page landing.
tl;dr: 15 = degraded RSS fallback, 124 = ytInitialData seed + InnerTube continuation loop. Not the same source getting bigger — one's plan B.