One of my extension's features imports a whole YouTube playlist into your notebook in one click — paste a playlist link, it pulls every video in as a source. I pointed a 124-video playlist at it to test. It imported 15.
Not 124. Fifteen.
That kicked off two days of chasing round numbers, and every round number turned out to be a lie told by someone else's system.
- My importer was reading YouTube's playlist RSS feed — clean, official, no scraping. Turns out that feed hard-caps at ~15 items no matter how big the playlist is. 15 wasn't my data. It was YouTube's feed limit wearing my data's clothes.
So I switched to scraping the playlist page's embedded JSON (ytInitialData) — the same blob the page itself renders from. First run after the switch: 0 videos. The structure had shifted under me. The old playlistVideoRenderer.videoId path was gone; today's YouTube wraps each video in a lockupViewModel with the id sitting in contentId. Fixed the path.
- Now it pulled 100. Progress! But the playlist had 124, and 100 is exactly one page. YouTube paginates: to get page 2 you send back a "continuation token" you find at the bottom of page 1. I was finding the token — I could log it — but my loop behaved like there was none.
This one was mine, and it's a good one. My tree-walker returned a single token as it recursed: walk each child, if a child returns a token, keep it. The problem is that after finding the real token, the walk kept going into sibling branches that returned nothing — and "nothing" overwrote my good token with null. The right answer was there for a moment, then a later, emptier branch clobbered it. A DFS that should have been "first non-null wins" was quietly doing "last write wins." Fix: stop returning one token; push every token into an array and take the first. Instantly: 124.
Except — not from inside the actual extension. From a standalone console test, 124. From the extension, still 100.
- My panel runs in an iframe on the notebook page. When it fetched YouTube's pagination endpoint, the request carried Origin: chrome-extension://… — and YouTube's API takes one look at that and returns 403. The page fetch (a plain GET of HTML) worked fine; the API call (the one thing that gets you past 100) didn't. I only caught it because I finally logged the proxy's status codes: GET 200, POST 403.
Fix: move the fetches into the background service worker, and add a declarativeNetRequest rule that rewrites Origin/Referer to https://www.youtube.com for that endpoint. Now the request looks like it came from YouTube itself. 200. 124 videos. Done.
The thing I keep thinking about: 15 and 100 were both round numbers, and both were somebody else's limit masquerading as my result. When a scraper stops at a suspiciously clean number, that's rarely where your data ends — it's where a page size, a feed cap, or a rate limit begins. The number is a fingerprint of the wall you just hit, not of the thing you're counting.
That's the tax on building a tool that lives on top of someone else's product with no API contract. Every layer — the feed, the DOM shape, the pagination, the CORS policy — can drift or bite, and none of it is yours to stabilize. You just get good at reading round numbers as clues.
What's the most misleading "round number" bug you've hit — where the value looked like an answer but was actually a limit?
— building NotebookBloom in public, #16
Top comments (25)
Dealing with YouTube API pagination and rate limits can be incredibly frustrating when building batch importers. I remember hitting a wall with a similar scraper where the API would silently drop requests if you exceeded the quota, forcing me to implement exponential backoff and chunked processing. Did you end up using the standard YouTube Data API v3 with page tokens, or did you have to scrape the initial page payload directly to bypass those strict quota limits? Handling the state for partial imports is a great edge case to solve for user experience.
Yeah, this is exactly where it hurt. Short answer: I skipped Data API v3 entirely.
The dealbreaker was the API key. It's a client-side extension, so any key I ship gets shared across every single user, and playlistItems.list eats quota fast. One power user importing a few big playlists and everyone else is locked out for the rest of the day. Dead on arrival.
So I went the scrape route. Pull ytInitialData off the playlist page for the first batch, then for pagination I hit YouTube's own internal endpoint (youtubei/v1/browse) with the continuation token from the bottom of each page — same InnerTube API their web player uses. No key, no quota to blow through. The tradeoff is the obvious one: it's undocumented and it drifts. They literally moved the videoId field on me halfway through the project.
On the silent-drop thing you mentioned — InnerTube doesn't have a hard published quota, but it'll soft-throttle if you hammer it, so I space the continuation calls out instead of firing them all at once. Haven't needed full exponential backoff yet, but I can see it coming the day someone points a 2,000-video playlist at it.
And 100% agreed on partial imports — that was the part I cared most about. It dedupes on re-run, so if it dies at video 80 you just run it again and it fills in the rest instead of starting from scratch. Failed items get surfaced with a count, never silently dropped. Silent drops are the cardinal UX sin here.
Bypassing the Data API entirely makes total sense for a client-side extension, since a shared quota is basically a ticking time bomb. Switching to scraping solves the immediate rate limit issue, but I'm curious how you handle the inevitable DOM changes or IP throttling from YouTube. Are you routing the scrape requests through a proxy network, or keeping it strictly local to the user's machine?
Strictly local, and honestly that's the part I'd defend hardest — no proxy, no relay, nothing of mine sits in the path. The requests fire from the user's own browser with their own YouTube cookies and their own IP. From YouTube's side it's indistinguishable from that person scrolling the playlist page themselves, because functionally that's what it is. The only sleight of hand is a declarativeNetRequest rule that rewrites the Origin/Referer header back to youtube.com — a request coming out of a chrome-extension:// origin gets a 403 otherwise. That's it. No server of mine ever touches YouTube.
Which also means there's no shared IP to throttle. This was the whole reason the API key was a non-starter — one shared key, one shared quota, one power user nukes it for everybody. Going local flips that completely: every user brings their own rate budget. Ten thousand users don't stack up against one bucket, they're ten thousand separate buckets. A proxy network would have quietly re-introduced the exact single-choke-point I was running away from, plus a hosting bill I don't want and a privacy story I couldn't stand behind. Hard pass.
The DOM drift you're right about — that's the standing tax and there's no clever way out, only two dampers. First, I don't scrape rendered HTML, I read ytInitialData and the InnerTube JSON, which drifts way slower than the visual DOM — though it still drifts, they moved the videoId field on me mid-project. Second, when the primary path returns nothing, it falls back to the official playlist RSS feed. RSS caps around 15 videos so it's a worse result, but "you got 15 instead of 124" degrades a lot more gracefully than a hard zero while I ship a fix. And the fix, when a field moves, is a one-line selector patch, not a re-architecture. It's a parasite product by nature — living on someone else's platform means budgeting for the day they redecorate. I priced that in from day one.
Relying entirely on the user's own session cookies and IP is a smart way to sidestep API quotas while keeping the architecture completely trustless. Using declarativeNetRequest to mimic natural scrolling is a clever workaround that keeps bot-detection happy without needing a backend. Out of curiosity, how does the extension handle the progressive loading chunks, like the initial 15 videos versus the final 124?
Ha, good question — but the 15 and the 124 aren't two stages of the same load. They're two completely different code paths, and which one you get depends on where the data comes from.
The 15 is the fallback. If the main path can't get a clean read on the page — playlist's not fully public, layout's doing something weird, whatever — I drop down to YouTube's RSS feed for that playlist. The feed is dead simple and never breaks, but it's hard-capped at 15 items on YouTube's end. So if you ever see exactly 15 come back, that's the tell that it fell back to RSS and you're only getting a slice.
The 124 is the real path. First chunk (~100) comes straight out of ytInitialData that's already embedded in the playlist page — no request needed, it's just sitting there in the HTML. Then for anything past that, YouTube itself paginates with a continuation token tucked at the bottom of each response. So I grab that token and POST it to youtubei/v1/browse (their own InnerTube endpoint), get the next ~100 plus the next token, and just loop until there's no token left. That's it — no page numbers, no "give me page 2," you follow the breadcrumb until it runs out.
The one gotcha that bit me there: the continuation token isn't always a single clean field. My first tree-walker did "last write wins" and happily overwrote a good token with a null further down the tree, so pagination just... stopped early. Fix was to collect every token I find into an array and take the first valid one instead of trusting the last. That's the difference between capping at 100 and actually getting to 124.
And each chunk gets appended + deduped as it lands, not at the end — so if it dies mid-loop, the resume picks up from what's already in instead of refetching. The progress you see counting up is literally each continuation page landing.
tl;dr: 15 = degraded RSS fallback, 124 = ytInitialData seed + InnerTube continuation loop. Not the same source getting bigger — one's plan B.
Falling back to the RSS feed when the primary extraction path fails is a solid graceful degradation strategy, especially since YouTube's feeds notoriously cap out at 15 items. It guarantees you at least get partial metadata instead of a hard failure when the layout changes or access gets restricted. Since the RSS feed is inherently limited, do you implement any pagination tricks or secondary fallbacks to eventually catch the remaining 109 videos, or does the user have to manually intervene?
Falling back to the RSS feed is a clever graceful degradation strategy when the primary scraping path hits a wall. Since RSS feeds often cap out at a limited number of items, I am curious how your main code path handles the pagination required to actually fetch all 124 videos. Do you rely on a headless browser for that main path, or are you reverse-engineering the internal API endpoints?
Reverse-engineering the internal API, no headless browser anywhere. A headless browser isn't even on the table for an MV3 extension — I've got no background page that can drive one, and spinning up a whole second browser to read a list I'm already looking at would be absurd overhead.
The one thing I'd push back on is the word "reverse-engineering" — makes it sound sneakier than it is. youtubei/v1/browse is the exact endpoint YouTube's own page hits to load more videos as you scroll. I'm just calling it directly with the continuation token they hand me, from a fetch running in the page's own context, so it's the user's live session — same cookies, same IP, no automation pretending to be a person. It's less "I cracked their API" and more "I stopped waiting for the scroll and asked for the next page myself." The whole loop is that one endpoint until the token runs dry.
That makes total sense regarding the MV3 constraints; spinning up a background service worker to handle a headless browser just to paginate a list would be massive overkill. I also agree with your pushback on the term reverse-engineering, since you are really just inspecting standard network traffic and documenting the youtubei endpoints rather than cracking an encrypted vault. Have you found the internal API to be relatively stable, or do you have to update your payload structures frequently when they tweak the frontend?
good question, and it's actually two different answers. the endpoint itself has been rock solid — youtubei/v1/browse, the continuation-token dance, all of it hasn't moved in months. what drifts is the response shape. the playlistVideoRenderer -> lockupViewModel switch i mentioned? that was them reshaping the json, not the api. so i almost never touch the request side, but the parser i treat as disposable — narrow path reads, and it fails loud the second a field moves so i catch it early instead of silently importing half a list. so far that's held up way better than i expected going in.
It makes perfect sense that the endpoint remains stable while YouTube constantly shuffles the internal JSON schema behind the scenes. Since you treat the parser as the fragile component, do you rely on a strict schema validation library like Zod to catch these structural shifts, or do you build in graceful fallback selectors for when the view model changes?
That distinction between a stable endpoint and a drifting payload is exactly why writing resilient scrapers is such a headache. Since the parser is the main pain point, do you use a schema validation library like Zod to catch those structural changes early, or just rely on aggressive fallback logic? It completely reframes the MV3 background task challenge from a timeout issue to a data-mapping one.
neither, and on purpose. no Zod — felt like carrying a full schema for a shape i don't even own, and it'd just tell me what i already know the moment import breaks. and graceful fallback is the actual trap here: a fallback selector that "kind of" works is how you silently import half a playlist and never notice. so i went the other way — one narrow path to the field i need, and if it's not there i throw hard and stop the whole import. loud and empty beats quiet and half-right when the data isn't mine to trust. you're dead on that it's a data-mapping problem, not a timeout one — that reframing is basically the whole job.
Failing loudly is definitely better than silently corrupting data just to keep the pipeline moving. Skipping Zod makes total sense when the API shape is out of your control and you only need one specific field anyway. Do you find that this strict, narrow path causes friction when the API does minor structural tweaks, or does it just force you to update the selector immediately?
it forces the update — but only when the tweak actually touches my one field, which is the part i like. minor stuff two levels away from what i read just sails right through, i never even see it. so the friction isn't constant, it's targeted: the pipeline goes dark exactly when it should, never for something cosmetic. and honestly "update the selector" is a 10-minute job — i open the raw response, find where my field wandered off to, fix the one path. the expensive version was the old one, where it "worked" and i found out three weeks later half my imports were short. i'll take a loud 10-minute fix over a quiet three-week one every time.
Treating friction as targeted rather than constant is a great way to handle fragile selectors without getting alert fatigue. If the pipeline only halts for the exact critical field you care about, cosmetic DOM changes just become background noise. Since fixing the selector is only a 10-minute raw response check, have you ever written a quick script to automatically diff the JSON and highlight exactly which nested key shifted?
not yet — a full diff would light up like a christmas tree because that payload changes in a hundred harmless places. if it bites me again, i'll save a known-good response and diff only the branches around video IDs and continuation tokens. so far, searching one known video ID in the raw JSON has been faster than building the dete
Diffing the entire payload is definitely a nightmare when the API injects random tracking parameters everywhere. Filtering the diff down to just the continuation tokens and video ID branches is a much smarter debugging strategy than trying to parse the massive raw JSON manually. Have you considered writing a quick script to strip out those volatile fields before diffing, or is the raw text search fast enough to keep you in the flow?
raw search still wins for now. i grab one video ID from the playlist, ctrl+f it, and the surrounding branch usually gives me the new path in under a minute. a normalizer starts earning its keep when this breaks often enough that maintaining it is cheaper than searching. i'm not there yet.
Using a known video ID for raw search is a highly pragmatic shortcut until the API inevitably shuffles the JSON structure again. I completely agree with your cost-benefit analysis on normalizers, since over-engineering a parser for an unpredictable payload is a classic trap. When the ctrl-f method eventually breaks, are you planning to pivot to a strict schema-based normalizer or just write a quick regex patch to get by?
neither. i'd add one narrow structural path to the tree-walker and keep the hard failure when it finds zero valid 11-character video IDs. regex over nested JSON is how i'd recreate the silent-half-import bug, and a full schema would break on fields i never use. patch the seam, not model the whole payload.
Patching the seam rather than modeling the whole payload is exactly the right philosophy for brittle third-party APIs. By keeping the hard failure on zero valid IDs, you completely bypass the silent-half-import bug that plagued the original tool by forcing an immediate crash instead of a partial import. Have you found that this narrow tree-walker path holds up well when the API inevitably tweaks the nesting depth, or do you have to update the structural path frequently?
nesting depth hasn't mattered so far because the walker is recursive. YouTube can wrap the same node three layers deeper and it still gets found. i only patch when the node itself changes shape — like playlistVideoRenderer becoming lockupViewModel. that's happened once, not constantly. depth is noise; identity is the seam.
"Depth is noise; identity is the seam" is a perfect distillation of why brittle path matching fails against YouTube's shifting payload. Relying on a recursive walker to hunt down the structural identity regardless of arbitrary nesting wrappers is a highly resilient pattern. Have you found that this identity-based patching holds up when YouTube introduces entirely new view models for different device types, or do you maintain a fallback mapping for those edge cases?