DEV Community

shaojie gong
shaojie gong

Posted on

A playlist has 124 videos. My tool imported 15, then 100, then finally 124.

One of my extension's features imports a whole YouTube playlist into your notebook in one click — paste a playlist link, it pulls every video in as a source. I pointed a 124-video playlist at it to test. It imported 15.

Not 124. Fifteen.

That kicked off two days of chasing round numbers, and every round number turned out to be a lie told by someone else's system.

  1. My importer was reading YouTube's playlist RSS feed — clean, official, no scraping. Turns out that feed hard-caps at ~15 items no matter how big the playlist is. 15 wasn't my data. It was YouTube's feed limit wearing my data's clothes.

So I switched to scraping the playlist page's embedded JSON (ytInitialData) — the same blob the page itself renders from. First run after the switch: 0 videos. The structure had shifted under me. The old playlistVideoRenderer.videoId path was gone; today's YouTube wraps each video in a lockupViewModel with the id sitting in contentId. Fixed the path.

  1. Now it pulled 100. Progress! But the playlist had 124, and 100 is exactly one page. YouTube paginates: to get page 2 you send back a "continuation token" you find at the bottom of page 1. I was finding the token — I could log it — but my loop behaved like there was none.

This one was mine, and it's a good one. My tree-walker returned a single token as it recursed: walk each child, if a child returns a token, keep it. The problem is that after finding the real token, the walk kept going into sibling branches that returned nothing — and "nothing" overwrote my good token with null. The right answer was there for a moment, then a later, emptier branch clobbered it. A DFS that should have been "first non-null wins" was quietly doing "last write wins." Fix: stop returning one token; push every token into an array and take the first. Instantly: 124.

Except — not from inside the actual extension. From a standalone console test, 124. From the extension, still 100.

  1. My panel runs in an iframe on the notebook page. When it fetched YouTube's pagination endpoint, the request carried Origin: chrome-extension://… — and YouTube's API takes one look at that and returns 403. The page fetch (a plain GET of HTML) worked fine; the API call (the one thing that gets you past 100) didn't. I only caught it because I finally logged the proxy's status codes: GET 200, POST 403.

Fix: move the fetches into the background service worker, and add a declarativeNetRequest rule that rewrites Origin/Referer to https://www.youtube.com for that endpoint. Now the request looks like it came from YouTube itself. 200. 124 videos. Done.

The thing I keep thinking about: 15 and 100 were both round numbers, and both were somebody else's limit masquerading as my result. When a scraper stops at a suspiciously clean number, that's rarely where your data ends — it's where a page size, a feed cap, or a rate limit begins. The number is a fingerprint of the wall you just hit, not of the thing you're counting.

That's the tax on building a tool that lives on top of someone else's product with no API contract. Every layer — the feed, the DOM shape, the pagination, the CORS policy — can drift or bite, and none of it is yours to stabilize. You just get good at reading round numbers as clues.

What's the most misleading "round number" bug you've hit — where the value looked like an answer but was actually a limit?

— building NotebookBloom in public, #16

Top comments (0)