A 586 KB HTTP 200 from YouTube that contains zero videos

#webscraping #python #api #youtube

Quick answer: GET https://www.youtube.com/@mkbhd/videos returns HTTP 200 and 585,922 bytes of HTML that contains zero videos. It is Google's EU consent-wall shell. Send two cookies nobody has to log in for — CONSENT=PENDING+987 and SOCS=CAI — and the same URL returns 1,200,225 bytes with a real ytInitialData payload and 30 video IDs. The block was never a block. It was a dialog.

Why does a 586 KB page contain no videos?

Because size feels like evidence and isn't. Half a megabyte of HTML reads like a page that loaded, so the natural next move is to blame the parser and start tuning selectors against markup that was never going to contain a video.

It's the consent interstitial. Google serves it to unauthenticated requests from EU-adjacent egress, and it is a full, heavy, perfectly valid page. The signal that matters is not the status code and not the byte count — it is whether the payload marker is present:

def carries_channel_data(html: str) -> bool:
    """The only honest reachability check: is the data actually here?"""
    if "ytInitialData" not in html:
        logger.warning("consent shell: 200 OK, %d bytes, no ytInitialData marker", len(html))
        return False
    return True

A 200 is not reachability, and a big 200 is not reachability either. Assert on the data marker you intend to parse, never on the status line.

What do the two cookies actually do?

They record a consent choice, which is all the interstitial was waiting for. CONSENT=PENDING+987 and SOCS=CAI skip the redirect. That is the entire fix.

No login. No account. No OAuth. No session warming. No headless browser. The distinction worth being precise about: this is not an authentication bypass — it never touches private data, and it asks for exactly what a logged-out visitor sees after clicking a button. It just stops the redirect from eating the response.

How do you get past the first 30 videos?

The same payload that carries the first batch also carries a continuationCommand token, and you POST it back to YouTube's internal /youtubei/v1/browse endpoint to get the next batch, each response carrying the token for the one after it.

async def browse_continuation(token: str) -> dict:
    """POST an innertube /browse continuation and get the next page of items."""
    ...

That is a keyset cursor by another name: you never compute an offset, you hand back the token the server gave you. It has two practical consequences. You cannot jump to page 7 — you walk. And you cannot lose your place, because the server is the one tracking it.

Isn't YouTube blocked from datacenter IPs?

Partly, and the scope matters more than the headline. Our own record is specific about it: /youtubei/v1/player is the endpoint that gets refused from datacenter ranges. The /browse continuations — the ones this Actor walks — work fine from datacenter egress and in the cloud.

"YouTube blocks datacenter IPs" is true enough to be repeated and vague enough to be useless. Which endpoint is the whole question, and an endpoint-scoped answer saves you from paying for residential proxy you don't need.

What does a row look like?

{
  "video_id": "6D__H_DO2Xk",
  "title": "The Best Smartphone You Can Buy",
  "url": "https://www.youtube.com/watch?v=6D__H_DO2Xk",
  "view_count": 4821993,
  "published_time_text": "2 months ago",
  "duration_seconds": 903,
  "thumbnail_url": "https://i.ytimg.com/vi/6D__H_DO2Xk/hqdefault.jpg",
  "channel_handle": "@mkbhd"
}

Measured on a live 90-row cloud run across two channels — 45 rows from each, which is the number that proves the continuation actually ran rather than just the first page.

😈 YouTube Channel Videos Scraper exports every uploaded video from any public YouTube channel — video ID, title, URL, view count, published time, duration and thumbnail — walking the innertube continuation so you get the whole channel, not the first screen. We handle the consent wall, the continuation tokens, the retries and the endpoint-specific IP rules, so you get rows instead of a 586 KB page full of nothing. $5.00 per 1,000 videos.

FAQ

Do I need a Google account or an API key?
No. No login, no OAuth, no quota. These are public channel pages plus YouTube's own continuation endpoint.

Is this the YouTube Data API?
No, and that's usually the point — the official API bills you quota units and caps daily. This reads the same public surface the website reads.

How many videos can I pull from one channel?
The continuation walks until the channel runs out. Set a per-channel cap if you only want the recent tail.

Can I scrape several channels in one run?
Yes — pass a list of handles or channel URLs. Each channel is fault-isolated, so one bad handle never zeroes the rest of the run.

Why did my own scraper return 200 and nothing?
Almost certainly the consent shell described above. Check for ytInitialData before you touch your selectors.

DEV Community