YouTube's innertube player API: 200 OK doesn't mean you got the data

#youtube #python #webscraping #apify

Building a YouTube Shorts sponsorship scraper, the innertube player endpoint answers every request with HTTP 200 — including the ones where it silently gave up.

Quick answer

YouTube's public youtubei/v1/player endpoint returns 200 OK with an empty videoDetails object when the requesting IP is degraded (shared datacenter pool, rate-limited, etc.) — no error field, no non-2xx status, just a payload with nothing useful in it. If your client checks response.status_code == 200 and moves on, you'll silently ship empty rows. The fix is to treat an empty videoDetails dict as its own signal, log it, and let the caller decide whether to retry with a different proxy — never trust the status code alone.

Why does the player endpoint return 200 with no data?

Here's the actual guard, straight from the scraper:

try:
    details = resp.json().get("videoDetails") or {}
except ValueError:
    logger.warning("youtube %s player returned non-JSON", video_id)
    return {}
if not details:
    # HTTP 200 with no videoDetails is the signature of an IP-degraded
    # response (e.g. routing through the shared datacenter proxy pool).
    logger.warning(
        "youtube %s player returned 200 but no videoDetails — "
        "likely IP-degraded (try proxy OFF / a residential group)",
        video_id,
    )
return details

We rotate through the chrome131 impersonation profile on every session and route through Apify's residential proxy pool when a run needs volume, specifically because the datacenter-degraded case above is real and reproducible. When it happens anyway, the Actor doesn't fabricate a row — it logs the video ID and moves to the next Short, so one bad response never poisons a batch.

The cookie the Shorts tab won't work without

Before any of that, you have to list a channel's Shorts in the first place. Fetch youtube.com/@handle/shorts without a consent cookie and the tab renders zero video IDs — not an error, just an empty page. The header block that fixes it is one line:

# Consent cookie is REQUIRED — without it the shorts tab returns 0 IDs.
CONSENT_HEADERS = {"Accept-Language": "en-US", "Cookie": "SOCS=CAI;CONSENT=YES+1"}

That cookie rides on every request in the session — the shorts-list GET and the innertube POST both need it. Miss it on one and that call quietly returns nothing.

A transparent score beats a black-box "sponsored" flag

Most creator-intel tools give you a boolean and no way to audit it. This Actor scores each Short as a weighted sum over three text sources — the spoken transcript, the description, and the title — and returns the exact phrases that fired:

STRONG_WEIGHT = 0.5    # "sponsored by", "use code", "paid partnership"
WEAK_WEIGHT = 0.15     # "check out", "discount", "% off"
HASHTAG_WEIGHT = 0.4   # #ad, #sponsored, #partner
MENTION_DOMAIN_WEIGHT = 0.1  # @mentions and *.com/.io domains in the description
SPONSORED_THRESHOLD = 0.5

A single #sponsored hashtag alone scores 0.4 — disclosed, but under the 0.5 bar for is_likely_sponsored. Say "this video is sponsored by Surfshark, use code MARQUES" out loud and you cross 0.5 on two strong phrases before the hashtag even gets counted. The transcript is the primary signal on purpose: caption-only scrapers miss every creator who discloses verbally and skips the hashtag.

The transcript fetch itself has to bridge a real library API break — youtube-transcript-api moved from a static get_transcript() method (0.6.x) to an instance .fetch() method (1.x) between releases, so the Actor tries the instance API first and falls back to the static one on AttributeError rather than pinning to one version and breaking on the other.

Is scraping public YouTube metadata and transcripts legal?

The data path here is entirely public: the Shorts tab HTML anyone's browser loads, the same player endpoint every youtube.com page calls client-side, and transcripts YouTube itself renders as captions. No login, no private API keys, no paid data pulled from behind auth. Standard care applies — use the data for analysis and outreach, not republishing full transcripts wholesale.

FAQ

Do I need a YouTube API key?
No — this path uses the same public youtubei/v1/player key YouTube ships in every page load (not a secret), plus the public Shorts tab HTML.

What if a Short has no captions?
has_transcript comes back false and transcript_chars is 0 — the row still ships with title, description, tags and hashtag/mention signals; it just can't score the spoken-word phrases.

Why would my sponsorship_score differ from a hashtag-only tool's?
Because hashtag-only tools can't see "sponsored by X" said out loud with no #ad anywhere in the description — this Actor's primary signal is the transcript, not the caption.

How is pricing structured?
$0.20 to start a run, then $0.004 per Short written to the dataset — roughly $4.00 per 1,000 Shorts, and you only pay for Shorts that actually land.

Try it: YouTube Shorts Sponsorship Signals — feed it channel handles, get back one row per Short with a transparent, auditable sponsorship score.

We do the dirty work so your dataset stays clean. 😈

DEV Community