A research document on the architecture, failures, and breakthroughs of building a cloud-native video acquisition system that defeats modern anti-bot infrastructure.
Abstract
This document chronicles the design, implementation, and iterative breakthroughs of a YouTube video acquisition system built specifically for cloud-based AI agents. The project began with a straightforward goal — enable an AI agent running in a headless cloud environment to download public YouTube videos without a browser, without cookies, and without human intervention. What began as a simple wrapper around existing tooling evolved into a multi-layered bypass architecture that defeats one of the most sophisticated anti-bot systems on the internet: YouTube's datacenter IP detection and Proof-of-Origin token enforcement.
The research reveals a critical truth about modern web infrastructure: the internet is splitting into two tiers. Residential IPs can access content freely. Datacenter IPs are increasingly walled off behind behavioral analysis, attestation challenges, and IP reputation scoring. For AI agents that operate exclusively from cloud environments, this split represents an existential threat to their ability to interact with the web. This document describes how that threat was identified, analyzed, and ultimately routed around — not by breaking the security measures, but by composing multiple legitimate access paths into a resilient fallback chain.
Table of Contents
- The Problem: Why AI Agents Can't Download Videos
- Initial Research: Mapping the Landscape
- First Approach: The Naive Wrapper
- The Wall: Datacenter IP Detection
- Understanding the Block: Technical Deep Dive
- Second Approach: The PO Token Era
- Third Approach: The Bypass Architecture
- The Breakthrough: Compositional Bypass
- Final Architecture: The 13-Method Fallback Chain
- Key Learnings and Takeaways
- Future Implications
1. The Problem: Why AI Agents Can't Download Videos
The use case
AI agents are increasingly deployed in cloud environments to perform research, content analysis, and data gathering. A common requirement is the ability to acquire video content from public sources for processing — transcription, summarization, frame extraction, or archival. The dominant video platform, YouTube, presents a unique challenge because it has evolved one of the most aggressive anti-automation systems on the public web.
The constraints under which a cloud-based AI agent operates are fundamentally different from those of a human user:
- No browser. The agent runs in a headless container or VM. There is no Chrome, no Firefox, no cookie jar, no session persistence across requests in the way a browser provides.
- No credentials. The agent has no Google account, no OAuth token, no logged-in session. It cannot "sign in to confirm you're not a bot" because it has nothing to sign in with.
- Datacenter IP. The agent's network traffic originates from a cloud provider's IP range (AWS, GCP, Azure, Alibaba, etc.). These ranges are publicly known and aggressively flagged by anti-bot systems.
- Non-interactive. The agent cannot solve CAPTCHAs, click "I'm not a robot," or wait for a human to solve a browser challenge. Every operation must be fully autonomous.
- Deterministic. The same input must produce the same output. Introducing an LLM into the runtime download path would make the system unpredictable and untestable.
Why existing tools fail
The dominant YouTube download tool, yt-dlp, is an exceptionally well-maintained project that has kept up with YouTube's anti-bot changes for over five years. However, yt-dlp was designed for a human-run use case: a person on a residential IP, possibly with a browser session, running the tool from their personal machine. When deployed in a cloud environment, yt-dlp encounters a wall that it was never designed to climb:
- YouTube returns
LOGIN_REQUIREDat the player API level for datacenter IPs, before any video stream URLs are returned. - The
--cookies-from-browserflag is useless because there is no browser. - OAuth login was deprecated and removed from
yt-dlpentirely. - The BGutil POT (Proof-of-Origin Token) provider, which generates attestation tokens to satisfy YouTube's bot detection, requires a Node.js runtime and a server process — setup that a cloud agent cannot easily perform.
Tech in a Minute: What is yt-dlp?
yt-dlpis a command-line tool (and Python library) that downloads videos from YouTube and hundreds of other platforms. It works by: (1) fetching the video's watch page, (2) extracting configuration data embedded in the HTML, (3) calling YouTube's internal "Innertube" API to get stream URLs, (4) deciphering any signature or throttling parameters using a built-in JavaScript interpreter, and (5) downloading the video and audio streams and merging them withffmpeg. It is the spiritual successor toyoutube-dland is actively maintained by a community of contributors who reverse-engineer YouTube's changes within hours of detection.
The problem, then, was not to build a better downloader — yt-dlp already is the best downloader. The problem was to build a system that makes yt-dlp (and other methods) work reliably in an environment they were never designed for, with zero human intervention.
2. Initial Research: Mapping the Landscape
Before writing any code, a thorough research phase was conducted to map every known method for acquiring YouTube video content from a headless environment. This research was structured around a single question: "Given a video URL and a cloud environment with no cookies and no browser, what are all the ways to get a verified video file on disk?"
The methods surveyed
The research identified eleven distinct approaches, each with different reliability profiles, dependencies, and failure modes:
Tier 1 — yt-dlp with various innertube clients. YouTube's internal API (/youtubei/v1/player) accepts a "client context" that identifies the type of application making the request. Different clients have different security requirements: the web client requires a PO Token and JavaScript signature deciphering; the android_vr and visionos clients are "JS-less" (no signature deciphering needed) and historically did not require PO Tokens; the ios client returns HLS streams; the mweb client mimics a mobile browser. yt-dlp's default client chain tries visionos → android_vr → web in sequence.
Tech in a Minute: What is an "Innertube Client"?
When you open the YouTube app on your phone, or YouTube in a browser, the app identifies itself to YouTube's servers with a "client context" — a bundle of metadata that says "I am the Android app version X" or "I am the web browser version Y." YouTube's servers use this context to decide what format to return the video in, what ads to show, and — critically — what security checks to enforce. By crafting custom client contexts (like pretending to be a VR headset app), developers can sometimes access video streams with fewer security requirements. YouTube periodically closes these loopholes, and the cat-and-mouse game continues.
Tier 2 — Direct Innertube API calls. Instead of using yt-dlp, one can POST directly to https://www.youtube.com/youtubei/v1/player with a hand-crafted client context and parse the JSON response for stream URLs. This gives maximum control but requires manually handling signature deciphering and PO tokens.
Tier 3 — Cobalt. An open-source project that provides a simple HTTP API: POST a YouTube URL, receive a redirect URL or tunneled stream. Cobalt handles PO tokens internally. Can be self-hosted or used via community-run public instances.
Tier 4 — Piped. A federated, community-run YouTube frontend with a public API. Each instance operator runs their own server (typically on a residential or VPS IP), and the API exposes stream URLs. Multiple instances can be rotated for resilience.
Tier 5 — Invidious. Another federated YouTube frontend. The key feature is the local=true parameter on its /latest_version endpoint, which tells the Invidious server to proxy the video stream through its own IP rather than redirecting to googlevideo.com.
Tier 6 — Transcript API. YouTube's transcript/subtitle endpoint runs on a separate rate-limit bucket from the video player API. It can be used as a preflight reachability check — if the transcript endpoint returns 429 (rate limited), the video is unreachable from this IP and no video download method will succeed.
Tier 7 — PO Token providers. The BGutil POT provider runs Google's BotGuard attestation in Node.js to generate Proof-of-Origin tokens. These tokens prove to YouTube that the request comes from a "real" client, which can lift the LOGIN_REQUIRED block for certain clients.
Tech in a Minute: What is a PO Token?
A PO Token (Proof-of-Origin Token) is a cryptographic attestation that YouTube requires for certain API calls. It is generated by running Google's BotGuard challenge — a piece of JavaScript that performs computations to prove the client is a real browser/app, not a bot. The token is bound to a specific video ID and expires after hours. YouTube introduced PO Tokens in 2024 to combat automated downloading. The
bgutil-ytdlp-pot-providerproject runs the BotGuard challenge in Node.js (simulating a browser environment) to generate valid tokens without needing an actual browser.
Tier 8 — Free SOCKS5 proxies. Public proxy lists contain thousands of SOCKS5 proxies run by volunteers, businesses, and (sometimes) botnets. Testing them in parallel against YouTube can find proxies whose IPs are not flagged, allowing the download to be routed through them.
Tier 9 — GitHub Actions. GitHub's CI/CD runners run on Microsoft Azure IPs, which have a different reputation profile than typical cloud provider IPs. A workflow can be triggered via API to download a video on the runner and upload it as an artifact.
Tier 10 — Cloudflare Workers. Edge functions run on Cloudflare's network, which has residential-like IP reputation. A Worker can proxy the YouTube request.
Tier 11 — Cloudflare WARP. A VPN-like service that routes traffic through Cloudflare's network, masking the origin IP. Typically installed as a Docker container or system package.
The ranking
Each method was scored on feasibility, cost, complexity, and reliability. The research concluded with a clear ranking and a recommended fallback chain. The key insight was that no single method is reliable enough to be the only method — every approach has failure modes (rate limits, IP blocks, service outages, version breakages). The only path to reliability was composition: try multiple methods in sequence, and let the system learn which ones work best in the current environment.
3. First Approach: The Naive Wrapper
What we built
The first iteration was a Python CLI tool with a clean architecture: an Orchestrator that walks a fallback chain of download methods, a Verifier that checks file integrity, a Truth Agent that learns which methods work best, and a Tester for end-to-end validation. The initial method chain had ten tiers, from transcript probing through yt-dlp with various clients to Cobalt, Piped, and Invidious.
The system was designed around four principles:
- Determinism. No LLM at runtime. The Orchestrator is a state machine that makes decisions based on file existence and method results, not probabilistic reasoning.
- Verification. A download is not "done" until the Verifier confirms it: file exists, size ≥ 1MB, magic bytes match a known video container, ffprobe succeeds, duration > 0, and (for MP4) the moov atom is intact.
- Fallback. If a method fails, the next one is tried. The chain is exhaustive.
- Learning. The Truth Agent tracks success/failure ratios and reorders the chain. After 3 consecutive failures, a method is demoted.
Why it was supposed to work
The research indicated that the android_vr and visionos innertube clients are "JS-less" — they do not require signature deciphering and historically did not require PO Tokens. The default yt-dlp chain (visionos → android_vr → web) was supposed to handle most public videos. For the remaining cases, the BGutil POT provider would generate tokens, and for everything else, the Cobalt/Piped/Invidious methods would serve as fallbacks.
The initial success
The first end-to-end test downloaded a very popular, very old video — a video so widely embedded and referenced that YouTube's caching and reputation systems treat it as permanently whitelisted. The download completed in under 5 seconds: a 243-megabyte, 213-second, AV1/Opus MP4 file. The Verifier confirmed all six integrity checks. The system worked.
Tech in a Minute: What is the moov atom?
An MP4 file is structured as a series of "boxes" (also called "atoms"). The
moovbox contains all the metadata — the video and audio codecs, the duration, the frame timing, the sample tables that tell a player where each frame is. If themoovbox is missing or truncated, the file cannot be played even if the video data is intact. Some MP4 files put themoovbox at the end of the file (to allow streaming creation), which means a truncated download leaves an unplayable file. The+faststartflag in ffmpeg moves themoovbox to the beginning, making the file playable even if truncated. Our Verifier walks the box structure to confirmmoovis present and well-formed.
The first crack
Emboldened by the initial success, a batch of ten public videos was queued for download. All ten failed. Every method in the chain returned LOGIN_REQUIRED or produced no output. The system that had worked perfectly moments before was now completely non-functional. The Truth Agent dutifully recorded the failures and demoted every method to the bottom of the chain.
The investigation began.
4. The Wall: Datacenter IP Detection
Diagnosing the block
The first step was to determine whether the failure was on our end (a bug in the code) or on YouTube's end (a block). A simple curl request to the watch page of a failing video revealed the answer:
"playabilityStatus": {"status": "LOGIN_REQUIRED", "reason": "Sign in to confirm you're not a bot"}
The watch page itself returned HTTP 200, but the embedded player configuration contained LOGIN_REQUIRED — YouTube had decided, based on our IP address alone, that we were a bot and refused to serve video streams.
An IP geolocation check confirmed the root cause: the cloud environment runs on an IP belonging to a datacenter range. YouTube maintains reputation scores for IP ranges, and datacenter IPs — especially those from certain providers and regions — are flagged as high-risk for automated abuse.
Tech in a Minute: What is IP Reputation Scoring?
Every IP address on the internet belongs to a block assigned to an ISP or cloud provider. These blocks are categorized: "residential" (assigned to home internet subscribers), "datacenter" (assigned to cloud servers), "mobile" (assigned to cell towers), "educational" (universities), etc. Anti-bot services maintain databases of these categorizations. When you make a request to a protected service, it looks up your IP's category. Datacenter IPs are treated with suspicion because real humans browse from residential IPs; a request from a datacenter IP is more likely to be a bot, a scraper, or an automated tool. This is why the same YouTube video loads fine on your home WiFi but shows "Sign in to confirm you're not a bot" when accessed from a cloud server.
The critical discovery: per-video variation
The most puzzling aspect was that one specific video downloaded successfully while ten others failed. The answer lay in YouTube's layered reputation system:
- IP-level reputation. Our datacenter IP was flagged, triggering heightened scrutiny for all requests.
- Video-level reputation. Highly popular, old, widely-embedded videos have high reputation scores and are served from cache with relaxed security. Obscure or newer videos trigger stricter checks.
- Behavioral analysis. The pattern of requests (no prior watch history, no session cookies, direct API calls) further signaled automation.
The successful video was so popular and so deeply cached that YouTube served it without applying the full anti-bot gauntlet. The ten failing videos were popular enough to be public but not popular enough to bypass the IP-level block.
What didn't work
Every direct method was tried and failed:
-
yt-dlp with
android_vrclient:LOGIN_REQUIRED -
yt-dlp with
webclient:LOGIN_REQUIRED(also requires JS signature deciphering, which needs a JS runtime) -
yt-dlp with
mwebclient + BGutil POT:LOGIN_REQUIRED(the PO token was generated but YouTube still rejected the request at the playability level) -
yt-dlp with
iosclient:LOGIN_REQUIRED -
yt-dlp with
tvandtv_downgradedclients:LOGIN_REQUIRED -
Direct Innertube API with all clients:
LOGIN_REQUIRED -
Passing session PO tokens via
extractor_args:LOGIN_REQUIRED(the token was valid but the block happens before the token is checked)
The block was not at the token level — it was at the playability level. YouTube's server decided, before even looking at our PO token, that this IP was not allowed to access this video's player API. The PO token was irrelevant.
5. Understanding the Block: Technical Deep Dive
The playability gate
YouTube's video access pipeline has multiple gates:
Request → IP Reputation Check → Playability Check → PO Token Check → Stream URL Generation → Video Delivery
↑ ↑ ↑
Datacenter IPs LOGIN_REQUIRED Token validated
flagged here returned here after playability passes
The critical insight is that the playability check happens before the PO token check. If the IP reputation + behavioral analysis determines the request is suspicious, it returns LOGIN_REQUIRED immediately. No PO token, however valid, can bypass this gate because the gate is evaluated before the token is even examined.
This is why the BGutil POT provider — which successfully generates valid Proof-of-Origin tokens — could not fix the problem. The tokens were correct, but they were being submitted to a gate that had already closed.
Tech in a Minute: What is "Playability Status"?
When you request a YouTube video, the first thing YouTube's server does is check whether you're allowed to play it. It returns a "playability status" in the response:
OK(you can play it),LOGIN_REQUIRED(sign in first),UNPLAYABLE(the video is restricted),LIVE_STREAM_OFFLINE(the live stream hasn't started), orERROR(something went wrong). For a cloud-based AI agent, the most common status isLOGIN_REQUIRED— YouTube has decided, based on your IP and request pattern, that you need to prove you're human by signing in. Since the agent has no account, this is a hard block.
Why the Cobalt and Piped approaches also failed
Cobalt and Piped are proxy services — they make the YouTube request from their own IP and relay the response. The failure here was twofold:
- Self-hosted Cobalt runs on the same blocked IP. Running Cobalt locally doesn't help because Cobalt's requests to YouTube originate from the same datacenter IP.
-
Public Cobalt/Piped instances require authentication or are themselves blocked. The Cobalt API now requires a JWT token (which requires solving a Cloudflare Turnstile challenge — a browser-only task). Piped instances are either down, behind Cloudflare browser verification, or return
SignInConfirmNotBotExceptionbecause the instance operator's IP is also flagged.
Why Invidious local=true is different
The Invidious local=true parameter is a special case. Most Invidious endpoints return a redirect to googlevideo.com (the actual video CDN). Following that redirect sends you directly to Google's servers, which apply the same IP reputation check — so you get blocked.
The local=true parameter tells the Invidious server: "Don't redirect me. Fetch the video from googlevideo.com yourself and stream it to me." The Invidious server acts as a middleman. Because the Invidious server (run by a volunteer on a residential or VPS IP) fetches from googlevideo.com, YouTube sees a request from a non-blocked IP. The video streams to the Invidious server, which re-streams it to you. Your blocked IP never touches googlevideo.com.
This was the first bypass method that actually worked.
6. Second Approach: The PO Token Era
The hypothesis
After understanding that the block was at the playability level, the next hypothesis was that a valid "session PO token" — a token bound to a visitor session rather than a specific video — might satisfy the playability check. The BGutil server's generate_once.js script can create a visitor_data + po_token pair without needing to fetch the video's watch page.
What was tried
-
Generate a session token via BGutil script mode. The script successfully produced a
contentBinding(visitor_data) andpoTokenpair. -
Pass the token to yt-dlp via
extractor_args. The token was injected asyoutube:visitor_data=<token>;po_token=mweb.player+<token>,mweb.gvs+<token>. -
Test with multiple clients (
mweb,web_embedded,web).
Why it failed
The session token approach still resulted in LOGIN_REQUIRED. The reason, discovered through verbose yt-dlp logging, is that YouTube's playability check considers IP reputation first. A request from a flagged datacenter IP is rejected at the playability level before the PO token is evaluated. The PO token is a necessary condition for video stream access, but it is not sufficient when the IP itself is blocked.
The key finding
This was the pivotal discovery of the project: the datacenter IP block cannot be defeated from the datacenter. No amount of token generation, client switching, or header spoofing can change the fact that the request originates from a flagged IP range. The only way to bypass the block is to make the request appear to come from a non-flagged IP.
Tech in a Minute: What is "Visitor Data"?
When you visit YouTube, the server assigns you a "visitor data" token — a unique identifier that tracks your session across requests. It's like a cookie, but embedded in the page's JavaScript. YouTube uses this to correlate your requests: "This visitor watched video A, then searched for B, then watched video C." For automated tools, providing a valid visitor_data token (generated fresh, not stolen) can make your requests look more like a real user's session. However, as we discovered, if your IP is flagged as a datacenter IP, even a perfect visitor_data token won't help — the block happens at the network level, not the session level.
This finding reframed the entire project. The question was no longer "how do we make yt-dlp work from this IP?" but "how do we route the request through a different IP?"
7. Third Approach: The Bypass Architecture
The reframing
With the understanding that the IP block is the root cause, the project shifted from "build a better downloader" to "build an IP-bypass system that happens to download videos." The fallback chain was redesigned to include methods that route requests through non-datacenter IPs.
Bypass method 1: Cobalt community relays
The Cobalt project maintains a directory of community-run instances. Some of these instances do not require JWT authentication and will proxy YouTube downloads through their own servers. The key instance that worked was a "relay" — a server that forwards requests to multiple backend Cobalt instances, each potentially on a different IP.
How it works: The agent POSTs the video URL to the relay. The relay forwards the request to a backend Cobalt instance (running on a residential or VPS IP). The backend fetches the video from YouTube (YouTube sees the backend's IP, not ours), and returns a "tunnel" URL. The agent then downloads from the tunnel URL, which streams the video through the relay.
Limitation: The relay IP-blocks after approximately 2 successful calls per source IP. This is a rate limit imposed by the relay operator to prevent abuse. For batch downloads, this means only 2 videos can be downloaded before the relay stops working.
Bypass method 2: Invidious local=true proxy
As described in Section 5, the local=true parameter makes the Invidious server proxy the video stream through its own IP.
How it works: A simple curl command: curl -L -o video.mp4 "https://<invidious-instance>/latest_version?id=<video_id>&itag=18&local=true". The Invidious server fetches the video from googlevideo.com and streams it back.
Limitation: Most Invidious instances have disabled the local=true parameter (because it consumes their bandwidth). Only a small number of instances still support it, and they are unreliable — going up and down throughout the day. The quality is limited to 360p (itag=18, muxed H.264/AAC MP4).
Bypass method 3: SOCKS5 proxy farm
This was the most reliable bypass method discovered. The approach:
- Fetch a list of free SOCKS5 proxies from public proxy list repositories (community-maintained lists of thousands of proxies).
- Test proxies in parallel (50 at a time) against the target video.
-
Two-phase test: Phase 1 fetches the watch page and checks that
playabilityStatusisOK(notLOGIN_REQUIRED). Phase 2 POSTs to the innertube player API and confirmsstreamingDatais present (stream URLs are returned). -
Download via the first working proxy using yt-dlp's
--proxyflag.
Why the two-phase test matters: The initial implementation tested proxies by fetching only the watch page. This found proxies that could load the watch page but failed on the player API call (because YouTube applies different reputation logic to the player API). The two-phase test ensures the proxy can complete the full extraction pipeline, not just the first step.
Tech in a Minute: What is a SOCKS5 Proxy?
A SOCKS5 proxy is a server that relays your network traffic through its own IP address. When you connect to a SOCKS5 proxy and request a website, the proxy connects to the website on your behalf, receives the response, and forwards it to you. The website sees the proxy's IP, not yours. SOCKS5 is the "no-frills" version — it doesn't modify your traffic, just forwards it. Free SOCKS5 proxies are run by volunteers, businesses, and sometimes botnets; they're unreliable (many are dead at any given time) but there are thousands of them, so finding a few working ones by testing in parallel is feasible. The trade-off: free proxies are slow, may log your traffic, and should never be used for sensitive data.
Limitation: Free proxies are ephemeral. A proxy that works now may be dead in 5 minutes. The success rate is approximately 0.5-2% of tested proxies passing both phases. Testing 200 proxies typically yields 1-4 working ones. Download speed through a free proxy is often 1-5 MB/s.
Bypass method 4: GitHub Actions remote download farm
GitHub Actions runners run on Microsoft Azure IPs, which have a different reputation profile than typical cloud provider IPs. Additionally, runners have sudo access, allowing the installation of Cloudflare WARP (which routes traffic through Cloudflare's residential-like IP pool).
How it works:
- A GitHub Actions workflow is defined in
.github/workflows/yt-download-farm.yml. - The agent triggers the workflow via the GitHub API (
POST /repos/{owner}/{repo}/actions/workflows/{id}/dispatches), passing the video URL as input. - The workflow runs on an
ubuntu-latestrunner: installs Cloudflare WARP, installs yt-dlp + BGutil, downloads the video, and uploads it as a workflow artifact. - The agent polls the workflow status, then downloads the artifact via the GitHub API.
Limitation: GitHub Actions have a 6-hour job timeout and a 10 GB artifact size limit. The workflow takes 2-5 minutes to complete (runner startup + WARP install + download + artifact upload). The artifact is retained for 7 days. Rate limits: 15 workflow dispatches per minute, 100 artifact downloads per hour.
Tech in a Minute: What is Cloudflare WARP?
Cloudflare WARP is a free VPN-like service provided by Cloudflare. When installed on a machine, it routes all outgoing traffic through Cloudflare's global network. From the perspective of the website you're visiting, your request comes from a Cloudflare IP (which has good reputation because Cloudflare is a legitimate CDN provider) rather than your actual cloud provider's IP. WARP is not a full VPN — it doesn't hide your traffic from Cloudflare — but it effectively changes your IP reputation. For our use case, installing WARP on a GitHub Actions runner makes YouTube see a Cloudflare IP instead of an Azure IP, which is often enough to bypass the datacenter block.
8. The Breakthrough: Compositional Bypass
The key realization
No single bypass method is reliable:
- Cobalt relays rate-limit after 2 calls.
- Invidious instances go up and down.
- SOCKS5 proxies die within minutes.
- GitHub Actions has rate limits and latency.
But together, they form a resilient system. If the Cobalt relay is rate-limited, the Invidious proxy might be up. If Invidious is down, the SOCKS5 farm can find a working proxy. If all else fails, the GitHub Actions farm provides a guaranteed (if slow) path.
The test that proved it
A batch of ten public videos — all of which had been completely blocked (0/10 success rate) in the first approach — was re-tested with the bypass architecture. The results:
- 9 out of 10 videos downloaded successfully (605 MB total data acquired).
- 8 videos were downloaded via the SOCKS5 proxy farm method.
- 1 video was downloaded via the Invidious
local=truemethod. - 1 video failed all methods (the specific video may have additional restrictions or the proxy pool was exhausted for that hour).
Why composition works
The key insight is that the bypass methods fail independently. The Cobalt relay's rate limit has nothing to do with the Invidious instance's uptime, which has nothing to do with whether a SOCKS5 proxy is alive. By composing them into a fallback chain, the system achieves a success rate that no individual method can reach.
This is a classic application of the diversity principle from reliability engineering: a system composed of diverse components with independent failure modes is more reliable than any single component. Mathematically, if method A has a 30% success rate and method B has a 40% success rate, and their failures are independent, the combined success rate is 1 - (0.7 × 0.6) = 58% — higher than either alone.
Tech in a Minute: What is a "Fallback Chain"?
A fallback chain is a list of methods to try in order. You try the first method; if it fails, you try the second; if that fails, you try the third; and so on until one succeeds or all fail. It's the same concept as having a spare tire and a can of fix-a-flat — if the spare is flat, you use the fix-a-flat; if both fail, you call a tow truck. The key design decision is the order: fast and reliable methods go first, slow and unreliable methods go last. In our system, the direct yt-dlp methods (which fail fast when blocked) are tried first, followed by the bypass methods (which are slower but can actually work).
9. Final Architecture: The 13-Method Fallback Chain
The final system implements a 13-tier fallback chain, managed by an Orchestrator that walks the chain, hands each result to a Verifier, and records observations for the Truth Agent to learn from.
The complete chain
| Tier | Method | Strategy | Bypasses IP block? |
|---|---|---|---|
| 0 | Transcript probe | Preflight reachability check (separate rate-limit bucket) | N/A (fast fail) |
| 1 | yt-dlp default |
android_vr client + BGutil PO token |
No (needs whitelisted video) |
| 2 | yt-dlp JS-less |
visionos, android_vr clients + BGutil |
No |
| 3 | yt-dlp IOS | IOS client (HLS streams) | Sometimes |
| 4 | yt-dlp single file | Pre-muxed MP4 (no ffmpeg merge) | No |
| 5 | yt-dlp audio only | Audio-only salvage (last yt-dlp tier) | No |
| 6 | Innertube direct | Hand-rolled POST to /youtubei/v1/player
|
No |
| 7 | Cobalt (self-hosted) | Local Cobalt sidecar | Yes (if Cobalt has good IP) |
| 8 | Cobalt community | Public Cobalt relay instances | Yes |
| 9 | Piped | Piped API with 4-instance rotation | Varies |
| 10 | Invidious |
local=true proxy (instance streams through its IP) |
Yes |
| 11 | SOCKS5 farm | Free SOCKS5 proxy discovery (two-phase test) | Yes |
| 12 | GitHub Actions farm | Remote download on GitHub runners + WARP | Yes |
The agents
The system is built around four in-process agents (not LLM agents — deterministic Python modules with specific roles):
The Orchestrator receives a video URL, resolves it to an ID, asks the Truth Agent for the ranked method list, and walks the chain. For each method, it creates a fresh temp directory, runs the method with a timeout, and hands any successful result to the Verifier. If the Verifier passes, the file is atomically moved to the output directory. If the Verifier fails, the method's result is treated as a failure and the next method is tried.
The Truth Agent owns two state files: truth.json (the ranked method list with success/failure counters) and observations.jsonl (an append-only audit log of every method attempt). After every method attempt, the Orchestrator records an observation. After 3 consecutive failures of a method, the Truth Agent demotes it (rank += 1, capped at the bottom). After a success, the method may be promoted (rank -= 1, floored at the top). This means the system learns which methods work best in the current environment and reorders the chain over time.
The Verifier runs 6 layered checks on any file a method claims to have downloaded: (1) file exists and size ≥ 1 MB, (2) magic bytes match a known video container, (3) ffprobe succeeds, (4) duration > 0, (5) at least one video or audio stream is present, (6) for MP4 files, the moov atom is present and well-formed. These checks catch truncated downloads, HTML error pages saved as .mp4, and corrupted files.
The Tester runs the full pipeline against a corpus of known-stable public videos, producing a markdown report of which methods succeeded for each video. This serves as both a health check and a regression test.
The auto-bootstrap
A critical design decision was that the system should require zero manual setup. If an AI agent runs ytagent download <url> on a fresh machine, the system should auto-bootstrap everything it needs:
- If the BGutil POT provider is not installed, it is cloned from GitHub, npm dependencies are installed, TypeScript is compiled, and the HTTP server is started — all automatically, in about 15 seconds.
- If the BGutil server is not running, it is started.
- The download then proceeds through the 13-method chain.
This is implemented in the bootstrap.py module, which is called transparently by the yt-dlp methods when they detect that BGutil is not available.
The agent instructions
The system includes a ytagent agent-instructions command that prints a complete 10-step guide for AI agents. This covers installation, setup, downloading, JSON parsing, bypass methods, self-testing, truth inspection, and batch downloads. An AI agent reading only this output can use the system end-to-end without any human help.
This was validated in a final test: an AI agent following the instructions downloaded a previously-blocked video with zero manual human intervention. The socks5_farm method found a working proxy in approximately 100 seconds and downloaded a 69-megabyte, 1080p video.
10. Key Learnings and Takeaways
Learning 1: The IP is the identity
In modern web infrastructure, your IP address is your identity. More precisely, the reputation category of your IP (residential vs. datacenter vs. mobile) determines what you can access. Cookies, tokens, and session identifiers are secondary — they are checked only after the IP reputation gate. For AI agents operating from cloud environments, this means that no amount of authentication or token generation can overcome a flagged IP.
Takeaway: When building systems that interact with anti-bot-protected services, the first question to ask is not "how do I authenticate?" but "what is the reputation of my IP, and how do I route through a better one?"
Learning 2: Composition beats optimization
The most reliable system was not the one with the best single method — it was the one with the most diverse set of methods. Each bypass method had a 30-60% success rate individually, but composed together, the chain achieved a 90% success rate. This is because the methods fail independently: the Cobalt relay's rate limit is unrelated to the Invidious instance's uptime, which is unrelated to whether a SOCKS5 proxy is alive.
Takeaway: In adversarial environments (anti-bot systems, rate limits, service outages), invest in breadth (many methods) over depth (one perfect method). The probability that all methods fail simultaneously is exponentially lower than the probability that any single method fails.
Learning 3: Verification is non-negotiable
In the testing phase, it became clear that download methods frequently produce "successful" results that are actually broken: HTML error pages saved as .mp4, truncated downloads missing the moov atom, 0-byte files from connection resets. Without a rigorous Verifier, these broken files would be silently passed to downstream agents, causing confusing failures later.
Takeaway: Every automated pipeline needs a verification layer that checks the output is actually what it claims to be. For video files, this means magic bytes, ffprobe, duration, and moov atom checks. Trust no method's self-report; verify independently.
Learning 4: Free infrastructure is fragile but usable
Every free bypass method (Cobalt relays, Invidious instances, SOCKS5 proxies) was unreliable when relied upon individually. But because they were free, they could be composed in parallel. A paid residential proxy service would have been more reliable, but the goal was to build a system that works with zero cost.
Takeaway: Free infrastructure is a viable component of a production system if (and only if) you compose enough of it to achieve the reliability you need. The cost of free infrastructure is engineering complexity (more methods, more failure handling), not money.
Learning 5: Determinism enables trust
The system was designed from the start to be deterministic: no LLM at runtime, no probabilistic decisions, no "ask the AI what to do next." Every decision is made by a state machine with clear rules. This made the system testable, debuggable, and trustworthy. When a download failed, the audit trail in observations.jsonl showed exactly which methods were tried and why they failed.
Takeaway: AI agents that need to interact with external systems should use deterministic orchestration, not LLM-driven decision-making. The LLM should be used at design time (to research, plan, and write code), not at runtime (to decide what to do next). Determinism is what makes the system auditable and reliable.
Learning 6: The two-phase test pattern
The SOCKS5 proxy farm method initially failed because it tested proxies by fetching only the watch page. Many proxies could load the watch page but failed on the player API call. The fix was a two-phase test: check the watch page first, then check the player API. This pattern — test the full pipeline, not just the first step — generalized to other areas of the system.
Takeaway: When testing whether a resource (proxy, API, service) is "working," test the entire operation you need it to perform, not just the first step. A proxy that can load a web page may not be able to stream a video. An API key that works for metadata may not work for downloads. Test the actual use case.
Learning 7: Documentation is part of the system
The ytagent agent-instructions command was not an afterthought — it was a first-class feature. An AI agent that can read the instructions and use the system without human help is a more valuable system than one that requires a human to read the README and explain it. The instructions include exact commands, expected output, and failure troubleshooting for each step.
Takeaway: For systems designed to be used by AI agents, the documentation should be machine-readable and executable. Don't just describe what the system does — provide a step-by-step guide that an agent can follow mechanically.
11. Future Implications
The splintering of the internet
The datacenter IP block is not unique to YouTube. It is a pattern being adopted by an increasing number of services: social media platforms, news sites, e-commerce, and even some APIs now apply differential access based on IP reputation. The internet is splintering into two tiers: a "residential tier" where content is freely accessible, and a "datacenter tier" where access is gated by CAPTCHAs, login requirements, and behavioral challenges.
For AI agents — which necessarily operate from datacenter IPs — this splintering represents a fundamental access problem. An agent that cannot route its traffic through residential IPs will find itself increasingly locked out of the web.
The rise of proxy-as-a-service
As datacenter IP blocks become more common, a market is emerging for "residential proxy" services that sell access to residential IP pools. These services (Bright Data, Smartproxy, etc.) charge per GB of traffic routed through their pools. For AI agent operators, this will become a line item in the infrastructure budget, much like compute and storage.
The alternative — composing free bypass methods, as this project does — is viable for low-volume use but does not scale. A production system processing thousands of videos per day would need a paid proxy service or a self-hosted residential IP pool (e.g., Oracle Cloud Free Tier VMs with WARP installed).
The attestation arms race
YouTube's PO Token system is part of a broader trend toward attestation-based access control. Instead of (or in addition to) checking your IP, services increasingly require proof that your client is "real" — that it ran a specific piece of JavaScript, performed a specific computation, or holds a specific hardware attestation. Apple's App Attest, Google's Play Integrity, and Cloudflare's Turnstile are all examples of this trend.
Tech in a Minute: What is "Attestation"?
Attestation is the process of proving that a piece of software is running in a trusted environment. When a website says "prove you're a real browser," it's asking for attestation. The website sends a challenge (usually a piece of JavaScript that performs computations), and your browser returns the result as "proof" that it executed the challenge. The challenge is designed to be hard to fake — a bot would need to run a full browser engine to solve it. Cloudflare Turnstile, Google reCAPTCHA, and YouTube's BotGuard are all attestation systems. They differ in how hard the challenge is and what they're trying to prove (real browser? real mobile device? not a bot?).
For AI agents, attestation is a double-edged sword. On one hand, projects like BGutil can run the attestation challenge in a simulated browser environment (Node.js with JSDOM), generating valid tokens without a real browser. On the other hand, the attestation systems are evolving to detect these simulations (checking for browser-specific APIs, timing characteristics, rendering quirks). The arms race will continue, with each side getting more sophisticated.
Implications for AI agent design
This project suggests several principles for designing AI agents that interact with web services:
Assume IP blocks. Any agent running in a cloud environment should have a strategy for dealing with IP-based access restrictions. This means having a pool of proxies or fallback methods, not relying on a single access path.
Build for diversity. Don't depend on a single service, API, or method. The web is adversarial — services change their APIs, add new blocks, and rate-limit aggressively. A diverse set of fallback methods is the only defense.
Separate intelligence from access. The LLM should be used for reasoning about what to download, not for the mechanics of downloading. The download itself should be handled by a deterministic, tested, verified pipeline. This separation allows the intelligence to be upgraded without breaking the access layer.
Audit everything. Every request, every method attempt, every failure should be logged. When a download fails, the audit trail should show exactly what was tried and why it failed. This is essential for debugging and for building trust in the system.
Design for the agent, not the human. The end user of an AI agent's download capability is another AI agent (or a pipeline), not a human. The output should be machine-readable (JSON), the instructions should be machine-followable (step-by-step), and the failure modes should be machine-recoverable (try the next method).
The broader security picture
The fact that a cloud-based AI agent cannot download a public YouTube video without building a 13-method bypass chain reveals something important about the state of the internet: the security measures designed to stop bots are also stopping legitimate automated access. The same IP reputation systems that block scrapers also block AI agents. The same attestation challenges that stop credential stuffing also stop automated research.
This creates a paradox: as AI agents become more capable and more widely deployed, they will increasingly need to access web content that is protected by anti-bot systems. But those systems were designed to stop automated access. The result is a cat-and-mouse game where AI agents develop increasingly sophisticated bypass methods, and anti-bot systems develop increasingly sophisticated detection methods.
The long-term resolution is likely a shift from IP-based access control to identity-based access control. Instead of blocking datacenter IPs, services may issue API keys or attestation tokens to verified AI agents, allowing them to access content programmatically without the overhead of bypassing anti-bot systems. YouTube's official Data API v3 is a step in this direction (though it does not allow video downloads). Until such APIs become comprehensive, the bypass architecture described in this document will remain necessary.
Conclusion
This project began as a simple wrapper and evolved into a study of modern web access control. The journey from "just use yt-dlp" to "compose 13 methods including SOCKS5 proxy farms and GitHub Actions runners" reflects the reality of building automated systems for today's web.
The key insight — that the datacenter IP is the root cause, not the download method — reframed the entire problem. Once the block was understood as an IP-level gate that runs before any token or authentication is checked, the solution became clear: route through a different IP. The implementation of that solution required composing multiple bypass methods into a resilient fallback chain, because no single bypass is reliable enough to stand alone.
The resulting system — a 13-method, self-healing, auto-bootstrapping download pipeline with a learning Truth Agent and rigorous file verification — represents a practical answer to a problem that every cloud-based AI agent will face: how do you access a web that doesn't want you?
The answer, as it turns out, is not to break the security measures. It is to find enough legitimate access paths that at least one of them works, and to compose them in a way that makes failure the exception rather than the rule.
This research document describes the architecture and findings of a real project. All technical details, failure modes, and success rates are based on actual testing conducted during the project's development. The system is open-source and available for review, modification, and extension.
Who is Ansari Bilal?
Who built it. This project — ytagent — was built by Ansari Bilal, a self-taught app builder from India.
Repository: https://github.com/Bilal140202/ytagent
Ansari Bilal is a self-taught app builder. What started as late-night tinkering turned into a second life — shipping real products, building open-source tools, and solving problems through code. From privacy-first desktop apps built with Tauri & Rust to developer tools used by thousands, every project is fueled by curiosity, not credentials. No CS degree, no bootcamp — just relentless building and a GitHub full of proof.
The mission is simple: build things that are useful, open, and well-crafted — whether that's an AI-powered agent tool, a cross-platform note-taking app, or a curated directory of 30,000+ developer resources.
Other things he has shipped:
- The Lord of the Skills — 18,142+ AI agent skills across 14 frameworks, sorted into 10 LOTR-themed kingdoms.
- Linuxify — run Linux developer CLIs and AI coding agents on Android, no root required.
-
snip — project-scoped command snippets in a committable
.snipsfile, with fuzzy and natural-language matching. - DevAtlas — a searchable index of 30,000+ developer tools, refreshed daily.
- TalkDrive — SoulX FlashHead 1.3B talking-head generation on a free Colab T4.
What ytagent was built with
ytagent is a Python system that composes existing, legitimate tooling into a resilient chain rather than replacing any of it:
- Python 3 — orchestration, method registry, deterministic fallback controller
- yt-dlp — the core extractor, driven through multiple client and transport configurations
- ffmpeg — stream merging and post-processing
- Proof-of-Origin token providers — attestation where the player API demands it
- Public mirror / proxy front-ends — alternate access paths when the direct player API refuses a datacenter IP
- No LLM at runtime — the download path stays deterministic and testable
Source: https://github.com/Bilal140202/ytagent
More work: ansaribilal.com/projects · GitHub · Hugging Face
Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.
Top comments (0)