DEV Community: Ansari Mohammed Bilal

OSINT with Sploitas: What I Found About My Own Digital Footprint

Ansari Mohammed Bilal — Sat, 08 Aug 2026 10:02:18 +0000

Last updated: September 10, 2025 \u00b7 6-minute read

I always assumed my digital footprint was small. I don\u2019t overshare on social media, I use a password manager, and I\u2019ve been privacy-conscious for years. But last week I decided to stop assuming and start verifying \u2014 by running the same OSINT tools security researchers use on targets against myself.

The results were humbling. 14 tools, 30 minutes, and I found 9 email addresses I\u2019d forgotten about, 17 abandoned accounts, and my personal data on at least 6 data broker sites.

The Starting Point: Where I Looked

I started with the Cyber Detective\u2019s OSINT tools collection on GitHub \u2014 a repo that catalogs over 1,000 services for open-source intelligence gathering. The maintainer describes it as a \u201ccollection of different services, techniques, tricks and notes about OSINT.\u201d It\u2019s the kind of resource that makes you realize how much of your life is indexable.

I also pulled evaluation criteria from Wiz\u2019s \u201c9 Top OSINT Tools & How to Evaluate Them\u201d article, which breaks down what actually makes an OSINT tool useful versus just noisy. Their framework: data breadth, freshness, and actionability.

For the actual scanning, I used Sploitas and a handful of tools from the Cyber Detective collection \u2014 email lookup services, username enumeration tools, and data broker aggregators. The point wasn\u2019t to become a security researcher. The point was to see what a stranger with my email address could find in 30 minutes.

What I Found

1. Emails I Forgot Existed

Running my primary email through OSINT email lookup tools surfaced 9 addresses tied to my identity that I hadn\u2019t used in years. A university email. A throwaway from a 2019 hackathon. Three variations of my name with different providers. Two work emails from previous jobs I\u2019d assumed were deactivated.

2. Abandoned Accounts

The username enumeration tools found 17 accounts across platforms I either barely used or completely forgot about. Old GitHub forks from 2021. A Medium account with one draft. A Fiverr profile. A Stack Overflow account with two answers from college. A Dribbble I signed up for and never touched.

Each one is a potential vector. An abandoned account with a reused password is a foothold waiting to be exploited.

3. Data Brokers

This was the uncomfortable part. Searching my name and email across data broker aggregator tools showed hits on at least 6 different brokers \u2014 sites that buy, compile, and resell personal information. Name, address history, phone number, associated emails, age. All publicly purchasable for anywhere from $2 to $30.

A thread on r/cybersecurity was blunt about the reality: paying services like DeleteMe or Incogni only handle the request loop. They don\u2019t prevent your data from reappearing. One commenter put it well \u2014 \u201cit does not remove your info from the internet, all it does is request that the data brokers delete the copy they have.\u201d

The Cleanup Process

Here\u2019s exactly what I did, in order:

Step 1: Delete abandoned accounts. I went through all 17 accounts and either deleted them or updated the password to a unique 32-character string and removed all personal info. The latter is faster for platforms that make deletion intentionally difficult.

Step 2: Opt out of data brokers. I submitted removal requests to all 6 brokers manually. Most have an opt-out form buried in their footer. A few required email verification or a scan of my ID. The entire process took about 90 minutes spread across two evenings.

Step 3: Consolidate email addresses. I set up forwarding on the forgotten emails to my primary address, then deactivated the ones I could. For the ones tied to accounts I still need (university alumni, bank), I updated the recovery info.

Step 4: Run it again. A week later, I re-ran the same tools. Two of the six data brokers had removed my listing. The rest were \u201cprocessing.\u201d Three abandoned accounts I thought I\u2019d deleted were still showing up in username searches \u2014 the platforms had a 30-day grace period.

What This Means for How I Build

This experiment reinforced something I already believed but now feel more urgently about: privacy isn\u2019t a feature, it\u2019s an architecture decision. Every app I build starts with the assumption that user data stays on the user\u2019s device. That\u2019s why I built PrivacyThink to run LLMs locally, and why every tool in the Lab is designed to minimize data leaving your machine.

If a random person can reconstruct 80% of my digital identity using free tools in 30 minutes, then the tools I build have a responsibility to not add to that pile.

TL;DR

OSINT tools found 9 forgotten emails, 17 abandoned accounts, and 6 data broker listings tied to my identity
The Cyber Detective\u2019s GitHub collection (1,000+ services) and Sploitas were my starting points
Cleanup: delete abandoned accounts, opt out of data brokers manually, consolidate emails
Data broker removal is a recurring process, not a one-time fix
This directly connects to why I build privacy-first tools like PrivacyThink and keep experiments local-first

The Projects section has the tools I\u2019ve built with privacy in mind, and the Lab has the ones still in progress. If you haven\u2019t checked your own footprint, do it this week \u2014 not next month.

Not affiliated with Sploitas, OSINT Industries, Epieos, or any data broker mentioned. Tools used: Sploitas, Epieos, OSINT Industries, Cyber Detective\u2019s OSINT collection on GitHub.

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

Superhero UI Review: The Boldest React Component Library of 2026?

Ansari Mohammed Bilal — Fri, 07 Aug 2026 10:24:33 +0000

7-minute read · October 2026

Every React component library in 2026 looks the same: neutral gray, tiny radius, muted primary. Superhero UI shows up wearing a cape.

I rebuilt a real internal dashboard with it over a weekend. Here's what I found.

What Superhero UI Actually Is

A React + Tailwind component set with a distinctly comic-book aesthetic — thick borders, halftone patterns, bold shadows, dramatic hover states. Built on Radix primitives (like shadcn), themed as if Marvel designed a design system.

The Good

Instant personality. A landing page built with default Superhero components looks like someone made it, not like a Vercel template.
Radix underneath = accessibility comes free. Focus rings, keyboard nav, ARIA — all handled.
Zero-install feel. Copy-paste like shadcn. You own the code.

The Painful

Not for enterprise dashboards. Halftone patterns behind a data table make my product manager cry.
Print / PDF export looks weird. Bold shadows don't rasterize cleanly.
Small community. ~1.2k stars vs shadcn's 70k+. Fewer recipes on Stack Overflow.

Head to Head

Library	Vibe	Best for
shadcn/ui	Neutral, minimal	SaaS, admin, "safe" B2B
Aceternity	Cinematic, animated	Marketing pages, hero moments
Magic UI	Playful, motion-heavy	Landing pages that need to feel alive
Superhero	Bold, pop-art	Personal sites, indie SaaS, portfolios

When I'd Reach For It

Portfolio sites where you want to stand out.
Indie SaaS marketing pages for a fun product category (games, kids, creators).
Hackathon demos where "distinctive" > "polished".

When I Wouldn't

Anything B2B fintech, health, government. The visual language is wrong.
Data-heavy apps. Chart libraries look strange next to halftone.

Combine, Don't Commit

The pattern that worked best in my rebuild: Superhero for marketing pages, shadcn for the app. Same Tailwind theme tokens, two visual registers. Users get personality on the way in and clarity once they're inside.

Bottom Line

Superhero UI is the honest answer to "why does every React app look the same?" It won't win the enterprise, but for personal work and indie launches, it's the most fun I've had installing a component library in 2026.

Grab it from the docs, or read my take on shadcn vs Aceternity vs Magic UI for the full field guide.

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

AI Testing That Actually Works: TestSprite Setup (After Trying Everything Else)

Ansari Mohammed Bilal — Fri, 07 Aug 2026 10:24:28 +0000

Last updated: August 27, 2025 · 8-minute read

I wrote 47 Cypress tests for a SaaS dashboard in March. By June, 12 of them were broken — not because the app was buggy, but because a frontend dependency updated and three CSS class names changed. I spent a Saturday fixing selectors.

That's the traditional testing trap. You don't maintain tests — you maintain selectors. And in the age of AI-generated code where your component structure changes every session, selector-based testing is a losing game.

I tried every AI testing tool I could find. Most of them were exactly what a Reddit thread on r/QualityAssurance warned about: "I've tried a few of the newer AI tools, they can help with quick test generation, but often feel overhyped and not very cost effective."

TestSprite is the exception. Here's why, and how to set it up.

The Problem With Traditional Testing in 2025

The top thread on r/QualityAssurance — "What AI QA testing tools/services are you actually using in 2025?" — had a telling response distribution. Most comments fell into two camps: "I use Playwright/Cypress with some AI-assisted test generation" or "AI testing tools aren't ready for production yet."

A related thread titled "AI in QA/Automation — What tools are you actually using and for what?" was even more blunt. The consensus: AI can generate test code, but the tools that claim to autonomously test your app end-to-end are still unreliable.

The core issue is that vibe-coded apps — apps built primarily through AI coding agents — have a specific failure mode that traditional testing doesn't catch well. The AI doesn't write buggy code in the traditional sense. It writes code that looks correct but makes architectural assumptions that don't hold. Intellyx's analysis of TestSprite called this out directly: it "protects against vibecoding and GenAI architectural hallucinations by orchestrating agents that generate and run end-to-end tests."

That's the gap TestSprite targets — not "does this button work" but "does this AI-generated architecture actually hold together."

What TestSprite Actually Does

TestSprite is an AI testing agent that takes a different approach from traditional test frameworks. Instead of writing selectors and assertions, you give it your app URL and a prompt. It:

Navigates your app and builds a structural understanding of the pages
Generates a test plan based on what it finds
Writes and executes tests — autonomously
Debugs failures and retries
Reports with screenshots, logs, and a summary

It integrates as an MCP server in Cursor and other AI IDEs, which means your coding agent can trigger tests without leaving the editor. The YouTube video "TestSprite MCP Server + Cursor + Copilot = One Prompt to Test" demonstrates the workflow — you literally type "test the checkout flow" and it goes.

Setup Walkthrough

Step 1: Create a TestSprite Account

Go to testsprite.com and sign up. They have a free tier with limited runs — enough to evaluate whether it works for your use case.

Step 2: Add the MCP Server to Cursor

Open Cursor Settings (Cmd+Shift+J on Mac), navigate to Tools & Integration, click "Add custom MCP," and add this configuration:

{
  "mcpServers": {
    "testsprite": {
      "command": "npx",
      "args": ["-y", "testsprite-mcp"]
    }
  }
}

Restart Cursor. The TestSprite tools should now appear in your MCP tool panel.

Step 3: Point It at Your App

In a Cursor chat, type something like:

"Run an end-to-end test on http://localhost:5173. Test the login flow, dashboard load, and task creation."

TestSprite will navigate your app, build a test plan, execute it, and return results — all within the chat. No test files to maintain. No selectors to update.

Step 4: Review Results

TestSprite returns a report with:

Pass/fail status for each test case
Screenshots at failure points
Console errors and network logs
A natural language summary of what broke and why

If something fails, you can ask it to debug: "The login test failed. Fix the issue and re-run." It'll investigate, propose a fix, and verify.

TestSprite vs Cypress vs Playwright vs Raw AI

Cypress: Selector-based. You write tests that depend on DOM structure. When your AI agent changes a div to a section, your test breaks. Good for stable apps with slow-moving frontends. Bad for anything AI-generated.

Playwright: Better selectors (can use text, role, aria labels instead of CSS classes), better cross-browser support, and Microsoft's MCP server means you can now use it inside Claude Code directly. The command claude mcp add playwright npx @playwright/mcp@latest adds browser automation to your agent. But you still write and maintain test code.

Raw AI (just asking Claude to test): I've tried this. "Here's my codebase, write tests." It generates Jest or Playwright scripts that are reasonable but brittle. Same maintenance problem as Cypress — the tests are tied to implementation details.

TestSprite: Doesn't write persistent test files. It generates tests on-the-fly based on the current state of your app. This means no test rot. The tradeoff is that it's slower per-run (it has to rediscover your app each time) and you don't get CI/CD integration in the traditional sense.

When AI Testing Makes Sense (And When It Doesn't)

Use TestSprite when:

You're building with AI agents and your UI changes frequently
You need quick validation that a feature works end-to-end before committing
You're prototyping and don't want to invest in a test suite yet
You want to catch architectural hallucinations, not just unit-level bugs

Stick with Playwright/Cypress when:

You have a stable, shipping product with a test suite you already maintain
You need CI/CD pipelines with fast, deterministic test runs
Your team writes tests as part of the development process
Regulatory compliance requires auditable, version-controlled test files

Don't use any AI testing tool when:

Your app has no users yet and you're still figuring out the product
The cost of the tool exceeds the cost of the bugs it catches
Your testing needs are simple enough that a 20-line Playwright script covers them

TL;DR

Traditional test frameworks (Cypress, Playwright) break when AI agents frequently restructure your code.
Most AI testing tools are overhyped — the r/QualityAssurance consensus is clear on this.
TestSprite works because it doesn't maintain persistent test files — it generates and runs tests on-the-fly against your current app state.
Setup takes under 5 minutes via MCP in Cursor or Claude Code.
It's not a replacement for Playwright in a CI/CD pipeline — it's a different category of tool for a different stage of development.
Best use case: validating AI-generated code before you ship it to users.

This is part of a broader toolkit I've been assembling for building with AI agents. The Google Antigravity 2.0 writeup covers the orchestration layer, and the Builds section has the projects where these testing workflows run in production.

Not affiliated with TestSprite. Setup tested with Cursor and Claude Code in August 2025. Tools used: TestSprite (free tier), Playwright MCP, Claude Code.

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

The Datacenter IP Block: Building a Self-Healing YouTube Download System for AI Agents

Ansari Mohammed Bilal — Wed, 05 Aug 2026 11:36:02 +0000

A research document on the architecture, failures, and breakthroughs of building a cloud-native video acquisition system that defeats modern anti-bot infrastructure.

Abstract

This document chronicles the design, implementation, and iterative breakthroughs of a YouTube video acquisition system built specifically for cloud-based AI agents. The project began with a straightforward goal — enable an AI agent running in a headless cloud environment to download public YouTube videos without a browser, without cookies, and without human intervention. What began as a simple wrapper around existing tooling evolved into a multi-layered bypass architecture that defeats one of the most sophisticated anti-bot systems on the internet: YouTube's datacenter IP detection and Proof-of-Origin token enforcement.

The research reveals a critical truth about modern web infrastructure: the internet is splitting into two tiers. Residential IPs can access content freely. Datacenter IPs are increasingly walled off behind behavioral analysis, attestation challenges, and IP reputation scoring. For AI agents that operate exclusively from cloud environments, this split represents an existential threat to their ability to interact with the web. This document describes how that threat was identified, analyzed, and ultimately routed around — not by breaking the security measures, but by composing multiple legitimate access paths into a resilient fallback chain.

The Problem: Why AI Agents Can't Download Videos
Initial Research: Mapping the Landscape
First Approach: The Naive Wrapper
The Wall: Datacenter IP Detection
Understanding the Block: Technical Deep Dive
Second Approach: The PO Token Era
Third Approach: The Bypass Architecture
The Breakthrough: Compositional Bypass
Final Architecture: The 13-Method Fallback Chain
Key Learnings and Takeaways
Future Implications

1. The Problem: Why AI Agents Can't Download Videos

The use case

AI agents are increasingly deployed in cloud environments to perform research, content analysis, and data gathering. A common requirement is the ability to acquire video content from public sources for processing — transcription, summarization, frame extraction, or archival. The dominant video platform, YouTube, presents a unique challenge because it has evolved one of the most aggressive anti-automation systems on the public web.

The constraints under which a cloud-based AI agent operates are fundamentally different from those of a human user:

No browser. The agent runs in a headless container or VM. There is no Chrome, no Firefox, no cookie jar, no session persistence across requests in the way a browser provides.
No credentials. The agent has no Google account, no OAuth token, no logged-in session. It cannot "sign in to confirm you're not a bot" because it has nothing to sign in with.
Datacenter IP. The agent's network traffic originates from a cloud provider's IP range (AWS, GCP, Azure, Alibaba, etc.). These ranges are publicly known and aggressively flagged by anti-bot systems.
Non-interactive. The agent cannot solve CAPTCHAs, click "I'm not a robot," or wait for a human to solve a browser challenge. Every operation must be fully autonomous.
Deterministic. The same input must produce the same output. Introducing an LLM into the runtime download path would make the system unpredictable and untestable.

Why existing tools fail

The dominant YouTube download tool, yt-dlp, is an exceptionally well-maintained project that has kept up with YouTube's anti-bot changes for over five years. However, yt-dlp was designed for a human-run use case: a person on a residential IP, possibly with a browser session, running the tool from their personal machine. When deployed in a cloud environment, yt-dlp encounters a wall that it was never designed to climb:

YouTube returns LOGIN_REQUIRED at the player API level for datacenter IPs, before any video stream URLs are returned.
The --cookies-from-browser flag is useless because there is no browser.
OAuth login was deprecated and removed from yt-dlp entirely.
The BGutil POT (Proof-of-Origin Token) provider, which generates attestation tokens to satisfy YouTube's bot detection, requires a Node.js runtime and a server process — setup that a cloud agent cannot easily perform.

Tech in a Minute: What is yt-dlp?

yt-dlp is a command-line tool (and Python library) that downloads videos from YouTube and hundreds of other platforms. It works by: (1) fetching the video's watch page, (2) extracting configuration data embedded in the HTML, (3) calling YouTube's internal "Innertube" API to get stream URLs, (4) deciphering any signature or throttling parameters using a built-in JavaScript interpreter, and (5) downloading the video and audio streams and merging them with ffmpeg. It is the spiritual successor to youtube-dl and is actively maintained by a community of contributors who reverse-engineer YouTube's changes within hours of detection.

The problem, then, was not to build a better downloader — yt-dlp already is the best downloader. The problem was to build a system that makes yt-dlp (and other methods) work reliably in an environment they were never designed for, with zero human intervention.

2. Initial Research: Mapping the Landscape

Before writing any code, a thorough research phase was conducted to map every known method for acquiring YouTube video content from a headless environment. This research was structured around a single question: "Given a video URL and a cloud environment with no cookies and no browser, what are all the ways to get a verified video file on disk?"

The methods surveyed

The research identified eleven distinct approaches, each with different reliability profiles, dependencies, and failure modes:

Tier 1 — yt-dlp with various innertube clients. YouTube's internal API (/youtubei/v1/player) accepts a "client context" that identifies the type of application making the request. Different clients have different security requirements: the web client requires a PO Token and JavaScript signature deciphering; the android_vr and visionos clients are "JS-less" (no signature deciphering needed) and historically did not require PO Tokens; the ios client returns HLS streams; the mweb client mimics a mobile browser. yt-dlp's default client chain tries visionos → android_vr → web in sequence.

Tech in a Minute: What is an "Innertube Client"?

When you open the YouTube app on your phone, or YouTube in a browser, the app identifies itself to YouTube's servers with a "client context" — a bundle of metadata that says "I am the Android app version X" or "I am the web browser version Y." YouTube's servers use this context to decide what format to return the video in, what ads to show, and — critically — what security checks to enforce. By crafting custom client contexts (like pretending to be a VR headset app), developers can sometimes access video streams with fewer security requirements. YouTube periodically closes these loopholes, and the cat-and-mouse game continues.

Tier 2 — Direct Innertube API calls. Instead of using yt-dlp, one can POST directly to https://www.youtube.com/youtubei/v1/player with a hand-crafted client context and parse the JSON response for stream URLs. This gives maximum control but requires manually handling signature deciphering and PO tokens.

Tier 3 — Cobalt. An open-source project that provides a simple HTTP API: POST a YouTube URL, receive a redirect URL or tunneled stream. Cobalt handles PO tokens internally. Can be self-hosted or used via community-run public instances.

Tier 4 — Piped. A federated, community-run YouTube frontend with a public API. Each instance operator runs their own server (typically on a residential or VPS IP), and the API exposes stream URLs. Multiple instances can be rotated for resilience.

Tier 5 — Invidious. Another federated YouTube frontend. The key feature is the local=true parameter on its /latest_version endpoint, which tells the Invidious server to proxy the video stream through its own IP rather than redirecting to googlevideo.com.

Tier 6 — Transcript API. YouTube's transcript/subtitle endpoint runs on a separate rate-limit bucket from the video player API. It can be used as a preflight reachability check — if the transcript endpoint returns 429 (rate limited), the video is unreachable from this IP and no video download method will succeed.

Tier 7 — PO Token providers. The BGutil POT provider runs Google's BotGuard attestation in Node.js to generate Proof-of-Origin tokens. These tokens prove to YouTube that the request comes from a "real" client, which can lift the LOGIN_REQUIRED block for certain clients.

Tech in a Minute: What is a PO Token?

A PO Token (Proof-of-Origin Token) is a cryptographic attestation that YouTube requires for certain API calls. It is generated by running Google's BotGuard challenge — a piece of JavaScript that performs computations to prove the client is a real browser/app, not a bot. The token is bound to a specific video ID and expires after hours. YouTube introduced PO Tokens in 2024 to combat automated downloading. The bgutil-ytdlp-pot-provider project runs the BotGuard challenge in Node.js (simulating a browser environment) to generate valid tokens without needing an actual browser.

Tier 8 — Free SOCKS5 proxies. Public proxy lists contain thousands of SOCKS5 proxies run by volunteers, businesses, and (sometimes) botnets. Testing them in parallel against YouTube can find proxies whose IPs are not flagged, allowing the download to be routed through them.

Tier 9 — GitHub Actions. GitHub's CI/CD runners run on Microsoft Azure IPs, which have a different reputation profile than typical cloud provider IPs. A workflow can be triggered via API to download a video on the runner and upload it as an artifact.

Tier 10 — Cloudflare Workers. Edge functions run on Cloudflare's network, which has residential-like IP reputation. A Worker can proxy the YouTube request.

Tier 11 — Cloudflare WARP. A VPN-like service that routes traffic through Cloudflare's network, masking the origin IP. Typically installed as a Docker container or system package.

The ranking

Each method was scored on feasibility, cost, complexity, and reliability. The research concluded with a clear ranking and a recommended fallback chain. The key insight was that no single method is reliable enough to be the only method — every approach has failure modes (rate limits, IP blocks, service outages, version breakages). The only path to reliability was composition: try multiple methods in sequence, and let the system learn which ones work best in the current environment.

3. First Approach: The Naive Wrapper

What we built

The first iteration was a Python CLI tool with a clean architecture: an Orchestrator that walks a fallback chain of download methods, a Verifier that checks file integrity, a Truth Agent that learns which methods work best, and a Tester for end-to-end validation. The initial method chain had ten tiers, from transcript probing through yt-dlp with various clients to Cobalt, Piped, and Invidious.

The system was designed around four principles:

Determinism. No LLM at runtime. The Orchestrator is a state machine that makes decisions based on file existence and method results, not probabilistic reasoning.
Verification. A download is not "done" until the Verifier confirms it: file exists, size ≥ 1MB, magic bytes match a known video container, ffprobe succeeds, duration > 0, and (for MP4) the moov atom is intact.
Fallback. If a method fails, the next one is tried. The chain is exhaustive.
Learning. The Truth Agent tracks success/failure ratios and reorders the chain. After 3 consecutive failures, a method is demoted.

Why it was supposed to work

The research indicated that the android_vr and visionos innertube clients are "JS-less" — they do not require signature deciphering and historically did not require PO Tokens. The default yt-dlp chain (visionos → android_vr → web) was supposed to handle most public videos. For the remaining cases, the BGutil POT provider would generate tokens, and for everything else, the Cobalt/Piped/Invidious methods would serve as fallbacks.

The initial success

The first end-to-end test downloaded a very popular, very old video — a video so widely embedded and referenced that YouTube's caching and reputation systems treat it as permanently whitelisted. The download completed in under 5 seconds: a 243-megabyte, 213-second, AV1/Opus MP4 file. The Verifier confirmed all six integrity checks. The system worked.

Tech in a Minute: What is the moov atom?

An MP4 file is structured as a series of "boxes" (also called "atoms"). The moov box contains all the metadata — the video and audio codecs, the duration, the frame timing, the sample tables that tell a player where each frame is. If the moov box is missing or truncated, the file cannot be played even if the video data is intact. Some MP4 files put the moov box at the end of the file (to allow streaming creation), which means a truncated download leaves an unplayable file. The +faststart flag in ffmpeg moves the moov box to the beginning, making the file playable even if truncated. Our Verifier walks the box structure to confirm moov is present and well-formed.

The first crack

Emboldened by the initial success, a batch of ten public videos was queued for download. All ten failed. Every method in the chain returned LOGIN_REQUIRED or produced no output. The system that had worked perfectly moments before was now completely non-functional. The Truth Agent dutifully recorded the failures and demoted every method to the bottom of the chain.

The investigation began.

4. The Wall: Datacenter IP Detection

Diagnosing the block

The first step was to determine whether the failure was on our end (a bug in the code) or on YouTube's end (a block). A simple curl request to the watch page of a failing video revealed the answer:

"playabilityStatus": {"status": "LOGIN_REQUIRED", "reason": "Sign in to confirm you're not a bot"}

The watch page itself returned HTTP 200, but the embedded player configuration contained LOGIN_REQUIRED — YouTube had decided, based on our IP address alone, that we were a bot and refused to serve video streams.

An IP geolocation check confirmed the root cause: the cloud environment runs on an IP belonging to a datacenter range. YouTube maintains reputation scores for IP ranges, and datacenter IPs — especially those from certain providers and regions — are flagged as high-risk for automated abuse.

Tech in a Minute: What is IP Reputation Scoring?

Every IP address on the internet belongs to a block assigned to an ISP or cloud provider. These blocks are categorized: "residential" (assigned to home internet subscribers), "datacenter" (assigned to cloud servers), "mobile" (assigned to cell towers), "educational" (universities), etc. Anti-bot services maintain databases of these categorizations. When you make a request to a protected service, it looks up your IP's category. Datacenter IPs are treated with suspicion because real humans browse from residential IPs; a request from a datacenter IP is more likely to be a bot, a scraper, or an automated tool. This is why the same YouTube video loads fine on your home WiFi but shows "Sign in to confirm you're not a bot" when accessed from a cloud server.

The critical discovery: per-video variation

The most puzzling aspect was that one specific video downloaded successfully while ten others failed. The answer lay in YouTube's layered reputation system:

IP-level reputation. Our datacenter IP was flagged, triggering heightened scrutiny for all requests.
Video-level reputation. Highly popular, old, widely-embedded videos have high reputation scores and are served from cache with relaxed security. Obscure or newer videos trigger stricter checks.
Behavioral analysis. The pattern of requests (no prior watch history, no session cookies, direct API calls) further signaled automation.

The successful video was so popular and so deeply cached that YouTube served it without applying the full anti-bot gauntlet. The ten failing videos were popular enough to be public but not popular enough to bypass the IP-level block.

What didn't work

Every direct method was tried and failed:

yt-dlp with android_vr client: LOGIN_REQUIRED
yt-dlp with web client: LOGIN_REQUIRED (also requires JS signature deciphering, which needs a JS runtime)
yt-dlp with mweb client + BGutil POT: LOGIN_REQUIRED (the PO token was generated but YouTube still rejected the request at the playability level)
yt-dlp with ios client: LOGIN_REQUIRED
yt-dlp with tv and tv_downgraded clients: LOGIN_REQUIRED
Direct Innertube API with all clients: LOGIN_REQUIRED
Passing session PO tokens via extractor_args: LOGIN_REQUIRED (the token was valid but the block happens before the token is checked)

The block was not at the token level — it was at the playability level. YouTube's server decided, before even looking at our PO token, that this IP was not allowed to access this video's player API. The PO token was irrelevant.

5. Understanding the Block: Technical Deep Dive

The playability gate

YouTube's video access pipeline has multiple gates:

Request → IP Reputation Check → Playability Check → PO Token Check → Stream URL Generation → Video Delivery
           ↑                      ↑                    ↑
           Datacenter IPs         LOGIN_REQUIRED       Token validated
           flagged here           returned here        after playability passes

The critical insight is that the playability check happens before the PO token check. If the IP reputation + behavioral analysis determines the request is suspicious, it returns LOGIN_REQUIRED immediately. No PO token, however valid, can bypass this gate because the gate is evaluated before the token is even examined.

This is why the BGutil POT provider — which successfully generates valid Proof-of-Origin tokens — could not fix the problem. The tokens were correct, but they were being submitted to a gate that had already closed.

Tech in a Minute: What is "Playability Status"?

When you request a YouTube video, the first thing YouTube's server does is check whether you're allowed to play it. It returns a "playability status" in the response: OK (you can play it), LOGIN_REQUIRED (sign in first), UNPLAYABLE (the video is restricted), LIVE_STREAM_OFFLINE (the live stream hasn't started), or ERROR (something went wrong). For a cloud-based AI agent, the most common status is LOGIN_REQUIRED — YouTube has decided, based on your IP and request pattern, that you need to prove you're human by signing in. Since the agent has no account, this is a hard block.

Why the Cobalt and Piped approaches also failed

Cobalt and Piped are proxy services — they make the YouTube request from their own IP and relay the response. The failure here was twofold:

Self-hosted Cobalt runs on the same blocked IP. Running Cobalt locally doesn't help because Cobalt's requests to YouTube originate from the same datacenter IP.
Public Cobalt/Piped instances require authentication or are themselves blocked. The Cobalt API now requires a JWT token (which requires solving a Cloudflare Turnstile challenge — a browser-only task). Piped instances are either down, behind Cloudflare browser verification, or return SignInConfirmNotBotException because the instance operator's IP is also flagged.

Why Invidious `local=true` is different

The Invidious local=true parameter is a special case. Most Invidious endpoints return a redirect to googlevideo.com (the actual video CDN). Following that redirect sends you directly to Google's servers, which apply the same IP reputation check — so you get blocked.

The local=true parameter tells the Invidious server: "Don't redirect me. Fetch the video from googlevideo.com yourself and stream it to me." The Invidious server acts as a middleman. Because the Invidious server (run by a volunteer on a residential or VPS IP) fetches from googlevideo.com, YouTube sees a request from a non-blocked IP. The video streams to the Invidious server, which re-streams it to you. Your blocked IP never touches googlevideo.com.

This was the first bypass method that actually worked.

6. Second Approach: The PO Token Era

The hypothesis

After understanding that the block was at the playability level, the next hypothesis was that a valid "session PO token" — a token bound to a visitor session rather than a specific video — might satisfy the playability check. The BGutil server's generate_once.js script can create a visitor_data + po_token pair without needing to fetch the video's watch page.

What was tried

Generate a session token via BGutil script mode. The script successfully produced a contentBinding (visitor_data) and poToken pair.
Pass the token to yt-dlp via extractor_args. The token was injected as youtube:visitor_data=<token>;po_token=mweb.player+<token>,mweb.gvs+<token>.
Test with multiple clients (mweb, web_embedded, web).

Why it failed

The session token approach still resulted in LOGIN_REQUIRED. The reason, discovered through verbose yt-dlp logging, is that YouTube's playability check considers IP reputation first. A request from a flagged datacenter IP is rejected at the playability level before the PO token is evaluated. The PO token is a necessary condition for video stream access, but it is not sufficient when the IP itself is blocked.

The key finding

This was the pivotal discovery of the project: the datacenter IP block cannot be defeated from the datacenter. No amount of token generation, client switching, or header spoofing can change the fact that the request originates from a flagged IP range. The only way to bypass the block is to make the request appear to come from a non-flagged IP.

Tech in a Minute: What is "Visitor Data"?

When you visit YouTube, the server assigns you a "visitor data" token — a unique identifier that tracks your session across requests. It's like a cookie, but embedded in the page's JavaScript. YouTube uses this to correlate your requests: "This visitor watched video A, then searched for B, then watched video C." For automated tools, providing a valid visitor_data token (generated fresh, not stolen) can make your requests look more like a real user's session. However, as we discovered, if your IP is flagged as a datacenter IP, even a perfect visitor_data token won't help — the block happens at the network level, not the session level.

This finding reframed the entire project. The question was no longer "how do we make yt-dlp work from this IP?" but "how do we route the request through a different IP?"

7. Third Approach: The Bypass Architecture

The reframing

With the understanding that the IP block is the root cause, the project shifted from "build a better downloader" to "build an IP-bypass system that happens to download videos." The fallback chain was redesigned to include methods that route requests through non-datacenter IPs.

Bypass method 1: Cobalt community relays

The Cobalt project maintains a directory of community-run instances. Some of these instances do not require JWT authentication and will proxy YouTube downloads through their own servers. The key instance that worked was a "relay" — a server that forwards requests to multiple backend Cobalt instances, each potentially on a different IP.

How it works: The agent POSTs the video URL to the relay. The relay forwards the request to a backend Cobalt instance (running on a residential or VPS IP). The backend fetches the video from YouTube (YouTube sees the backend's IP, not ours), and returns a "tunnel" URL. The agent then downloads from the tunnel URL, which streams the video through the relay.

Limitation: The relay IP-blocks after approximately 2 successful calls per source IP. This is a rate limit imposed by the relay operator to prevent abuse. For batch downloads, this means only 2 videos can be downloaded before the relay stops working.

Bypass method 2: Invidious `local=true` proxy

As described in Section 5, the local=true parameter makes the Invidious server proxy the video stream through its own IP.

How it works: A simple curl command: curl -L -o video.mp4 "https://<invidious-instance>/latest_version?id=<video_id>&itag=18&local=true". The Invidious server fetches the video from googlevideo.com and streams it back.

Limitation: Most Invidious instances have disabled the local=true parameter (because it consumes their bandwidth). Only a small number of instances still support it, and they are unreliable — going up and down throughout the day. The quality is limited to 360p (itag=18, muxed H.264/AAC MP4).

Bypass method 3: SOCKS5 proxy farm

This was the most reliable bypass method discovered. The approach:

Fetch a list of free SOCKS5 proxies from public proxy list repositories (community-maintained lists of thousands of proxies).
Test proxies in parallel (50 at a time) against the target video.
Two-phase test: Phase 1 fetches the watch page and checks that playabilityStatus is OK (not LOGIN_REQUIRED). Phase 2 POSTs to the innertube player API and confirms streamingData is present (stream URLs are returned).
Download via the first working proxy using yt-dlp's --proxy flag.

Why the two-phase test matters: The initial implementation tested proxies by fetching only the watch page. This found proxies that could load the watch page but failed on the player API call (because YouTube applies different reputation logic to the player API). The two-phase test ensures the proxy can complete the full extraction pipeline, not just the first step.

Tech in a Minute: What is a SOCKS5 Proxy?

A SOCKS5 proxy is a server that relays your network traffic through its own IP address. When you connect to a SOCKS5 proxy and request a website, the proxy connects to the website on your behalf, receives the response, and forwards it to you. The website sees the proxy's IP, not yours. SOCKS5 is the "no-frills" version — it doesn't modify your traffic, just forwards it. Free SOCKS5 proxies are run by volunteers, businesses, and sometimes botnets; they're unreliable (many are dead at any given time) but there are thousands of them, so finding a few working ones by testing in parallel is feasible. The trade-off: free proxies are slow, may log your traffic, and should never be used for sensitive data.

Limitation: Free proxies are ephemeral. A proxy that works now may be dead in 5 minutes. The success rate is approximately 0.5-2% of tested proxies passing both phases. Testing 200 proxies typically yields 1-4 working ones. Download speed through a free proxy is often 1-5 MB/s.

Bypass method 4: GitHub Actions remote download farm

GitHub Actions runners run on Microsoft Azure IPs, which have a different reputation profile than typical cloud provider IPs. Additionally, runners have sudo access, allowing the installation of Cloudflare WARP (which routes traffic through Cloudflare's residential-like IP pool).

How it works:

A GitHub Actions workflow is defined in .github/workflows/yt-download-farm.yml.
The agent triggers the workflow via the GitHub API (POST /repos/{owner}/{repo}/actions/workflows/{id}/dispatches), passing the video URL as input.
The workflow runs on an ubuntu-latest runner: installs Cloudflare WARP, installs yt-dlp + BGutil, downloads the video, and uploads it as a workflow artifact.
The agent polls the workflow status, then downloads the artifact via the GitHub API.

Limitation: GitHub Actions have a 6-hour job timeout and a 10 GB artifact size limit. The workflow takes 2-5 minutes to complete (runner startup + WARP install + download + artifact upload). The artifact is retained for 7 days. Rate limits: 15 workflow dispatches per minute, 100 artifact downloads per hour.

Tech in a Minute: What is Cloudflare WARP?

Cloudflare WARP is a free VPN-like service provided by Cloudflare. When installed on a machine, it routes all outgoing traffic through Cloudflare's global network. From the perspective of the website you're visiting, your request comes from a Cloudflare IP (which has good reputation because Cloudflare is a legitimate CDN provider) rather than your actual cloud provider's IP. WARP is not a full VPN — it doesn't hide your traffic from Cloudflare — but it effectively changes your IP reputation. For our use case, installing WARP on a GitHub Actions runner makes YouTube see a Cloudflare IP instead of an Azure IP, which is often enough to bypass the datacenter block.

8. The Breakthrough: Compositional Bypass

The key realization

No single bypass method is reliable:

Cobalt relays rate-limit after 2 calls.
Invidious instances go up and down.
SOCKS5 proxies die within minutes.
GitHub Actions has rate limits and latency.

But together, they form a resilient system. If the Cobalt relay is rate-limited, the Invidious proxy might be up. If Invidious is down, the SOCKS5 farm can find a working proxy. If all else fails, the GitHub Actions farm provides a guaranteed (if slow) path.

The test that proved it

A batch of ten public videos — all of which had been completely blocked (0/10 success rate) in the first approach — was re-tested with the bypass architecture. The results:

9 out of 10 videos downloaded successfully (605 MB total data acquired).
8 videos were downloaded via the SOCKS5 proxy farm method.
1 video was downloaded via the Invidious local=true method.
1 video failed all methods (the specific video may have additional restrictions or the proxy pool was exhausted for that hour).

Why composition works

The key insight is that the bypass methods fail independently. The Cobalt relay's rate limit has nothing to do with the Invidious instance's uptime, which has nothing to do with whether a SOCKS5 proxy is alive. By composing them into a fallback chain, the system achieves a success rate that no individual method can reach.

This is a classic application of the diversity principle from reliability engineering: a system composed of diverse components with independent failure modes is more reliable than any single component. Mathematically, if method A has a 30% success rate and method B has a 40% success rate, and their failures are independent, the combined success rate is 1 - (0.7 × 0.6) = 58% — higher than either alone.

Tech in a Minute: What is a "Fallback Chain"?

A fallback chain is a list of methods to try in order. You try the first method; if it fails, you try the second; if that fails, you try the third; and so on until one succeeds or all fail. It's the same concept as having a spare tire and a can of fix-a-flat — if the spare is flat, you use the fix-a-flat; if both fail, you call a tow truck. The key design decision is the order: fast and reliable methods go first, slow and unreliable methods go last. In our system, the direct yt-dlp methods (which fail fast when blocked) are tried first, followed by the bypass methods (which are slower but can actually work).

9. Final Architecture: The 13-Method Fallback Chain

The final system implements a 13-tier fallback chain, managed by an Orchestrator that walks the chain, hands each result to a Verifier, and records observations for the Truth Agent to learn from.

The complete chain

Tier	Method	Strategy	Bypasses IP block?
0	Transcript probe	Preflight reachability check (separate rate-limit bucket)	N/A (fast fail)
1	yt-dlp default	`android_vr` client + BGutil PO token	No (needs whitelisted video)
2	yt-dlp JS-less	`visionos, android_vr` clients + BGutil	No
3	yt-dlp IOS	IOS client (HLS streams)	Sometimes
4	yt-dlp single file	Pre-muxed MP4 (no ffmpeg merge)	No
5	yt-dlp audio only	Audio-only salvage (last yt-dlp tier)	No
6	Innertube direct	Hand-rolled POST to `/youtubei/v1/player`	No
7	Cobalt (self-hosted)	Local Cobalt sidecar	Yes (if Cobalt has good IP)
8	Cobalt community	Public Cobalt relay instances	Yes
9	Piped	Piped API with 4-instance rotation	Varies
10	Invidious	`local=true` proxy (instance streams through its IP)	Yes
11	SOCKS5 farm	Free SOCKS5 proxy discovery (two-phase test)	Yes
12	GitHub Actions farm	Remote download on GitHub runners + WARP	Yes

The agents

The system is built around four in-process agents (not LLM agents — deterministic Python modules with specific roles):

The Orchestrator receives a video URL, resolves it to an ID, asks the Truth Agent for the ranked method list, and walks the chain. For each method, it creates a fresh temp directory, runs the method with a timeout, and hands any successful result to the Verifier. If the Verifier passes, the file is atomically moved to the output directory. If the Verifier fails, the method's result is treated as a failure and the next method is tried.

The Truth Agent owns two state files: truth.json (the ranked method list with success/failure counters) and observations.jsonl (an append-only audit log of every method attempt). After every method attempt, the Orchestrator records an observation. After 3 consecutive failures of a method, the Truth Agent demotes it (rank += 1, capped at the bottom). After a success, the method may be promoted (rank -= 1, floored at the top). This means the system learns which methods work best in the current environment and reorders the chain over time.

The Verifier runs 6 layered checks on any file a method claims to have downloaded: (1) file exists and size ≥ 1 MB, (2) magic bytes match a known video container, (3) ffprobe succeeds, (4) duration > 0, (5) at least one video or audio stream is present, (6) for MP4 files, the moov atom is present and well-formed. These checks catch truncated downloads, HTML error pages saved as .mp4, and corrupted files.

The Tester runs the full pipeline against a corpus of known-stable public videos, producing a markdown report of which methods succeeded for each video. This serves as both a health check and a regression test.

The auto-bootstrap

A critical design decision was that the system should require zero manual setup. If an AI agent runs ytagent download <url> on a fresh machine, the system should auto-bootstrap everything it needs:

If the BGutil POT provider is not installed, it is cloned from GitHub, npm dependencies are installed, TypeScript is compiled, and the HTTP server is started — all automatically, in about 15 seconds.
If the BGutil server is not running, it is started.
The download then proceeds through the 13-method chain.

This is implemented in the bootstrap.py module, which is called transparently by the yt-dlp methods when they detect that BGutil is not available.

The agent instructions

The system includes a ytagent agent-instructions command that prints a complete 10-step guide for AI agents. This covers installation, setup, downloading, JSON parsing, bypass methods, self-testing, truth inspection, and batch downloads. An AI agent reading only this output can use the system end-to-end without any human help.

This was validated in a final test: an AI agent following the instructions downloaded a previously-blocked video with zero manual human intervention. The socks5_farm method found a working proxy in approximately 100 seconds and downloaded a 69-megabyte, 1080p video.

10. Key Learnings and Takeaways

Learning 1: The IP is the identity

In modern web infrastructure, your IP address is your identity. More precisely, the reputation category of your IP (residential vs. datacenter vs. mobile) determines what you can access. Cookies, tokens, and session identifiers are secondary — they are checked only after the IP reputation gate. For AI agents operating from cloud environments, this means that no amount of authentication or token generation can overcome a flagged IP.

Takeaway: When building systems that interact with anti-bot-protected services, the first question to ask is not "how do I authenticate?" but "what is the reputation of my IP, and how do I route through a better one?"

Learning 2: Composition beats optimization

The most reliable system was not the one with the best single method — it was the one with the most diverse set of methods. Each bypass method had a 30-60% success rate individually, but composed together, the chain achieved a 90% success rate. This is because the methods fail independently: the Cobalt relay's rate limit is unrelated to the Invidious instance's uptime, which is unrelated to whether a SOCKS5 proxy is alive.

Takeaway: In adversarial environments (anti-bot systems, rate limits, service outages), invest in breadth (many methods) over depth (one perfect method). The probability that all methods fail simultaneously is exponentially lower than the probability that any single method fails.

Learning 3: Verification is non-negotiable

In the testing phase, it became clear that download methods frequently produce "successful" results that are actually broken: HTML error pages saved as .mp4, truncated downloads missing the moov atom, 0-byte files from connection resets. Without a rigorous Verifier, these broken files would be silently passed to downstream agents, causing confusing failures later.

Takeaway: Every automated pipeline needs a verification layer that checks the output is actually what it claims to be. For video files, this means magic bytes, ffprobe, duration, and moov atom checks. Trust no method's self-report; verify independently.

Learning 4: Free infrastructure is fragile but usable

Every free bypass method (Cobalt relays, Invidious instances, SOCKS5 proxies) was unreliable when relied upon individually. But because they were free, they could be composed in parallel. A paid residential proxy service would have been more reliable, but the goal was to build a system that works with zero cost.

Takeaway: Free infrastructure is a viable component of a production system if (and only if) you compose enough of it to achieve the reliability you need. The cost of free infrastructure is engineering complexity (more methods, more failure handling), not money.

Learning 5: Determinism enables trust

The system was designed from the start to be deterministic: no LLM at runtime, no probabilistic decisions, no "ask the AI what to do next." Every decision is made by a state machine with clear rules. This made the system testable, debuggable, and trustworthy. When a download failed, the audit trail in observations.jsonl showed exactly which methods were tried and why they failed.

Takeaway: AI agents that need to interact with external systems should use deterministic orchestration, not LLM-driven decision-making. The LLM should be used at design time (to research, plan, and write code), not at runtime (to decide what to do next). Determinism is what makes the system auditable and reliable.

Learning 6: The two-phase test pattern

The SOCKS5 proxy farm method initially failed because it tested proxies by fetching only the watch page. Many proxies could load the watch page but failed on the player API call. The fix was a two-phase test: check the watch page first, then check the player API. This pattern — test the full pipeline, not just the first step — generalized to other areas of the system.

Takeaway: When testing whether a resource (proxy, API, service) is "working," test the entire operation you need it to perform, not just the first step. A proxy that can load a web page may not be able to stream a video. An API key that works for metadata may not work for downloads. Test the actual use case.

Learning 7: Documentation is part of the system

The ytagent agent-instructions command was not an afterthought — it was a first-class feature. An AI agent that can read the instructions and use the system without human help is a more valuable system than one that requires a human to read the README and explain it. The instructions include exact commands, expected output, and failure troubleshooting for each step.

Takeaway: For systems designed to be used by AI agents, the documentation should be machine-readable and executable. Don't just describe what the system does — provide a step-by-step guide that an agent can follow mechanically.

11. Future Implications

The splintering of the internet

The datacenter IP block is not unique to YouTube. It is a pattern being adopted by an increasing number of services: social media platforms, news sites, e-commerce, and even some APIs now apply differential access based on IP reputation. The internet is splintering into two tiers: a "residential tier" where content is freely accessible, and a "datacenter tier" where access is gated by CAPTCHAs, login requirements, and behavioral challenges.

For AI agents — which necessarily operate from datacenter IPs — this splintering represents a fundamental access problem. An agent that cannot route its traffic through residential IPs will find itself increasingly locked out of the web.

The rise of proxy-as-a-service

As datacenter IP blocks become more common, a market is emerging for "residential proxy" services that sell access to residential IP pools. These services (Bright Data, Smartproxy, etc.) charge per GB of traffic routed through their pools. For AI agent operators, this will become a line item in the infrastructure budget, much like compute and storage.

The alternative — composing free bypass methods, as this project does — is viable for low-volume use but does not scale. A production system processing thousands of videos per day would need a paid proxy service or a self-hosted residential IP pool (e.g., Oracle Cloud Free Tier VMs with WARP installed).

The attestation arms race

YouTube's PO Token system is part of a broader trend toward attestation-based access control. Instead of (or in addition to) checking your IP, services increasingly require proof that your client is "real" — that it ran a specific piece of JavaScript, performed a specific computation, or holds a specific hardware attestation. Apple's App Attest, Google's Play Integrity, and Cloudflare's Turnstile are all examples of this trend.

Tech in a Minute: What is "Attestation"?

Attestation is the process of proving that a piece of software is running in a trusted environment. When a website says "prove you're a real browser," it's asking for attestation. The website sends a challenge (usually a piece of JavaScript that performs computations), and your browser returns the result as "proof" that it executed the challenge. The challenge is designed to be hard to fake — a bot would need to run a full browser engine to solve it. Cloudflare Turnstile, Google reCAPTCHA, and YouTube's BotGuard are all attestation systems. They differ in how hard the challenge is and what they're trying to prove (real browser? real mobile device? not a bot?).

For AI agents, attestation is a double-edged sword. On one hand, projects like BGutil can run the attestation challenge in a simulated browser environment (Node.js with JSDOM), generating valid tokens without a real browser. On the other hand, the attestation systems are evolving to detect these simulations (checking for browser-specific APIs, timing characteristics, rendering quirks). The arms race will continue, with each side getting more sophisticated.

Implications for AI agent design

This project suggests several principles for designing AI agents that interact with web services:

Assume IP blocks. Any agent running in a cloud environment should have a strategy for dealing with IP-based access restrictions. This means having a pool of proxies or fallback methods, not relying on a single access path.
Build for diversity. Don't depend on a single service, API, or method. The web is adversarial — services change their APIs, add new blocks, and rate-limit aggressively. A diverse set of fallback methods is the only defense.
Separate intelligence from access. The LLM should be used for reasoning about what to download, not for the mechanics of downloading. The download itself should be handled by a deterministic, tested, verified pipeline. This separation allows the intelligence to be upgraded without breaking the access layer.
Audit everything. Every request, every method attempt, every failure should be logged. When a download fails, the audit trail should show exactly what was tried and why it failed. This is essential for debugging and for building trust in the system.
Design for the agent, not the human. The end user of an AI agent's download capability is another AI agent (or a pipeline), not a human. The output should be machine-readable (JSON), the instructions should be machine-followable (step-by-step), and the failure modes should be machine-recoverable (try the next method).

The broader security picture

The fact that a cloud-based AI agent cannot download a public YouTube video without building a 13-method bypass chain reveals something important about the state of the internet: the security measures designed to stop bots are also stopping legitimate automated access. The same IP reputation systems that block scrapers also block AI agents. The same attestation challenges that stop credential stuffing also stop automated research.

This creates a paradox: as AI agents become more capable and more widely deployed, they will increasingly need to access web content that is protected by anti-bot systems. But those systems were designed to stop automated access. The result is a cat-and-mouse game where AI agents develop increasingly sophisticated bypass methods, and anti-bot systems develop increasingly sophisticated detection methods.

The long-term resolution is likely a shift from IP-based access control to identity-based access control. Instead of blocking datacenter IPs, services may issue API keys or attestation tokens to verified AI agents, allowing them to access content programmatically without the overhead of bypassing anti-bot systems. YouTube's official Data API v3 is a step in this direction (though it does not allow video downloads). Until such APIs become comprehensive, the bypass architecture described in this document will remain necessary.

Conclusion

This project began as a simple wrapper and evolved into a study of modern web access control. The journey from "just use yt-dlp" to "compose 13 methods including SOCKS5 proxy farms and GitHub Actions runners" reflects the reality of building automated systems for today's web.

The key insight — that the datacenter IP is the root cause, not the download method — reframed the entire problem. Once the block was understood as an IP-level gate that runs before any token or authentication is checked, the solution became clear: route through a different IP. The implementation of that solution required composing multiple bypass methods into a resilient fallback chain, because no single bypass is reliable enough to stand alone.

The resulting system — a 13-method, self-healing, auto-bootstrapping download pipeline with a learning Truth Agent and rigorous file verification — represents a practical answer to a problem that every cloud-based AI agent will face: how do you access a web that doesn't want you?

The answer, as it turns out, is not to break the security measures. It is to find enough legitimate access paths that at least one of them works, and to compose them in a way that makes failure the exception rather than the rule.

This research document describes the architecture and findings of a real project. All technical details, failure modes, and success rates are based on actual testing conducted during the project's development. The system is open-source and available for review, modification, and extension.

Who is Ansari Bilal?

Who built it. This project — ytagent — was built by Ansari Bilal, a self-taught app builder from India.

Repository: https://github.com/Bilal140202/ytagent

Ansari Bilal is a self-taught app builder. What started as late-night tinkering turned into a second life — shipping real products, building open-source tools, and solving problems through code. From privacy-first desktop apps built with Tauri & Rust to developer tools used by thousands, every project is fueled by curiosity, not credentials. No CS degree, no bootcamp — just relentless building and a GitHub full of proof.

The mission is simple: build things that are useful, open, and well-crafted — whether that's an AI-powered agent tool, a cross-platform note-taking app, or a curated directory of 30,000+ developer resources.

Other things he has shipped:

The Lord of the Skills — 18,142+ AI agent skills across 14 frameworks, sorted into 10 LOTR-themed kingdoms.
Linuxify — run Linux developer CLIs and AI coding agents on Android, no root required.
snip — project-scoped command snippets in a committable .snips file, with fuzzy and natural-language matching.
DevAtlas — a searchable index of 30,000+ developer tools, refreshed daily.
TalkDrive — SoulX FlashHead 1.3B talking-head generation on a free Colab T4.

What ytagent was built with

ytagent is a Python system that composes existing, legitimate tooling into a resilient chain rather than replacing any of it:

Python 3 — orchestration, method registry, deterministic fallback controller
yt-dlp — the core extractor, driven through multiple client and transport configurations
ffmpeg — stream merging and post-processing
Proof-of-Origin token providers — attestation where the player API demands it
Public mirror / proxy front-ends — alternate access paths when the direct player API refuses a datacenter IP
No LLM at runtime — the download path stays deterministic and testable

Source: https://github.com/Bilal140202/ytagent

More work: ansaribilal.com/projects · GitHub · Hugging Face

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

AI Search Is the New Front Page: llms.txt, Schema, and What Actually Got Me Cited

Ansari Mohammed Bilal — Tue, 04 Aug 2026 11:37:53 +0000

Last updated: August 3, 2026 · 9-minute read

Somewhere in the last year, "how do I rank" quietly became two questions: how do I rank in search, and how do I get quoted by an assistant. They overlap, but not completely — and the second one has a much shorter list of requirements.

This is everything I implemented on this site, in the order that mattered.

First: the boring foundation still decides everything

No amount of AI-specific markup helps a page a crawler can't read. Before anything else:

Server-rendered or prerendered HTML. A client-only React app hands most crawlers an empty <div id="root">. I generate a static HTML shell for every route at build time — 66 files, each with real title, description and body content.
One canonical URL per page, self-referencing, agreeing with the sitemap.
robots.txt that doesn't block the assistants. GPTBot, ClaudeBot, PerplexityBot, Google-Extended — decide deliberately. Blocking them is a valid choice; blocking them by accident is not.

I wrote about how the pipeline generates all of this in the automated blog pipeline post.

Second: `llms.txt` — a table of contents for models

llms.txt is a plain markdown file at your root that tells a model what your site is and where the good stuff lives. It is not a magic ranking file; it is a cheap, high-signal summary that gets pulled into context when a model fetches your domain.

# Ansari Bilal — builder, indie dev, AI agent tinkerer

> Sales agent by day, builder by night. Writes about AI agents, MCP,
> indie builds and developer tooling.

## Core pages
- [Projects](https://ansaribilal.com/#projects): shipped apps, notebooks, Hugging Face Spaces
- [Blog](https://ansaribilal.com/blog): 50+ posts on agents, tooling and shipping
- [Uses](https://ansaribilal.com/uses): the actual stack

## Topic hubs
- [AI Agents](https://ansaribilal.com/topics/ai-agents)
- [UI & Components](https://ansaribilal.com/topics/ui-components)

Keep it under a page. Link, don't essay. Pair it with a machine-readable profile (/me.json) if you want a single fetch to answer "who is this person".

Third: structured data that describes reality

Three schema types cover almost every personal or product site:

Schema	Where	What it buys you
`Person` or `Organization`	Sitewide	Entity recognition — you become a thing, not a string
`Article` / `BlogPosting`	Every post	Author, date, publisher, image in the crawler's model
`FAQPage`	Pages with real Q&A	Direct answer extraction

The two mistakes I made and fixed: publisher must be an Organization (not a Person), and image should be an ImageObject with dimensions, not a bare URL. Both silently invalidated the block.

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Context Engineering Beat Prompt Engineering",
  "datePublished": "2026-08-03",
  "author": { "@type": "Person", "name": "Ansari Bilal" },
  "publisher": { "@type": "Organization", "name": "ansaribilal.com" },
  "image": { "@type": "ImageObject", "url": "https://ansaribilal.com/og.jpg", "width": 1200, "height": 630 }
}

Validate every type you emit. An invalid block is worth exactly zero.

Fourth: topic hubs instead of 50 orphan posts

Fifty unlinked posts look like fifty unrelated documents. Fifty posts grouped into six hub pages, each with an editorial intro and links to its cluster, look like a site with expertise on six subjects.

This is the change that moved the needle most for me — both for search and for assistants, which lean heavily on internal link structure to decide what a site is about. Each hub gets its own title, description, and a paragraph a model can quote verbatim.

Fifth: write the paragraph you want quoted

Assistants extract passages, not pages. The practical consequences:

Answer in the first 60 words under each heading, then elaborate. Inverted pyramid, every section.
Use real numbers and dates. "LCP under 2.5s on mobile" gets quoted; "fast loading" doesn't.
Make headings questions people ask. ## Does llms.txt actually do anything? outperforms ## Considerations.
Tables get extracted disproportionately often. If your content has a comparison in it, put it in a table.
State your position. Hedged content is unciteable — the model has nothing to attribute to you.

Sixth: feeds and freshness signals

An RSS feed, a sitemap with honest lastmod values on posts (and no lastmod on static pages you didn't change), and IndexNow submissions on publish. Note that Google's and Bing's old /ping?sitemap= endpoints are retired — they return errors now. Use the Indexing API, Bing's Webmaster API, or IndexNow instead.

What I measure

Signal	Where	Cadence
Impressions & average position	Search Console	Weekly
Referrals from `chatgpt.com`, `perplexity.ai`	Analytics referrers	Weekly
"Cited by" spot-checks	Ask the assistant a question you answered	Monthly
Core Web Vitals	PageSpeed API in CI	Daily

That last spot-check is unscientific and the most useful. Ask the question your post answers and see whether you're in the answer.

The honest summary

llms.txt is a nice-to-have. Prerendered HTML, valid schema, a coherent internal link graph, and passages written to be quoted are the actual work — and all four also happen to be plain good SEO. The tooling changed; the fundamentals didn't.

If you're starting from zero, 12 weeks of blogging has the traffic numbers behind all of this.

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

The 5 Sites I Actually Use for Design Inspiration (Not Dribbble)

Ansari Mohammed Bilal — Tue, 04 Aug 2026 11:37:48 +0000

Last updated: August 13, 2025 · 6-minute read

I open Dribbble roughly once a month. Every time I do, I close it within 90 seconds feeling worse about design than when I opened it.

That's not a Dribbble problem — it's a category problem. Dribbble is a portfolio platform. People upload work designed to impress other designers, not to be implemented. The medium article "Designing for Dribbble Killed Real Web Creativity" called it out directly: the platform incentivizes pretty pixels over functional interfaces, and the result is an entire ecosystem of work that was never meant to be built.

A thread on r/UI_Design titled "Dribbble Quality Downhill" put it more bluntly — "about 95% of Dribbble nowadays are mockups created solely to look good and flashy on Dribbble and to share to other social media." A senior web designer in r/FigmaDesign was even more direct: "I'm a senior web designer and have never looked on Dribbble. I prefer live, real-world examples."

If you're building things — not decorating portfolios — you need references from shipped products, not concept art. Here's my actual rotation.

1. godly.website — The Creative Edge

Godly is a curated gallery of visually aggressive, often experimental web designs. Think parallax scrolling, 3D transforms, WebGL effects, the kind of stuff that wins Awwwards but would get you fired if you proposed it for a SaaS dashboard.

What I use it for: Landing page hero sections, scroll interactions, and "what's technically possible right now" research. When I need to see what the cutting edge looks like — not what I should copy, but what's pushing boundaries — this is where I go.

Pros: Updated daily, zero signup required, the curation is sharp. Every site listed is live and production-built.

Cons: Practically useless for anything behind a login. The designs skew heavily toward creative agencies and portfolio sites. As one Hacker News commenter noted about Godly: "I find it a helpful resource, but I also agree too many of these are design studios and are of no interest to me." If you're building a B2B tool, 80% of what's here won't apply.

Verdict: Best for front-end devs who want to see what CSS and WebGL can actually do. Not for anyone looking for a pricing page layout.

2. mobbin.com — The Real-World Pattern Library

Mobbin is a library of 400,000+ screenshots from real mobile and web apps. No mockups. No concepts. Actual screens from shipped products, organized by flow (onboarding, settings, checkout, search, empty states).

What I use it for: Any pattern that involves user interaction. Need to see how 50 different apps handle dark mode toggles? Mobbin. Need a reference for a notification center layout? Mobbin. This is the closest thing to "let me see how real products solved this" that exists.

Pros: The search is genuinely good — you can filter by platform, flow, and even specific UI elements. Reddit threads on design inspiration consistently mention it as the go-to. One thread on r/web_design: "Good list, mobbin and godly are in my rotation too."

Cons: The free tier is limited. You hit a paywall fast if you're browsing heavily. And since everything is real apps, you won't find anything experimental here — it's all safe, production-grade patterns.

Verdict: The single most useful site on this list if you build apps. Worth the paid tier if you reference designs daily.

3. land-book.com — Landing Page Dissection

Land-book is a curated collection of landing pages from real SaaS products, startups, and agencies. Every entry is a live page you can click through, not a static screenshot.

What I use it for: Landing page structure. Hero section hierarchy, CTA placement, social proof blocks, feature grid layouts. When I need to understand how a page flows — not just what one screen looks like — this is better than Mobbin because you get the full scroll experience.

Pros: Every site is clickable. You can see hover states, scroll animations, responsive behavior. The curation skews toward products, not agencies, which means more practical reference material.

Cons: The library is smaller than Mobbin's. Updates are less frequent. Search is basic — you're browsing, not querying.

Verdict: Best for "I need a landing page and I want to see 20 good ones before I start."

4. siteinspire.com — The Minimalist's Reference

Siteinspire has been around since 2006. It's a straightforward gallery of well-designed websites, filterable by style, type, subject, and platform.

What I use it for: Clean, minimal designs. If Godly is the experimental wing, Siteinspire is the "this is tasteful and works" wing. Great for typography references, whitespace usage, and restrained color palettes.

Pros: 7,000+ sites, well-organized filters, no fluff. A Reddit thread on favorite inspiration sites listed it as a top pick alongside Mobbin.

Cons: Feels slightly dated compared to newer entries. The visual style of what gets featured is very "design agency 2019." You won't find dark-mode SaaS dashboards here.

Verdict: Reliable for clean, corporate-grade web design. The kind of thing you'd show a client who says "make it look professional."

5. saaslandingpage.com — Niche and Useful

A smaller site that does exactly what the URL says — it collects landing pages specifically from SaaS products.

What I use it for: B2B-specific patterns. Pricing tables, feature comparison sections, integration grids, testimonial layouts. Things you won't find on Godly or Siteinspire because they're boring — but essential if you sell software.

Pros: Highly relevant if you build SaaS. Every entry is a product landing page, not a portfolio or agency site. Quick to browse.

Cons: Small library. Not updated as frequently. The design is bare-bones — the site itself could use a designer.

Verdict: Niche, but if you're building a SaaS landing page, this should be your first stop.

The YouTube Reference Worth Mentioning

Self-Made Web Designer put out a video called "38 Must-Visit Web Design Inspiration Resources" that covers everything from the well-known (Godly, Mobbin) to the obscure. Worth a watch if you want to go deeper than these five — but the video itself is 20+ minutes and most of the 38 are redundant. These five cover 90% of what I actually open.

TL;DR

Godly — experimental, creative, award-worthy sites. Use for hero sections and scroll interactions.
Mobbin — 400k+ screenshots from real apps. Use for UI patterns and interaction flows.
Land-book — clickable landing page gallery. Use for landing page structure and flow.
Siteinspire — clean, minimal, corporate. Use for typography and whitespace reference.
SaaS Landing Page — B2B-specific patterns. Use for pricing tables and feature sections.
Dribbble — skip it if you build things. It's a portfolio platform, not a reference library.

If you're looking for design systems you can actually hand to an AI agent, the Refero Styles breakdown in the Post-Lovable stack post covers how I extract real design tokens from these references. And if you want to see what I've built using this approach, the Projects section has the output.

Not affiliated with Godly, Mobbin, Land-book, Siteinspire, or SaaS Landing Page. Tools used for this site's design research: Refero Styles, Aceternity UI.

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

I Built a Video Pipeline with 3 Open-Source Tools (No Editor Needed)

Ansari Mohammed Bilal — Mon, 03 Aug 2026 12:42:36 +0000

Last updated: September 17, 2025 \u00b7 7-minute read

Eight weeks ago, I ran an experiment turning HTML into video using agents and browser automation. The output was rough \u2014 10-second clips with inconsistent timing and fonts that broke at different resolutions. But the core idea stuck: if I can define a video as HTML, I can version-control it, template it, and automate it.

That early experiment is documented in my Week 3 post. Now, eight weeks later, the pipeline is production-grade. Here\u2019s what it actually looks like.

The Problem I\u2019m Actually Solving

Over on r/webdev, a post titled \u201cI Couldn\u2019t Find a Good Open-Source Web Video Editor, So I Built One\u201d hit 139 upvotes. The author built a React-based video editor because every open-source option was either abandoned, Electron-bloated, or missing basic features.

That post resonated because it described exactly the gap I\u2019d been feeling. But my approach was different. Instead of building a video editor, I eliminated the editor entirely.

The use case: I needed to generate short, structured videos programmatically \u2014 launch announcements, feature walkthroughs, data visualizations. The content was templatable. The format was repeatable. Opening Premiere or DaVinci Resolve for each one was wasteful when 90% of the \u201cediting\u201d was placing text on a background with a fade transition.

The Three-Tool Pipeline

Stage 1: HTML Templates

Every video starts as an HTML file. Not a mockup \u2014 the actual source of truth. I define layouts using plain HTML and CSS with a few conventions:

data-frame attributes mark animation keyframes
CSS animations handle transitions between frames
Media assets (images, audio) are referenced by relative path

This means I can write a \u201cvideo\u201d the same way I write a webpage. I use Tailwind for styling. I use CSS @keyframes for timing. The entire template is just a static HTML file that looks correct when opened in a browser.

Why HTML? Because every developer already knows it. Because you can version-control it in git. Because you can generate it with any templating engine. Because a browser is the most reliable rendering engine on the planet.

Stage 2: HyperFrames

HyperFrames is an open-source framework from HeyGen that turns HTML, CSS, and seekable animations into deterministic MP4 videos. Apache 2.0 licensed. The pitch: \u201cWrite HTML. Render video. Built for agents.\u201d

HyperFrames takes my HTML template and renders it frame-by-frame using a headless browser. It reads the data-frame attributes to understand timing, captures each frame as an image, and pipes them to ffmpeg for encoding.

The key feature that made HyperFrames the right choice is seekable animations. It doesn\u2019t just record a playback \u2014 it can seek to any point in the animation timeline and render that specific frame. Same input, same output, every time.

Stage 3: html-video + ffmpeg

For the final encoding, I use html-video combined with ffmpeg for audio muxing and format optimization:

ffmpeg -framerate 30 -i frames/%04d.png -i audio.mp3 \
  -c:v libx264 -preset medium -crf 23 \
  -c:a aac -b:a 128k \
  -shortest output.mp4

What the Pipeline Looks Like in Practice

video-project/
\u251c\u2500\u2500 templates/
\u2502   \u251c\u2500\u2500 launch-announcement.html
\u2502   \u2514\u2500\u2500 feature-walkthrough.html
\u251c\u2500\u2500 assets/
\u2502   \u251c\u2500\u2500 logo.svg
\u2502   \u2514\u2500\u2500 background.mp3
\u251c\u2500\u2500 scripts/
\u2502   \u251c\u2500\u2500 render.sh
\u2502   \u2514\u2500\u2500 encode.sh
\u251c\u2500\u2500 output/
\u2514\u2500\u2500 config.json

To create a new video, I either write the HTML template manually or generate it with an LLM agent. The agent gets the content brief and outputs a valid HTML file with the correct data-frame attributes. Then I run two shell commands. Done.

The Hard Parts

Frame-perfect timing was the biggest challenge. CSS animations don\u2019t guarantee sub-frame accuracy \u2014 a 1-second animation might complete at 0.98s or 1.02s depending on the browser\u2019s rendering pipeline. I solved this by using discrete keyframe steps instead of smooth transitions for anything that needs to sync precisely.

Font rendering was the second issue. Google Fonts load asynchronously, which means the first few frames might render in a fallback font. I embedded the fonts as base64 in the HTML template. Ugly in source code, but the output is consistent.

What Eight Weeks of Iteration Taught Me

The Week 3 experiment produced janky 10-second clips. The Week 10 pipeline produces broadcastable short-form content. The difference wasn\u2019t any single breakthrough \u2014 it was 40+ iterations on templates, timing calibration, and understanding where the frame-level rendering breaks down.

The biggest lesson: the tooling is finally good enough that the bottleneck is now content, not rendering. HyperFrames handles the hard part. My job is writing good HTML templates and having something worth saying in them.

TL;DR

Three-tool pipeline: HTML templates \u2192 HyperFrames (renders frames) \u2192 ffmpeg (encodes MP4)
No video editor needed. Code in, video out.
Deterministic rendering: same HTML always produces the same video
Hard parts: frame timing precision, font loading, audio sync
40+ iterations from janky 10-second clips to production-grade output

If you\u2019re generating video content programmatically \u2014 product demos, social posts, tutorial overlays \u2014 this pipeline eliminates the editor entirely. Check out the Lab for the templates I\u2019m open-sourcing, and the TalkDrive post for how I handle the audio layer with AI-generated voiceovers.

Not affiliated with HeyGen or HyperFrames. Tools used: HyperFrames (Apache 2.0), html-video, ffmpeg, Tailwind CSS. Built on a Linux workstation.

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

12 Weeks of Blogging as a Builder: What Actually Got Traffic

Ansari Mohammed Bilal — Mon, 03 Aug 2026 12:42:32 +0000

Last updated: September 24, 2025 \u00b7 9-minute read

I committed to writing one post per week for 12 weeks on this portfolio site \u2014 a React SPA with no CMS, no SSR, no Next.js metadata magic. Every post lives as a string in a TypeScript array. The entire blog is client-rendered. By conventional SEO wisdom, this should be a graveyard.

It isn\u2019t. Here\u2019s what actually happened.

The Posts

Week	Post	Category
1	I Replaced Lovable	Insights
2	Claude Code Memory MCP	Tutorials
3	HTML-to-Video Agents	Experiments
4	Aceternity vs Magic UI vs shadcn	Insights
5	Trigger.dev Background Jobs	Tutorials
6	5 Design Inspiration Sites	Insights
7	Refero Styles Experiment	Experiments
8	TestSprite AI Testing	Tutorials
9	Top 5 MCP Servers	Insights
10	OSINT with Sploitas	Experiments
11	Video Pipeline Build	Builds
12	This post	Insights

Traffic by Post Type

Tutorials consistently outperformed everything else. The CloudStream guide and the $300 AI Credits post each pulled 4-6x the traffic of the average insights or builds post. People bookmark these. They come back.

\u201cHonest review\u201d posts had the longest tail. The Lovable replacement post started slow but kept getting traffic week after week. Posts that take a clear stance have legs because people reference them in conversations.

Builds and experiments were the weakest performers for traffic \u2014 but strongest for credibility. The TalkDrive post didn\u2019t get huge traffic, but three people reached out directly about collaborating. The Hugging Face Spaces post and PrivacyThink writeup function more as proof-of-work than traffic drivers.

Social Distribution: Reddit vs X vs LinkedIn

I cross-posted every article to three platforms. The results were not even close.

Reddit: 8x the click-through of X/Twitter. Tutorial posts on relevant subreddits (r/webdev, r/selfhosted, r/privacy, r/ChatGPT) consistently drove the most qualified traffic. A key insight from Ross Simmonds\u2019 talk on Reddit\u2019s influence on search confirmed what I was seeing \u2014 Reddit is now a dominant force in how people discover content, especially for technical topics.

The nuance: Reddit rewards specificity. A post titled \u201cHow I Replaced Lovable with Three Tools\u201d performs better than \u201cMy Design System Journey.\u201d

X/Twitter: high impressions, low clicks. Thread summaries got 2,000-8,000 impressions per thread. But click-through hovered around 0.3-0.5%. X is a brand-building channel, not a traffic channel. As one Reddit thread on r/buildinpublic put it: \u201cYour customers are not scrolling Twitter looking for someone\u2019s revenue milestone. They\u2019re googling their problem.\u201d

LinkedIn: negligible for this audience. Posted every article. Almost zero engagement for technical content. LinkedIn works for enterprise thought leadership. It doesn\u2019t work for \u201chere\u2019s how to pipe HTML through HyperFrames into an MP4.\u201d

SEO Learnings

Internal Linking Is the Highest-ROI Move

Every post on this site links to at least two other posts. The Lovable post links to the Antigravity post and the AI Credits post. The CloudStream guide links to the Drive migration post. And so on.

Over on r/SEO, a thread confirmed this: \u201cInternal linking is one of the most underrated SEO tactics for new websites. It helps Google discover pages faster, understand topic authority, and distribute link equity.\u201d

My site has no external backlinks worth mentioning. The only reason Google finds and ranks individual blog posts is internal linking. Every post is reachable from every other post within 1-2 clicks. This is especially critical for a client-rendered SPA where Googlebot has to work harder.

Sitemap Strategy for a SPA

Since the blog is client-rendered, I generated a static sitemap.xml that lists every post slug. Google can\u2019t execute JavaScript to discover client-side routes \u2014 the sitemap is the only way crawlers find individual posts. Basic but essential.

Timing Doesn\u2019t Matter as Much as I Thought

I published every post on Wednesday mornings (IST). The theory was that US readers would see it Tuesday evening, Indian readers Wednesday morning, European readers Wednesday midday. In practice, the day of publication had almost no correlation with traffic. What mattered was the subreddit I posted it to and whether the title was specific enough to stop someone scrolling.

What Post Types Performed Best (Ranked)

Tutorials with concrete steps \u2014 CloudStream, AI Credits, Drive Migration. 4-6x average traffic.
Honest tool reviews with real alternatives \u2014 Lovable Replacement. Slow burn but consistent monthly traffic.
Experiments with surprising findings \u2014 OSINT footprint, Antigravity 2.0. Shareable because the results are interesting regardless of whether you\u2019re a developer.
Build stories \u2014 TalkDrive, Video Pipeline. Low traffic, high credibility.
Philosophy/career posts \u2014 Sales Agent, Why I Build. Lowest traffic but most personally satisfying.

What I\u2019d Do Differently

Write more tutorials. The traffic data is unambiguous. Tutorials with specific, copy-pasteable steps are what people search for and share.
Post to Reddit more strategically. Not every subreddit for every post. Match the post to the single best subreddit and write a Reddit-native title.
Start building an email list from week 1. I didn\u2019t, and now I\u2019m relying entirely on algorithmic discovery for return visitors.
Double down on internal linking. The best distribution strategy for a solo builder is writing useful things and linking them to each other.

The Honest Takeaway

Twelve weeks of consistent blogging taught me that the \u201cbuild in public\u201d meta-narrative \u2014 the revenue screenshots, the milestone threads, the \u201cday 47 of my startup\u201d posts \u2014 is mostly noise. What actually works is writing the thing someone is googling at 11pm when they\u2019re stuck on a problem.

The CloudStream guide works because someone is googling \u201cCloudStream repos not working.\u201d The Lovable replacement works because someone is googling \u201cLovable alternatives.\u201d The OSINT experiment works because someone is googling \u201ccheck my digital footprint.\u201d

Write the answer to the question. Link it to the other answers. Post it where people ask questions. That\u2019s the entire strategy.

The full post archive has everything, and the Projects section has the things I actually built instead of just wrote about. The Lab has the experiments that didn\u2019t make the cut but taught me something anyway.

No tools were used to game traffic. Analytics from Cloudflare. Cross-posting done manually. Not affiliated with Reddit, X, or LinkedIn.

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

Graphify: Turning Your Notes Into a Knowledge Graph AI Agents Can Actually Query

Ansari Mohammed Bilal — Mon, 03 Aug 2026 12:42:06 +0000

7-minute read · October 2026

I have ~1,400 Markdown notes accumulated over 6 years — Obsidian, then Notion, then back to plain files. RAG on top of them works, sort of. What actually works is Graphify, an AI-agent skill that extracts entities and relationships from every note and hands the agent a real graph to traverse.

RAG Isn't Enough (For Notes)

Semantic search on notes gives you the chunk that sounds most like the question. But my notes are dense with proper nouns — people, projects, tools, concepts. I don't want the chunk that sounds relevant; I want "every note that mentions Sanjay AND the Refero project." That's a graph query, not a vector one.

What Graphify Does

Walks a folder of Markdown.
Runs each file through an extraction pass — entities (Person, Project, Tool, Concept) and relationships (Sanjay WORKED_ON Refero, Refero USES Tailwind).
Writes it all to a lightweight graph store (SQLite by default; Neo4j if you want).
Exposes an MCP tool the agent can call: graph.query({ entities, relationships }).

Setup

git clone https://github.com/... graphify
cd graphify && npm install
npm run index -- ~/notes

Add to Claude Code:

{
  "mcpServers": {
    "graphify": {
      "command": "node",
      "args": ["/path/to/graphify/mcp.js", "--db", "~/notes/graph.db"]
    }
  }
}

Ask Claude: "Who has worked on projects using Tailwind, based on my notes?" — you'll get a real answer, not a hallucination.

Where It Shines

Research synthesis. Ask "connect these three concepts" and the agent traverses actual edges.
CRM-lite. Track people, projects, and last-touched dates without a real CRM.
Post ideas. Ask "which topics have I mentioned >5 times and never blogged about?" — free content pipeline.

Where It Struggles

Entity resolution is noisy. "Sanjay" and "S. Kumar" get treated as different people until you add aliases.
Extraction cost. First index of 1,400 notes cost me ~$3 in Claude tokens. Incremental updates are free.
Schema drift. If you rename an entity type mid-way, re-index the whole folder.

Pair With

Codebase memory MCP for a similar effect on code, not notes.
Lord of the Skills catalog to discover more skills like Graphify.

Bottom Line

RAG is a hammer. Graphify gives your agent a screwdriver too. For anything where the relationships matter more than the text — research, second brains, personal CRM — this is the shape of AI-native tooling I want more of in 2026.

I'm building a variant that indexes Twitter bookmarks. Follow along on X.

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

Testing Telegram Notifications for Auto-Published Posts

Ansari Mohammed Bilal — Mon, 03 Aug 2026 12:42:01 +0000

1-minute read

This is a test post to verify that the Telegram notification pipeline works end-to-end.

When the scheduled publisher flips this post from draft to published, the workflow should:

Regenerate the sitemap and feed
Commit and push the changes
Send a Telegram message with the blog URL

If you're reading this on the blog, it worked.

Built with: GitHub Actions, Telegram Bot API, Node.js.

Originally published at ansaribilal.com. I write about AI agents, indie builds and developer tooling — more posts here.

The Ultimate Collection of 18K+ AI Agent Skills Across 14 Frameworks

Ansari Mohammed Bilal — Thu, 02 Jul 2026 01:41:07 +0000

The AI agent landscape is currently in its "Wild West" phase. Every day, a new framework emerges — Claude Code, Cursor, Cline, OpenHands, AutoGen — and with each one comes a new way to define "skills" or "tools."

The problem? Most developers are reinventing the wheel. We're all writing the same or tool for our specific framework, while a perfectly good version exists in another repo, formatted just a little differently. This fragmentation slows down progress and creates unnecessary silos in the AI development community.

Today, we are launching The Lord of the Skills to solve this.

One Repo to Rule Them All 👑

We have aggregated 18,142 AI agent skills from over 307 repositories into a single, searchable directory. This isn't just a list of links; it's a curated, deduplicated, and themed catalog designed to be the "central library" for the agentic era.

When we started this project, we didn't realize how deep the rabbit hole went. We found skills buried in obscure Gists, hidden in the subdirectories of massive monorepos, and scattered across dozens of individual agent frameworks. By bringing them all into one place, we're enabling cross-framework discovery for the first time.

14 Frameworks Supported

Framework	Skills Count
Claude Code	8,104
Cursor	1,400+
Cline / Roo	2,200+
Google Antigravity 🆕	820
Aider	600+
OpenHands	400+
AutoGen / CrewAI / LangGraph	200+
Continue, Goose, Copilot & more	500+

Whether you are building local-first agents or massive cloud-based swarms, we've got you covered.

Mapping the Kingdoms 🗺️

Managing 18,000+ skills is a daunting task. A simple list would be unusable. To make it manageable and — dare we say — fun, we've organized the entire library into 10 LOTR-themed Kingdoms. This provides a mental model for discovery that goes beyond simple keyword searching.

⚔️ Gondor — Web & UI (10,141 skills)

Browser automation, UI interaction, web scraping, frontend frameworks (React, Vue, Tailwind).

⛏️ Moria — DevOps & Infrastructure (914 skills)

Terminal management, cloud infrastructure (Terraform, AWS, K8s), kernel debugging, database administration.

🌿 Lothlórien — Research & Analysis (496 skills)

Data science (Pandas, NumPy), academic search (ArXiv, Google Scholar), vector databases (Pinecone, Milvus), RAG.

🔥 Mordor — Security & Auditing (406 skills)

Penetration testing, red-teaming, security auditing, vulnerability scanning.

🧙 Isengard — Agents & Orchestration (1,630 skills)

Multi-agent orchestration, high-level design, workflow automation, state machines.

Plus 5 more kingdoms:

✦ Rivendell — Research & Knowledge (1,776 skills)
✎ The Shire — Writing & Content (1,272 skills)
🐴 Rohan — Testing & Verification (718 skills)
🌳 Fangorn — Documentation & Memory (708 skills)
🕸 Mirkwood — Specialized & Niche (81 skills)

Quality Over Quantity: The 357 Canonical Skills ⭐

With over 18k items, there is inevitably a lot of noise. That's why we've done the hard work of deduplication and curation. We have identified 357 Canonical Skills — the gold standard versions of the most common tools. If you need a "Google Search" skill, don't look through 50 different versions; just use the Canonical one.

Why it Matters: The End of Re-Invention 🎯

As agents move from "cool toys" to "production employees," the ability to quickly equip them with specialized skills becomes a major bottleneck.

Imagine you want to build a specialized agent for auditing cloud security. Instead of spending two weeks writing the AWS and security tool wrappers from scratch, you can:

Browse the Moria kingdom for cloud tools
Browse the Mordor kingdom for security scripts
Find the Canonical versions of each
Drop them into your framework of choice

You've just saved 10 days of development time. This is the power of a centralized skill directory.

Join the Fellowship 🤝

The Lord of the Skills is brand new and we're building this for the community, by the community. We have 5 good first issues on GitHub waiting for contributors.

Whether you're a Python expert, a prompt engineer, or just someone who loves Lord of the Rings, there's a place for you in the fellowship.

👉 Star the repo on GitHub — every star tells the compiler the kingdom was worth building.

One Repo to Rule Them All. ⚔

DEV Community: Ansari Mohammed Bilal

OSINT with Sploitas: What I Found About My Own Digital Footprint

The Starting Point: Where I Looked

What I Found

1. Emails I Forgot Existed

2. Abandoned Accounts

3. Data Brokers

The Cleanup Process

What This Means for How I Build

TL;DR

Superhero UI Review: The Boldest React Component Library of 2026?

What Superhero UI Actually Is

The Good

The Painful

Head to Head

When I'd Reach For It

When I Wouldn't

Combine, Don't Commit

Bottom Line

AI Testing That Actually Works: TestSprite Setup (After Trying Everything Else)

The Problem With Traditional Testing in 2025

What TestSprite Actually Does

Setup Walkthrough

Step 1: Create a TestSprite Account

Step 2: Add the MCP Server to Cursor

Step 3: Point It at Your App

Step 4: Review Results

TestSprite vs Cypress vs Playwright vs Raw AI

When AI Testing Makes Sense (And When It Doesn't)

TL;DR

The Datacenter IP Block: Building a Self-Healing YouTube Download System for AI Agents

Abstract

Table of Contents

1. The Problem: Why AI Agents Can't Download Videos

The use case

Why existing tools fail

2. Initial Research: Mapping the Landscape

The methods surveyed

The ranking

3. First Approach: The Naive Wrapper

What we built

Why it was supposed to work

The initial success

The first crack

4. The Wall: Datacenter IP Detection

Diagnosing the block

The critical discovery: per-video variation

What didn't work

5. Understanding the Block: Technical Deep Dive

The playability gate

Why the Cobalt and Piped approaches also failed

Why Invidious local=true is different

6. Second Approach: The PO Token Era

The hypothesis

What was tried

Why it failed

The key finding

7. Third Approach: The Bypass Architecture

The reframing

Bypass method 1: Cobalt community relays

Bypass method 2: Invidious local=true proxy

Bypass method 3: SOCKS5 proxy farm

Bypass method 4: GitHub Actions remote download farm

8. The Breakthrough: Compositional Bypass

The key realization

The test that proved it

Why composition works

9. Final Architecture: The 13-Method Fallback Chain

The complete chain

The agents

The auto-bootstrap

The agent instructions

10. Key Learnings and Takeaways

Learning 1: The IP is the identity

Learning 2: Composition beats optimization

Learning 3: Verification is non-negotiable

Learning 4: Free infrastructure is fragile but usable

Learning 5: Determinism enables trust

Learning 6: The two-phase test pattern

Learning 7: Documentation is part of the system

Why Invidious `local=true` is different

Bypass method 2: Invidious `local=true` proxy

Second: `llms.txt` — a table of contents for models